Patent Publication Number: US-2019187705-A1

Title: Method and system for personalized self capability aware route planning in autonomous driving vehicles

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     The present application is a continuation of U.S. patent application Ser. No. 15/845,173 filed Dec. 18, 2017 and is related to U.S. patent application Ser. No. 15/845,294 filed on Dec. 18, 2017, U.S. patent application Ser. No. 15/845,337 filed on Dec. 18, 2017 and U.S. patent application Ser. No. 15/845,423 filed on Dec. 18, 2017, all of which are incorporated herein by reference in their entireties. 
    
    
     BACKGROUND 
     Technical Field 
     The present teaching generally relates to autonomous driving. More specifically, the present teaching relates to planning and control in autonomous driving. 
     2. Technical Background 
     With recent technological advancement in artificial intelligence (AI), there is a surge in applying AI in different application fields. This includes the field of autonomous driving, in which planning and control are essential. This is shown in  FIG. 1  (Prior Art), in which an autonomous driving module  110  includes a planning module  120  and a vehicle control module  130 . Planning may include, as shown in  FIG. 2 , route planning, motion planning, or behavior planning. Route planning refers to the effort to plan a route from a source to a destination based on certain considerations. 
     Motion planning may generally refer to the effort of planning the movement of a vehicle to achieve certain effect. For example, the movement of the vehicle may be planned in a way that complies with the traffic regulations or safety. Motion planning is then to determine what movement the vehicle needs to make to achieve that. Behavior planning generally refers to the effort to plan how the vehicle should behave in different situations, e.g., the vehicle behavior while crossing an intersection, the vehicle behavior in staying within or following a lane, or the vehicle behavior in making a turn. For instance, in terms of overtaking a slow moving front vehicle, certain vehicle behavior may be planned. Behavior planning and motion planning may be related. For example, the planned vehicle behavior may need to be translated into motion in order to implement the behavior. 
     Vehicle control  130  as shown in  FIG. 1  may involve various aspects of control. This is illustrated in  FIG. 3 , which shows that vehicle control may involve, e.g., roadway specific control, motion specific control, mass specific control, geometry specific control, aerodynamic specific control, and tire specific control. 
     Surrounding information  100  in  FIG. 1  may be used for vehicle planning. Traditionally, surrounding information  100  includes, e.g., current location of the vehicle, intended destination, and/or traffic information. Utilizing such surrounding information, the conventional planning module  120  may devise, e.g., a plan for a route from the current location to the destination. Known criteria used in route planning may include, e.g., shortest distance, shortest time, use of highways, use of local roads, traffic, etc. Such criteria may be applied based on known information such as the distance of each road segment, known traffic patterns associated with roads, etc. 
     The planning module  120  may also perform motion planning, which is traditionally based on, e.g., rapidly exploring random trees (RRT) for state space or Markov Decision Process (MDP) for environmental modeling. The planning module  120  may generate, based on the planned route/motion, planning data to be fed to the vehicle control module  130  so that the vehicle control module  130  can proceed to control the vehicle in a way as planned. To make the vehicle to move to carry out the plan, the vehicle control module  130  may then generate control signals  140  which may be sent to different parts of the vehicle to implement the planned vehicle movement. Vehicle control is traditionally exercised based on generic vehicle kinematic models and/or different types of feedback controllers. 
     Each human driver generally operates or controls a vehicle differently with diverse preferences. Human drivers also operate vehicles adaptively based on real time situations, which may arise out of the present conditions of the vehicle itself, the extrinsic environment conditions that serve to limit the ability of the vehicle to operate, and/or the reaction or response to the current vehicle movement from passengers in the vehicle. For example, with children in the vehicle, a human driver may elect, for safety, to avoid (route planning) a route that is curvy on a snowy day. A human driver may drive in different manners when different passengers are riding in the vehicle to ensure comfort of the passenger. Although a human driver generally controls a vehicle by following a lane by staying roughly in the middle of the lane, the behavior may change when faced with a right turn. In this case, the same human driver may curve to the right side of the lane when the vehicle is approaching the point of the right turn. In addition, different human drivers may curve to the right in different ways. Furthermore, lane changing behavior may also differ with respect to different vehicles in different surrounding situations. The existing technologies do not address those issues, let alone providing solutions. 
     Therefore, there is a need to provide improved solutions for planning and control in autonomous driving. 
     SUMMARY 
     The teachings disclosed herein relate to methods, systems, and programming for online services. More particularly, the present teaching relates to methods, systems, and programming for developing a virtual agent that can have a dialog with a user. 
     In one example, a method for route planning for an autonomous driving vehicle is disclosed. A source location and a destination location are first obtained, where the destination location is where the autonomous driving vehicle is to drive to. One or more available routes between the source location and the destination location are identified. A self-aware capability model is instantiated with respect to the one or more available routes and is predictive of the operational capability of the autonomous driving vehicle with respect to each of the one or more available routes. The preference of a passenger present in the autonomous driving vehicle is determined in terms of a route to take for the autonomous vehicle to drive to the destination location. Based on the self-aware capability model and the preference of the passenger, a planned route to the destination location is then automatically selected for the autonomous driving vehicle. 
     In another example, a system for route planning for an autonomous driving vehicle is disclosed. The system comprises an interface unit, a global route planner, and a route selection engine. The interface unit is configured for obtaining information related to a source location and a destination location, wherein the destination location is where the autonomous driving vehicle is to drive to. The global route planner is configured for identifying one or more available routes between the source location and the destination location and determining a preference of a passenger present in the autonomous driving vehicle in terms of a route to take for the autonomous vehicle to drive to the destination location. The route selection engine configured for obtaining a self-aware capability model instantiated with respect to the one or more available routes, wherein the self-aware capability model is predictive of the operational capability of the autonomous driving vehicle with respect to the one or more available routes and selecting, from the one or more available routes, a planned route for the autonomous driving vehicle between the source location and the destination location based on the preference of the passenger and the self-aware capability model. 
     Other concepts relate to software for implementing the present teaching on developing a virtual agent. A software product, in accord with this concept, includes at least one machine-readable non-transitory medium and information carried by the medium. The information carried by the medium may be executable program code data, parameters in association with the executable program code, and/or information related to a user, a request, content, or information related to a social group, etc. 
     In one example, machine readable non-transitory medium is disclosed, wherein the medium has information stored thereon related to route planning for an autonomous driving vehicle so that the information, when read by the machine, causes the machine to perform following operational steps. A source location and a destination location are first obtained, where the destination location is where the autonomous driving vehicle is to drive to. One or more available routes between the source location and the destination location are identified. A self-aware capability model is instantiated with respect to the one or more available routes and is predictive of the operational capability of the autonomous driving vehicle with respect to each of the one or more available routes. The preference of a passenger present in the autonomous driving vehicle is determined in terms of a route to take for the autonomous vehicle to drive to the destination location. Based on the self-aware capability model and the preference of the passenger, a planned route to the destination location is then automatically selected for the autonomous driving vehicle. 
     Additional novel features will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and the accompanying drawings or may be learned by production or operation of the examples. The novel features of the present teachings may be realized and attained by practice or use of various aspects of the methodologies, instrumentalities and combinations set forth in the detailed examples discussed below. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The methods, systems and/or programming described herein are further described in terms of exemplary embodiments. These exemplary embodiments are described in detail with reference to the drawings. These embodiments are non-limiting exemplary embodiments, in which like reference numerals represent similar structures throughout the several views of the drawings, and wherein: 
         FIG. 1  (Prior Art) shows some essential modules in autonomous driving; 
         FIG. 2  illustrates exemplary types of planning in autonomous driving; 
         FIG. 3  illustrates commonly known types of vehicle control; 
         FIG. 4A  depicts an autonomous driving vehicle with a planning module and a vehicle control module, according to an embodiment of the present teaching; 
         FIG. 4B  illustrates exemplary types of real time data, according to an embodiment of the present teaching; 
         FIG. 5  depicts an exemplary high level system diagram of a planning module, according to an embodiment of the present teaching; 
         FIG. 6A  illustrates exemplary ways to realizing a safe-aware capability model, according to an embodiment of the present teaching; 
         FIG. 6B  illustrates an exemplary construct of a self-aware capability model with parameters, according to an embodiment of the present teaching; 
         FIG. 6C  illustrates exemplary types of intrinsic vehicle capability parameters, according to an embodiment of the present teaching; 
         FIG. 6D  illustrates exemplary types of extrinsic capability parameters, according to an embodiment of the present teaching; 
         FIG. 7  depicts an exemplary high level system diagram of a mechanism for generating self-aware capability parameters to be considered for planning, according to an embodiment of the present teaching; 
         FIG. 8  depicts an exemplary high level system diagram of a self-aware capability parameter generator, according to an embodiment of the present teaching; 
         FIG. 9  is a flowchart of an exemplary process for generating self-aware capability parameters, according to an embodiment of the present teaching; 
         FIG. 10  depicts an exemplary high level system diagram of a route planning module, according to an embodiment of the present teaching; 
         FIG. 11  is a flowchart of an exemplary process for route planning, according to an embodiment of the present teaching; 
         FIG. 12  depicts an exemplary high level system diagram of a global route planner, according to an embodiment of the present teaching; 
         FIG. 13  is a flowchart of an exemplary process for a global route planner, according to an embodiment of the present teaching; 
         FIG. 14A  depicts an exemplary high level system diagram of a motion planning module, according to an embodiment of the present teaching; 
         FIG. 14B  illustrates exemplary types of passenger models, according to an embodiment of the present teaching; 
         FIG. 14C  illustrates exemplary types of user reactions to be observed for motion planning, according to an embodiment of the present teaching; 
         FIG. 15  depicts an exemplary high level system diagram of a passenger observation analyzer, according to an embodiment of the present teaching; 
         FIG. 16  is a flowchart of an exemplary process of a passenger observation analyzer, according to an embodiment of the present teaching; 
         FIG. 17  is a flowchart of an exemplary process for a motion planning module, according to an embodiment of the present teaching; 
         FIG. 18  depicts an exemplary high level system diagram of a model training mechanism for generating different models for motion planning, according to an embodiment of the present teaching; 
         FIG. 19  illustrates different types of reactions to be observed and their roles in model training, according to an embodiment of the present teaching; 
         FIG. 20A  illustrates exemplary types of lane related planning, according to an embodiment of the present teaching; 
         FIG. 20B  illustrates exemplary types of behavior related to lane following, according to an embodiment of the present teaching; 
         FIG. 20C  illustrates exemplary types of behavior related to lane changing, according to an embodiment of the present teaching; 
         FIG. 21  depicts an exemplary high level system diagram of a lane planning module, according to an embodiment of the present teaching; 
         FIG. 22  is a flowchart of an exemplary process for a lane planning module, according to an embodiment of the present teaching; 
         FIG. 23A  illustrates the traditional approach to generate vehicle control signals based on a vehicle kinematic model; 
         FIG. 23B  depicts a high level system diagram of a vehicle control module that enables human-like vehicle control, according to an embodiment of the present teaching; 
         FIG. 23C  depicts a high level system diagram of a vehicle control module that enables personalized human-like vehicle control, according to an embodiment of the present teaching; 
         FIG. 24  depicts an exemplary high level system diagram of a human-like vehicle control unit, according to an embodiment of the present teaching; 
         FIG. 25  is a flow chart of an exemplary process of a human-like vehicle control unit, according to an embodiment of the present teaching; 
         FIG. 26  depicts an exemplary high level system diagram of a human-like vehicle control model generator, according to an embodiment of the present teaching; 
         FIG. 27  is a flowchart of an exemplary process of a human-like vehicle control model generator, according to an embodiment of the present teaching; 
         FIG. 28  depicts an exemplary high level system diagram of a human-like vehicle control signal generator, according to an embodiment of the present teaching; 
         FIG. 29  is a flowchart of an exemplary process of a human-like vehicle control signal generator, according to an embodiment of the present teaching; 
         FIG. 30  depicts the architecture of a mobile device which can be used to implement a specialized system incorporating the present teaching; and 
         FIG. 31  depicts the architecture of a computer which can be used to implement a specialized system incorporating the present teaching. 
     
    
    
     DETAILED DESCRIPTION 
     In the following detailed description, numerous specific details are set forth by way of examples in order to provide a thorough understanding of the relevant teachings. However, it should be apparent to those skilled in the art that the present teachings may be practiced without such details. In other instances, well known methods, procedures, components, and/or circuitry have been described at a relatively high-level, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings. 
     The present disclosure generally relates to systems, methods, medium, and other implementations for planning and control of route/vehicle behaviors in a self-capability aware, human-like, and personalized manner that is adaptive to real time situations.  FIG. 4A  shows an autonomous vehicle with a vehicle planning/control mechanism  410 , according to an embodiment of the present teaching. The autonomous vehicle planning/control mechanism  410  includes a planning module  440  and a vehicle control module  450 . Both modules take various types of information as input in order to achieve operations that are self-capability aware, human-like, personalized and adaptive to real time situations. For example, as shown, both the planning module  440  and the vehicle control module  450  receive historical manual driving data  430  in order to learn human like ways to handle the vehicle in different situations. These modules also receive real time data  480  in order to be aware of the dynamic situations surround the vehicle in order to adapt the operations accordingly. Furthermore, the planning module  440  accesses a self-aware capability model  490  which characterizes what limits the operational ability of the vehicle given the situation the vehicle is currently in. 
     Real time data  480  may include various types of information useful or relevant for planning and control of the vehicle.  FIG. 4B  illustrates exemplary types of real time data, according to an embodiment of the present invention. For example, exemplary real time data may include vehicle related data, time related data, passenger related data, weather related data, . . . , and data related to the nearby roads. Vehicle related data may include, e.g., the motion state, position, or conditions of the vehicle at the time. The motion state of a vehicle may involve, e.g., its current speed and driving direction. The real time position information may include, e.g., the current latitude, longitude, and altitude of the vehicle. The real time conditions of the vehicle may include the functional state of the vehicle such as whether the vehicle is currently in a full or partial functional state or specific parameters under which different components of the vehicle are operating, etc. 
     Real time data related to time may generally include current date, time, or month. Passenger related data may include various characteristics related to the passenger of the vehicle such as passenger reaction cues, which may include visual, acoustic, or behavior cues observed from the passenger, or conditions of the passenger such as mental state, physical state, or functional state of the passenger. The conditions of the passenger may be inferred based on the cues observed from the passenger reaction cues. Weather related data may include the weather of the locale where the vehicle is currently situated. The road related data may include information about the physical condition of the nearby road(s), e.g., wetness, steepness, or curviness of the road, or the local traffic condition such as congestion along the road. 
       FIG. 5  depicts an exemplary high level system diagram of the planning module  440 , according to an embodiment of the present teaching. In this exemplary embodiment, the planning includes, but not limited to, route planning, motion planning, and the planning of lane related behavior, including lane following, lane changing, etc. Accordingly, in this illustrated embodiment, the planning module  440  comprises a route planning module  550 , a motion planning module  560 , and a lane planning module  570 . Each of the modules aims at operating in a self-capability aware, human-like, and personalized manner. Each of the modules  550 ,  560 , and  570  takes, in addition to the surrounding information  420 , recorded human driving data  430 , real time data  480 , and the self-aware capability model  490  as inputs and generates their respective outputs to be used by the vehicle control module  450  to convert into the vehicle control signals  470  to control the vehicle. For example, the route planning module  550  generates the planned route information  520  as its output, the motion planning module  560  generates planned motion  530  as its output, and the lane planning module  570  generates planned lane control information  540  as its output. 
     Each of the planning modules may be triggered via some triggering signal. For instance, the route planning module  550  may be activated via a route planning trigger signal; the motion planning module  560  may be activated upon receiving a motion planning trigger signal; while the lane planning module  570  may start to operate when a lane planning trigger signal is received. Such a trigger signal may be manually provided (by, e.g., a driver or a passenger) or automatically generated based on, e.g., certain configuration or certain event. A driver may manually activate the route planning module  550  or any other planning module for the route/motion/lane planning, much like what people do to manually start, e.g., cruise control in a car. 
     The planning activities may also be activated by a certain configuration or an event. For example, the vehicle may be configured to activate route planning whenever the vehicle accepts an input indicating the next destination. This may be regardless what the current location of the vehicle is. In some embodiments, the planning modules may be always triggered on whenever the vehicle is on and depending on the situation, they may become engaged in different planning activities as needed. In different situations, they may also interact with each other in a manner called for by the situation. For example, the lane planning module  570  may determine to change lane in certain circumstance. Such a planned lane control is output by the lane planning module  570  and may be fed to the motion planning module  560  so that a specific path trajectory (planned motion) appropriate for carrying out the planned lane changing may be further planned by the motion planning module  560 . 
     Output of a planning module may be fed into another within the planning module  440  for either further planning or for providing an input for the future planning of another. For example, the output of the route planning module  550  (planned route  520 ) may be fed to the motion planning module  560  so that the route information may influence how the vehicle motion is planned. As discussed above, the output (planned lane control  540 ) of the lane planning module  570  may be fed to the motion planning module  560  so that the lane control behavior planned may be realized via planned motion control. Conversely, the output of the motion planning module  560  (the planned motion  530 ) may also be fed to the lane planning module  570  to influence the planning of the lane control behavior. For instance, in personalized motion planning, the motion planning module  560  may determine that the motion of the vehicle needs to be gentle due to the observation that the passenger of the vehicle prefers smooth motion. Such a determination is part of the motion planning and may be to be sent to the lane planning module  570  so that the lane control behavior of the vehicle may be carried out in a way that ensures smooth motion, e.g., change lane as little as possible. 
     To ensure that the vehicle behavior is planned and controlled in a self-capability aware manner, the route planning module  550 , the motion planning module  560 , and the lane planning module  570  also access the self-aware capability model  490  and use it to determine the planning strategy in a manner that takes into account of what the vehicle is actually capable of in the current scenario.  FIG. 6A  illustrates exemplary ways that the self-aware capability model  490  is realized, according to an embodiment of the present teaching. As illustrated, the self-aware capability model  490  may be constructed as a probabilistic model, a parameterized model, or a descriptive model. Such a model may be trained based on, e.g., learning. The model may include a variety of parameters to be used to characterize factors that may influence or have an impact on the actual ability of the vehicle. The model may be implemented as a probabilistic model with parameters being estimated probabilistically. The model may also be implemented as a parameterized model with explicit model attributes applicable to different real world conditions. The model  490  may also be provided as a descriptive model with enumerated conditions with values instantiated based on real time scenarios. 
     The self-aware capability model  490  in any situation may include various parameters, each of which is associated with some factors that may impact the actual ability of the vehicle so that the vehicle planning (route, motion, or lane) has to consider. In the following disclosure, self-aware capability model and self-aware capability parameters will be used interchangeably.  FIG. 6B  illustrates an exemplary construct of the self-aware capability model or parameters  510 , according to an embodiment of the present teaching. As illustrated, self-aware capability parameters  510  may include intrinsic capability parameters and extrinsic capability parameters. Intrinsic vehicle capability parameters may refer to parameters associated with the vehicle itself which may impact what the vehicle is capable of in operation and such parameters may be determined based on either how the vehicle is manufactured or how the vehicle is at the time. Extrinsic capability parameters may refer to the parameters or characteristics of the surrounding that are extrinsic to the vehicle but nevertheless may impact the way the vehicle can be operated. 
       FIG. 6C  illustrates exemplary types of intrinsic vehicle capability parameters, according to an embodiment of the present teaching. As illustrated, intrinsic vehicle capability parameters may include, but not limited to, characteristics of the vehicle in terms of, e.g., its engine, its safety measures, and its tires, etc. For instance, in terms of its engine, the intrinsic capability parameters may specify the maximum speed the vehicle is capable of, the control that can be exercised on the engine, including cruise control or any restrictions on manual control of the engine. In terms of safety measures, the intrinsic capability parameters may include information on what sensors the vehicle is equipped with, specific parameters related to breaks, or information about the seats of the vehicle. For example, some vehicles may have seats that are backed by metal support (stronger) and some with only plastic support. Some seats may have mechanism which allows automatic control to have vibrations and some may not. In terms of other components of the vehicle, the intrinsic capability parameters may also specify the type of the tires of the vehicle (which may have a bearing on what operation can be done) and whether the vehicle currently has snow tires installed or equipped with anti-skip measures. Such intrinsic vehicle capability parameters may be used to assess what types of routes and motions may be possible and which types of vehicle behaviors may be achievable. Thus, making such intrinsic capability parameters available to the planning modules allows the planning modules to plan appropriately without exceeding what the vehicle is actually capable of. 
       FIG. 6D  illustrates exemplary types of extrinsic capability parameters, according to an embodiment of the present teaching. As discussed above, extrinsic capability parameters specify information that are external to the vehicle but nevertheless may impact the ability of planning and such extrinsic capability parameters are used to determine appropriate planning given the conditions external to the vehicle. The ultimate output from the planning modules may be determined within the confine of both intrinsic and extrinsic capability parameters. As extrinsic capability parameters may include parameters describing the conditions or situations the vehicle is in or may encounter, they likely will impact what should be planned. For example, the surrounding situations related to the road(s), either close to the vehicle or even relatively remote from the vehicle. The road condition related parameters may indicate how crowded the road is (so that the driving speed cannot be planned to be fast), whether there are speed limits on the road (regulated minimum and maximum speeds as well as the actual permissible speed due to traffic), whether there is any accident along the road, or the surface condition of the road (so that the motion cannot be too swift), or whether the surface of the road presents certain conditions that will impede the ability of the vehicle in its planning. 
     There are other conditions external to the vehicle that may affect various planning activities. This includes the light or atomsphere related conditions as well as the surrounding of the vehicle. For instance, if the vehicle is positioned in such a way that there is a sun glare so that sensors may not work well, this will impact the planning decisions. If the vehicle is in an area with a heavy fog condition, such information is also important to the planning module. If there is a high level of precipitation, such information may also be taken into account by the planning modules. The surrounding traffic may also be important in terms of planning. For instance, extrinsic parameters may provide information related to nearby vehicles or objects so that the planning modules may consider such information in their respective planning. The extrinsic parameters may include information about such nearby vehicles/objects, e.g., the nearby vehicle is a big truck or a bicycle, which may also impact how the planning decision is made. In addition, events occur along the road the vehicle is on may also impact the planning. For instance, whether the vehicle is currently on a road that is in a school zone or whether there is a construction going on along the road the vehicle is currently on may also be important information to the planning modules for obvious reasons. 
     The extrinsic capability parameters may be acquired and updated continuously in time to support the planning modules to adapt their decisions based on external situations in real time. In some situations, the extrinsic capability parameters may also be predicted. For example, if the vehicle is driving on a road to the west in the afternoon, it may be predicted that there will be sun glare. Although such predicted extrinsic capability parameter may not be the real time information, it nevertheless will assist the planning module (e.g., the route planning module) to make an appropriate decision. For instance, if the intended destination for a vehicle is in the north west direction and there are roads available at the time to both the west and the north, knowing that there will be sun glare if heading to the west in late afternoon, the route planning module  550  may accordingly decide to presently take the road heading to the north first and later take a road to head to the west after sun is down to avoid sun glare (safer). Such predicted extrinsic capability parameters may be determined based on other information such as the current location of the vehicle and the intended destination of the vehicle. 
     With capability parameters (including both intrinsic and extrinsic), the vehicle becomes self-aware of both intrinsic and extrinsic capability related limitations, which may be crucial in terms of planning.  FIG. 7  depicts an exemplary high level system diagram for a mechanism  700  for generating self-aware capability parameters, according to an embodiment of the present teaching. In this illustrated embodiment, the mechanism  700  comprises a locale context determining unit  730  and a self-aware capability parameter generator  740 . The locale context determining unit  730  is to gather information locale to where the vehicle is and/or will be (i.e., both where the vehicle is presently and where the vehicle will be on its way to the destination) based on, e.g., information about the current location of the vehicle and/or the destination the vehicle is heading to. The self-aware capability parameter generator  740  is to generate both intrinsic and extrinsic capability parameters, e.g., on a continuous basis, based on information related to the vehicle and the locale context information determined based on, e.g., the current and future location of the vehicle. 
     To facilitate the self-aware capability parameter generator  740  to generate extrinsic capability parameters, the locale context determining unit  730  may retrieve information stored in a map configuration  750  and a road context configuration  760  based on the current location  720  and the destination information  710 . The locale context information related to the roads may include the surrounding or contextual information of the road the vehicle is currently on and/or the roads that the vehicle will be on subsequently. For example, the map configuration  750  may provide information about the roads from the current location to the intended destination, while the road context configuration  760  may provide some known or static information about the characteristics associated with roads, such as altitude, steepness, curviness of each road, etc. Such gathered static information about the roads may then be used by the self-aware capability parameter generator  740 . 
     The road conditions may change over time. For example, roads may become icy or slippery due to changes in weather conditions. Such dynamically changing context information about the roads may be acquired separately by, e.g., the self-aware capability parameter generator  740  on a continuous basis and used in generating extrinsic capability parameters that are reflective of the real time situations. As will be discussed below in reference to  FIG. 8  about the self-aware capability parameter generator  740 , both the current location and the source-destination information may also be sent to the self-aware capability parameter generator  740  in order for it to gather real time information about road conditions to determine the extrinsic capability parameters. 
     To generate intrinsic vehicle capability information, information related to the vehicle may be accessed from a vehicle information storage  750 . The vehicle information storage  750  may store vehicle parameters configured when the vehicle was manufactured such as whether the vehicle is equipped with cruise control or certain types of sensors. The storage  750  may also subsequently update information related to the parameters intrinsic to the vehicle. Such subsequent update may be generated due to, e.g., vehicle maintenance or repair or even update observed in real time. In discussion below in reference to  FIG. 8 , the self-aware capability parameter generator  740  includes also the mechanism to collect continuously any dynamic update of the vehicle related parameters consistent with the actual intrinsic capability of the vehicle. 
       FIG. 8  depicts an exemplary high level system diagram of the self-aware capability parameter generator  740 , according to an embodiment of the present teaching. In this illustrated embodiment, the self-aware capability parameter generator  740  comprises a locale context information processor  810 , a situation parameter determiner  820 , a self-aware capability parameter updater  830 , and various updaters that continuously and dynamically gather information of different aspects related to decision making of the vehicle. Such dynamic information updaters include, e.g., vehicle capability parameter updater  860 - a , weather sensitive parameter updater  860 - b , traffic sensitive parameter updater  860 - c , orientation sensitive parameter updater  860 - d , road sensitive parameter updater  860 - e , . . . , and time sensitive parameter updater  860 - f.    
     In some embodiments of the operation, upon receiving the locale context information from the locale context determining unit  730 , the locale context information processor  810  processes the received information and, e.g., extracts information related to the current route the vehicle is on and sends such information to the self-aware capability parameter updater  830 . Such information related to the current route may include steepness or curviness of the route or other types of static information such as the altitude and orientation of the route. The situation parameter determiner  820  receives the current location  720  and, e.g., separates location and time information and sends the information to the self-aware capability parameter identifier  830  so that it may use that information to identify capability parameters specific to the location and the precise time. 
     With the information about the location of the vehicle and the current time, the self-aware capability parameter updater  830  may access intrinsic capability models  840  and/or extrinsic capability models  850  to retrieve capability related parameter values specific to the current location and time. In some embodiments, the intrinsic capability models  840  may be configured to specify types of parameters relevant to the intrinsic capabilities of the vehicle and the current values thereof. Similarly, the extrinsic capability models  850  may be configured to specify types of parameters that have an impact on the ability of the vehicle to operate and their current values. 
     In operation, in order to keep the values of the parameters current, the intrinsic and extrinsic capability models ( 840  and  850 ) may regularly trigger the updaters ( 860 - a , . . . ,  860 - f ) to gather real time information and update the values of the corresponding parameters based on such gathered real time information. For example, the intrinsic capability models  840  may be configured to have a mechanism to activate the vehicle capability parameter updater  860 - a  to gather updated information related to the intrinsic capabilities of the vehicle. Such a mechanism may specify different modes of triggering. For instance, it may be on a regular schedule, e.g., daily or hourly. It may also specify to be triggered by some external event such as a signal received from a maintenance shop or a sensor in the vehicle that senses that some functional state of a component in the vehicle has been changed. In this case, the vehicle capability parameter updater  860 - a  may accept real time vehicle information from the sensor(s) and update the values/states of the relevant capability parameter in the intrinsic capability models to reflect that real time status of the vehicle. For instance, if during the operation of the vehicle, the headlight or a break may become non-functional. Such information sensed in real time may be gathered by the vehicle capability parameter updater  860 - a  and used to update the information stored in the intrinsic capability parameter storage  840 . Such updated information relates to the vehicle may then be used by the self-aware capability parameters generator  740  to generate intrinsic capability parameters. 
     Similarly, the extrinsic capability models  850  may be configured to specify the update mechanism(s) for updating different types of extrinsic capability parameters. The update mechanism may specify regularly scheduled update or update to be triggered by some events. Different types of extrinsic capability parameters may be configured to be updated based on different triggering mechanisms. For example, for weather related extrinsic capability parameters or extrinsic capability parameters that may be keyed on weather, e.g., visibility in the vicinity of the vehicle, the update may be made regularly, e.g., every few minutes. Similarly, traffic sensitive parameters, e.g., the actual permissible speed which is usually direct result of the traffic condition, may also be updated regularly. Different types of parameters, although all regularly updated, may have different update schedule, which may range from every few seconds to every few minutes or every few hours. 
     On the other hand, some extrinsic capability related parameters may be made upon the occurrence of some events. For instance, for an orientation sensitive parameter, e.g., whether sun glare exists, the update may be triggered when the vehicle is heading in certain directions. If the direction of the vehicle is heading changes from north to north-west at some afternoon time, this may trigger the orientation sensitive parameter updater  860 - d  to gather information related to sun glare and update the situation with regard to sun glare. In some situations, the update may indicate that there is no sub glare, e.g., when it is a cloudy day. In some situations, the update may indicate that there is sun glare. In either situation, such orientation sensitive information is then used to update the value of the corresponding extrinsic capability parameter stored in the extrinsic capability parameter storage  850 . Similarly, update of time sensitive parameters, such as visibility of the vehicle due to time of the day, may be triggered based on detected location, time zone of the location, and the specific time of the day at the moment. In some embodiments, the update of some of the capability parameters may also be triggered by event related to the detected updates of other capability parameter values. For example, the update of road sensitive parameters such as slippery road condition may be triggered when the update for the weather condition indicates that it started to rain or snow. 
     In the illustrated embodiments, the vehicle capability parameter updater  860 - a  receives the static vehicle information from storage  750  and dynamic vehicle information update from real time vehicle information feed which may be from multiple sources. Examples of such sources include dealers, vehicle maintenance places, sensors on the vehicle reporting the status change of components, or other sources. The weather sensitive parameter updater  860 - b  may receive both dynamic weather update and the updates of other weather sensitive capability parameters, e.g., precipitation, visibility, fog, or any other parameters that relate to weather and have the potential to impact the operation of the vehicle. Weather related information may be from multiple data sources that feed real time data. 
     The traffic sensitive parameter updater  860 - c  may receive both dynamic traffic reports and other information relating to the traffic that may influence the operation of the vehicle. Examples include the extent of the traffic jam (which may be used to determine whether the route of the vehicle needs to be re-planned) or the time of the accident that had caused the traffic (to estimate how long the delay will continue in order to determine whether to re-do the route planning). Traffic or traffic related information may be received from one or more sources for real time data feed. The orientation sensitive parameter updater  860 - d  may be configured to gather information along the road in the direction of the vehicle. Such orientation sensitive information may include sun glare in certain directions (e.g., east or west) or any potential situations in the direction of the road the vehicle is on (e.g., landslide situation ahead of the road). Similarly, the road sensitive parameter updater  860 - e  may, once triggered, gather information about various roads or road conditions with respect to the location of the vehicle, from one or more real time information feed sources. Such information may be related to the roads (e.g., open, close, detoured, school zone, etc.) or conditions thereof (e.g., slippery, icy, flooded, construction, etc.). The time sensitive parameter updater  860 - f  may be configured to collect from data source(s) real time data that depend on time. For example, the visibility of the road may depend on the time of day at the zone the vehicle is in. 
     The collected real time data may then be used to update the intrinsic capability models  840  and/or the extrinsic capability models  850 . Such updated data may be time stamped. The self-aware capability parameter updater  830  may then access both the intrinsic and extrinsic capability models  840  and  850  to determine relevant capability parameters and the updated values thereof. The retrieved intrinsic/extrinsic capability parameters may then be output so that they can be used by various planning modules as shown in  FIG. 5 . Specifically such generated self-aware capability parameters  510  are used by the route planning module  550  for route planning, as will be discussed with reference to  FIGS. 10-13 . The self-aware capability parameters are also used by the motion planning module  560  for personalized motion planning, which will be disclosed in detail with reference to  FIGS. 14-19 . The self-aware capability parameters are also used by the lane planning module  570  for lane control, which will be detailed with reference to  FIGS. 20-22 . 
       FIG. 9  is a flowchart of an exemplary process of the self-aware capability parameter generator  740 , according to an embodiment of the present teaching. First, the locale context information is received at  910  and location and time information are extracted at  920  and used at  930  by different updaters to obtain information feeds from different sources related to various aspects of intrinsic and extrinsic capabilities. Such acquired information is then used by different updaters at  940  to update the intrinsic capability parameters  840  and extrinsic capability parameters  850 . Based on the current location, time, and the received locale contextual information, the self-aware capability parameter updater  830  then identifies various intrinsic and extrinsic capability parameters  510  relevant to the vehicle at the present time to update, at  940 , the intrinsic/extrinsic capability parameters and generates, at  950 , the updated capability parameters. Such updated intrinsic/extrinsic capability parameters  510  are then output at  960 . 
     Such dynamically gathered self-aware capability parameters are to be used in various vehicle behavior planning operations, including route planning, motion planning, and lane related vehicle behavior planning. For example, in human driving, choosing a route to a destination is often done with consideration of factors captured by the self-aware capability parameters. For example, a human driver may choose a route to a desired destination based on, e.g., what the vehicle is equipped with or capable of (intrinsic capability parameters). If the vehicle is in such a condition that it cannot handle steep road well, then such road needs to be avoided. In addition, a human driver may also consider other factors such as weather of the day, conditions of the roads considered, events known or scheduled at certain time of day (extrinsic capability parameters). For instance, it one road points to the west and the sun will be setting at that time, perhaps there will be too much glare so that it is better to take another alternative road. For both safety and reliability, autonomous vehicles ought to also consider such intrinsic and extrinsic capabilities with respect to route choices during route planning. 
     Traditional approaches to route planning often adopt some cost function so that the cost of a route selected is minimized. For instance, conventional route planning considers, e.g., optimization of distance traveled, minimization of time required to arrive the destination, or minimize the fuel used to get to the destination. In some instances, conventional approaches may also consider traffic conditions in optimizing the cost, e.g., high traffic route may decrease the speed leading to increased time and fuel to get to the destination. Such optimization functions often assume that all vehicles can handle all routes in the same manner and all routes can be handled equally well. Such assumptions are often not true so that when autonomous vehicles apply such planning schemes, they often find unable to proceed or even become unsafe in some situations. The present teaching aims to achieve safe, realistic, and reliable route planning that is adaptive to the changing intrinsic and extrinsic capability related parameters. 
     As shown in  FIG. 5 , the self-aware capability parameters  510  are considered by the planning module  450  in achieving different planning tasks, including the route planning module  550 , the motion planning module  560 , and the lane planning module  570 . Below, in reference to  FIGS. 10-13 , details about the route planning module  550  are provided.  FIG. 10  depicts an exemplary high level system diagram of the route planning module  550 , according to an embodiment of the present teaching. The purpose of the route planning module  550  is to plan a route based on desired destination in a self-aware manner in terms of both intrinsic and extrinsic capabilities. In contrast, conventional route planning technologies consider mainly criteria such as shortest distance, shortest time, most use of highways/local ways, etc. without taking into account the dynamic intrinsic capability parameters and the real time extrinsic capability parameters. 
     In this illustrated embodiment, the route planning module  550  comprises a route selection preference determiner  1030  and a global route planner  1020 . The route selection preference determiner  1030  is to determine the preferences to be considered in selecting a route. The global route planner  1020  is to select an appropriate route based on a variety of information, including the self-aware capability parameters  150 . In some embodiments, the route planning activities may be triggered based on the route planning trigger signal as shown. Upon being activated, the global route planner  1020  may gather various types of dynamic information relevant to the present route planning operation. For example, the global route planner  1020  may rely on information related to the source/current location and the destination desired. The planning is performed with respect to the source/current location and the destination. The destination information may be determined in different ways. For example, it may optionally be received from a driver/passenger via an interface unit  1010 . 
     The global route planner  1020  may also take real time data  480  as input and plans a route accordingly. As discussed with respect to  FIG. 4B , real-time data include information related to real time vehicle related information (position), information about passenger observed in the vehicle, . . . , and road characteristics. Such real-time data provide surrounding information needed for the route planning. The global route planner  1020  also receives the self-aware capability parameters  510  that inform the planner what is possible given the dynamic intrinsic and extrinsic situations at the time of the planning. For instance, intrinsic capability parameters may indicate that the vehicle is currently not able to run fast due to some mechanical problems so that the global route planner  1020  may take that into account to, e.g., plan a route that involves mainly local roads and may possibly pass some car repair shops. Similarly, extrinsic capability parameters may indicate that in the north direction of the current location of the vehicle, the sun glare is quite severe so that the global route planner may base that information to avoid a nearby route that is in the north direction before the sun is set. The real-time data  480  and the self-aware capability parameters  510  provide information to the global route planner  1020  to enable it to plan a route that is appropriate given, e.g., the present time, the present location of the vehicle, the present weather, the present passenger&#39;s situation, and present road conditions. 
     The global route planner  1020  may also consider preference(s) to be applied in route planning. Such preferences may be specified by a driver/passenger via the user interface unit  1010  (which may be forwarded to the global route planner  1020 ) or may be obtained via other means (see disclosure below with reference to  FIG. 12 ). In considering the preferences to be applied, information stored in route selection preference configuration  1050  may also be accessed and considered. Such route selection preference configuration may specify some general preference in route selection in different scenarios, e.g., avoid steep/curvy roads in raining/snow scenarios, avoid small streets at night, avoid roads with very few gas stations, etc. The global route planner  1020  may forward relevant information received from real-time data  480  and self-aware capability parameters  510  to the route selection preference determiner  1030 , which is in turn used by the route selection determiner  1030  to retrieve certain route selection preference configuration from  1050 . For example, if it is currently snowing (from real-time data  480 ) and the vehicle has no snow tire (from the intrinsic capability parameter  510 ), such dynamic information may be forwarded from the global route planner  1020  to the route selection preference determiner  1030  so that selection preference configuration related to such dynamic scenarios may be retrieved from the route selection preference configuration  1050  (e.g., avoid steep/curvy road) and sends back to the global route planner  1020  so that it can be relied on in selecting an appropriate route. 
     To determine an appropriate route, in addition to knowing the selection preferences, the global route planner  1020  may also need to know additional information about the roads, such as what routes available from the current location of the vehicle to the intended destination. In addition, for each route available, the map/road configuration  1060  may also store characteristic information about each of the available roads/routes. Such characteristic information of the roads/routes may include, but not limited to, geometric characteristics such as nature of the roads/routes (highway or not), dimension of the roads/routes, steepness/curviness, condition of the roads/routes, etc. During planning, the global route planner  1020  may first determine the roads/routes available between the current location of the vehicle to the desired destination. To select an appropriate route to the destination, for such available roads/routes, their characteristic information may also be accessed by the global route planner  1020  so that selections may be made based on such characteristic information. 
     With the information about the available roads/routes available as well as the characteristic information about those available roads/routes, the global route planner  1020  may then select an appropriate route to the destination by matching the route selection preferences, determined by the route selection preference determiner  1030 , with the characteristic information of the available roads/routes. Details about the global route planner  1020  are provided with reference to  FIGS. 12-13 . 
     As discussed previously, the global route planner  1020  selects a planned route based on dynamic information from different sources, including real-time data  480  and self-aware capability parameters  510 . In addition to that, as the vehicle may be on move or the destination may change over time, the current location of the vehicle and the destination may also change in time, just like the real-time data  480  and the self-aware capability parameters  510 . When such information changes, it may impact the global route planned. For example, when the current location changes, the real-time data associated with the current location may also change, e.g., from good weather associated with the previous spot to the raining condition associated with the current location. This may in turn lead to a change in terms of route selection preferences and ultimately, the route selected. Thus, the global route planner  1020  may interact with the route selection preference determiner  1030  in a bi-directional manner and dynamic manner. Whenever there is a change that may warrant a re-determination of route selection preferences, the global route planner  1020  may then activate the route selection preference determiner  1030  to modify or re-generate the preferences to be used by the global route planner  1020  to determine an appropriate route given the situation. 
       FIG. 11  is a flowchart of an exemplary process for the route planning module  550 , according to an embodiment of the present teaching. Information about the destination of the vehicle and optionally about the preferences is received at  1110 . Real-time data  480  as well as self-aware capability parameters  510  are received by the global route planner  1020  at  1120  and various information related to the current scenario or situation of the vehicle may then be identified, at  1130 , from the received real-time data and the self-aware capability parameters. Based on the relevant information related to the current scenario, preferences specific to the current scenario are determined, at  1140 . To plan a route, the global route planner  1020  accesses, at  1150 , information about available roads/routes with respect to the current location and the desired destination as well as the characteristic information of such available roads/routes. At  1160 , based on the specific preferences determined based on the current scenario as well as the roads/routes information, the global route planner  1020  selects a route appropriate for the current situation. 
       FIG. 12  depicts an exemplary high level system diagram of the global route planner  1020 , according to an embodiment of the present teaching. In this illustrated embodiment, the global route planner  1020  comprises a self-aware capability parameter analyzer  1205 , an intrinsic capability based filter generator  1210 , and a route selection engine  1230 . Optionally, the global route planner  1020  also comprises a destination updater  1225  for dynamically determine and update the current destination. In the illustrated embodiment, the global route planner  1020  also optionally include a mechanism for personalizing preferences of a driver/passenger so that the route in selecting a route. The route selection preference determiner  1030  is to determine the preferences related to selecting a route based on the specific situation the vehicle is currently in, which differs from obtaining personalized preferences directed to a specific driver/passenger. 
     Illustratively, the optional mechanism to determine personalized preferences comprises a passenger driving data analyzer  1245 , a preference personalization module  1250 , and a passenger preference determiner  1240 . In operation, the passenger driving data analyzer  1245  receives recorded human driving data  430  as input and analyzes or learns from such data to understand route preferences relevant to specific drivers/passengers. For example, from the recorded human driving data  430 , it may be learned that a specific driver prefers to drive on local roads instead of highway or historically chose to use highways at nights even though it involved much longer distance. It may also learn preferences of all drivers associated with the vehicle. For instance, multiple people (husband, wife, and a child of a household) may be associated with the vehicle, i.e., anyone of these people may operate the vehicle. The passenger driving data analyzer  1245  may learn from the recorded human driving data  430  various types of information associated with the driving behavior of such drivers, which may enable the preference personalization module  1250 , upon receiving such driving behavior information, to establish personal preferences of each of such individuals. 
     Upon receiving the information related to each individual driver from the passenger driving data analyzer  1245 , the preference personalization module  1250  may then generate personalized preferences in terms of route choices. Such route related preferences may reflect not only route choices but also represent preferences of route choices in different situations such as specific time frames of a day, seasons, locations, etc. Such established preferences for each individual driver may then be stored in storage  1265 . At the time of route planning, the passenger preference determiner  1240  receives the real-time data  480  and based on various types of information in the real-time data  480  (e.g., month/day/time, passenger information, locale weather, etc.), the passenger preference determiner  1240  may access, from the route selection preference storage  1265 , relevant preferences that can be applied in current route planning. For example, if the real-time data indicate that the driver is a specific person and the time is currently 7:45 pm in January, etc., the passenger preference determiner  1240  may identify personalized route preferences in  1265  related to the current specific driver which are associated with the specific time frame and season of the year (e.g., a driver may prefer driving on highway in winter season). The personalized route selection preferences so identified may then be sent to the route selection engine  1230  so that personalized preferences of the driver/passenger at the time of the route planning can be considered in determining what route is to be selected. 
     As shown in  FIG. 12 , the route selection engine  1230  may also take the preferences estimated by the route selection preference determiner  1030  as input and use that in its route selection operation. In some embodiments, the route selection engine  1230  may rely on the preferences from  1030  without considering the personalized preferences of a driver, i.e., it may rely on merely the preferences identified by the route selection preference determiner  1030  in its route selection. 
     In selecting route appropriate for the current situation, the route selection engine  1230  may also receiving self-aware capability parameters  510 . In the illustrated embodiment, the self-aware capability parameter analyzer  1205  separates extrinsic capability parameters and intrinsic capability parameters and sends the extrinsic capability parameters to the route selection engine  1230  so that extrinsic conditions associated with the current situation the vehicle is in can be considered in selecting a route. For example, the extrinsic capability parameters may indicate that there is on-going construction on Route 7, the route selection engine  1230  may consider that and avoid Route 7. However, if the destination is currently set for a school on Route 7 and the driver&#39;s habit is to pick up children from the school each day at the present time (e.g., 3:30 pm), the route selection engine  1230  may elect to choose Route 7, given all things considered. 
     Similarly, intrinsic capability parameters may also be considered in selecting an appropriate route. In this illustrated embodiment, the intrinsic capability parameters are fed to the intrinsic capability based filter generator  1210 , which may create different filters  1215  based on the intrinsic capability parameters so that such filters may be utilized by the route selection engine to filter out routes that are not appropriate given the intrinsic capability parameters. For example, if the intrinsic capability parameters indicate that the vehicle has no snow tire, any routes that are steep and/or curvy may not be appropriate on a snowy day. 
     The route selection engine  1230  selects a route based on both the current location of the vehicle, tracked by a current location updater  1235 , and a destination, tracked by a destination updater  1225 . Depending on the situation, changed current location and destination may trigger the route selection engine  1230  to activate the route selection preference determiner  1030  to re-evaluate the preferences in route selection given the changed circumstance. 
       FIG. 13  is a flowchart of an exemplary process for the global route planner  1020 , according to an embodiment of the present teaching. At  1310 , self-aware capability parameters are received. Intrinsic capability parameters are used to generate, at  1320 , intrinsic capability based filter(s) so that certain routes can be filtered out as inappropriate given the intrinsic conditions of the vehicle. Extrinsic capability parameters are extracted, at  1330 , from the received self-aware capability parameters. At the same time, real-time data  480  are continuously received at  1340  and recorded human driving data are received at  1350 . Such data are then used to determine, at  1360 , personalized route selection preferences relevant to the current driver, current situation, and current time. At  1370 , self-aware capability parameters and/or the personalized preferences of the driver may then be used to select a route appropriate given all factors considered. At  1380 , the selected route is output. 
     The route planning according to the present teaching allows various types of information, such as real-time data and self-aware capability parameters, to be taken into account in route planning so that the planned routes are adaptive with respect to the vehicle condition at the time (via intrinsic capability parameters), the dynamic environment the vehicle is in at the time (via real-time data as well as extrinsic capability parameters), the passenger characteristics determined based on, e.g., dynamically updated real-time data (see  FIG. 4B ), as well as passenger personalized preferences. Similarly, such information may also be utilized in other types of planning operations so that the planned vehicle activities are adapt to the real-time situation, personalized based on individual preferences, and allows the vehicle to behave more human driver like. Below, more details are provided, with respect to  FIGS. 14-19  on personalized adaptive motion planning. 
     Human drivers control their vehicle motion in a manner that is comfortable. In most situations, human drivers also pay attention to the feedback or reaction of passengers who ride with them in the vehicle and respond to the vehicle motion. For example, some human drivers may prefer start and stop the vehicle smoothly. Some human drivers who usually start and stop the vehicle fairly abruptly may adjust their driving when they observe that passengers sitting in their vehicle respond in a certain way. Such human behavior may play an important role in autonomous vehicles. It is commonly recognized that driving behavior changes from person to person and how such behavior is to be adjusted in the presence of others in the same vehicle may also differ from person to person. 
     Traditionally, autonomous vehicles may adopt planning models that are trained to capture the characteristics of human driving behavior of the general population. Such generalized models do not customize the planning approach based on individual driver/passenger preferences or intents. The present teaching aims to provide personalized motion planning based on knowledge of the driver/passenger as well as the dynamic observations of driver/passenger response to vehicle motions. 
       FIG. 14A  depicts an exemplary high level system diagram of the motion planning module  560 , according to an embodiment of the present teaching. In this illustrated embodiment, the motion planning module  560  aims at personalized, human-like, and adaptive motion planning, i.e., motions of the vehicle are planned in accordance with, e.g., general and personal likings, which may include what is known to be the preferences of a passenger and what is the reaction or feedback of a of a passenger of the current motion of the vehicle. The motion planning module  560  according to the present teaching may comprise a generic motion planner  1450  and a passenger motion adapter  1460 . The motion planning module  560  may plan vehicle motion based on various considerations, including real-time situations the vehicle is in (e.g., on a curvy road, raining day, dim lighting, etc.), vehicle conditions (via intrinsic capability parameters), and personal preferences of the passenger in the vehicle (known preferences or dynamically determined based on driver feedback observed). Given those considerations, vehicle motion may be planned based on motion planning models, which may be invoked in a manner suitable for different scenarios. Motion planning models may include different models appropriate for the given situation in hand. 
       FIG. 14B  illustrates exemplary types of motion planning models, according to an embodiment of the present teaching. In the illustrated embodiment, motion planning models may include generic motion planning models ( 1450  in  FIG. 14A ), sub-category models ( 1480  in  FIG. 14A ), or personalized models ( 1430  in  FIG. 14A ). The generic motion planner  1450  may be preference based models or impact based models (see  FIG. 14B ). A preference based model may be provided specifying preferred vehicle motion in different scenarios based on general knowledge about vehicle operation. For instance, when roads are slippery or icy, it is preferred to plan motions that are slower without sharp turns. An impact based model may specify which kind of motion may cause what type of impact and such specification may be used to guide motion planning to achieve or avoid certain impact. 
     A sub-category model for motion planning may be, as compared with the generic models, directed to a sub-category of vehicles or a sub-category of drivers/passengers. For example, a sub-category model may be directed to sports cars and another sub-category model may be provided for vans. In addition, a sub-category model may be directed to teenager drivers and another sub-category model may be directed to senior citizens. Each sub-category model is tuned or specialized so that the motion planning for a matching sub-category can be performed more accurately. According to the present teaching, motion planning models may also include personalized models which may include individual models, each of which may specify preferences of each individual in terms of vehicle motions. For example, a passenger&#39;s individual preference model may specify that the passenger prefers smooth vehicle motion and another passenger&#39;s individual preference model may specify some different preferences. Such generic, sub-category and individual models for motion planning may be derived based on recorded human driving data so that the motion planned based on such models are more human-like. 
     With reference back to  FIG. 14A , in operation, to achieve personalized, human-like, and adaptive motion planning, the motion planning module  560  receives various types of information and utilizes different motion planning models. The received information includes planned route  520  from the route planning module, surround information  420 , real-time data  480 , and self-aware capability parameters  510 . Based on the real-time data  480  and the self-aware capability parameters  510 , the generic motion planner  1450  determines the situation the vehicle is in (e.g., raining, dark, etc.) and accordingly invokes appropriate generic motion planning models in  1440  to retrieve general motion planning information. In some embodiment, the generic motion planner  1450  may also determine relevant sub-category of the vehicle and/or passenger so that associated sub-category motion planning models may be retrieved from  1480  and utilized for motion planning. The generic motion planning models  1440  may specify general motion planning strategies, e.g., if it is a snowy day or the vehicle is on a curvy road, it is preferred to make the vehicle motion slower and steady. Each sub-category model may be provided to specify the generic motion planning strategies for the sub-category (e.g., a type of cars such as sports car or a sub-group of passengers such as senior citizens). 
     The motion planned by the generic motion planner  1450  (based on generic motion planning models and/or sub-category motion planning models) may be further adjusted or adapted according to personalized preferences. In the illustrated embodiment, this is achieved by the passenger motion adapter  1460 . There may be different ways to adapt the motion planned to meet personalized preferences. In some embodiments, personalized preferences may be accessed from individual passenger models  1430 . If the identity of the passenger is known, the associated individual passenger model for the passenger may be retrieved from  1430  and the specified preferences in vehicle motion may be used to determine how to achieve personalized motion planning. For instance, an individual model for a particular passenger may indicate that the passenger prefers a smooth ride without taking risks. 
     Another way to achieve personalized motion planning is adaptively adjusting motion planning based on dynamically observed information. As discussed previously with reference to  FIG. 4B , real-time data  480  includes information related to passenger characteristics, which can be passenger condition, . . . , and/or passenger reaction cues. Passenger condition may refer to mental, physical, and functional state of the passenger. Information to be used in personalized motion planning may also include other types of data collected related to the situation. A passenger observation analyzer  1420  may collect various types of information and extract relevant indications and then send such indications to the passenger motion adapter  1460  so that such dynamic and personalized information can be taken into account in motion planning. Details about the passenger observation analyzer  1420  are provided with reference to  FIGS. 15-16 . 
       FIG. 14C  illustrates exemplary types of observations collected for consideration in motion planning, according to an embodiment of the present teaching. Observations may include explicit expression from, e.g., the passenger such as voice or text input which may explicitly indicate what the passenger desires. For instance, the passenger may shout “faster!” or “I am going to be late!” or “I really get there on time.” Such explicit expression detected may be relied on in motion planning. The observations may also include detected scenarios that may indicate something in terms of motion planning. Scenario information may include event involved (which may indicate the urgency of the passenger is faced with), the current state of the passenger (such as age, known health conditions, etc.), time of day which may imply certain task the passenger has at that time (e.g., pick up children from school) which requires certain safety level. Observations may also include observed physical reactions of the passenger which may be considered as relevant for motion planning. For instance, sensors inside the vehicle may capture any data that may indicate the emotion of the passenger, the body language of the passenger, or the tone in passenger&#39;s voice, all of which may reflect the desire of the passenger in response to the current vehicle motion. For instance, the passenger may appear to be uneasy or even show anxiety, which may indicate that the vehicle motion is too rough for the passenger. A sharp tone in passenger&#39;s voice may indicate the same. Certain physical behavior may also suggest certain reaction from the passenger to the vehicle motion. For example, if the passenger is napping, yawning, looking sleepy, or reading, it may indicate that the passenger is comfortable with the vehicle motion. On the other hand, if it is observed that the passenger keeps checking watch, it may indicate that the passenger feels the vehicle is moving too slow. 
     According to the present teaching, in addition to personalized motion planning (e.g., not only with respect to sub-categories but also with respect to individuals), motion planning may also be adaptive to the current situation characterized by, e.g., self-aware capability parameters and real-time situations such as weather, road conditions, etc. The passenger motion adapter  1460  receives the extrinsic capability parameters from  1410  and plans motion accordingly. For example, if extrinsic capability parameters indicate that there is sun glare or foggy, motion may be planned accordingly (e.g., slow down). 
       FIG. 15  depicts an exemplary high level system diagram of the passenger observation analyzer  1420 , according to an embodiment of the present teaching. In this illustrated embodiment, the passenger observation analyzer  1420  is provided for obtaining the dynamic preferences of the passenger in terms of vehicle motion so that the motion planning can be adapted to the personal likings. The dynamic preferences of the passenger are derived based on analysis of the passenger&#39;s reaction cues to the current vehicle motion, observed via different sensors. The exemplary passenger observation analyzer  1420  comprises a sensor activator  1500 , a plurality of in-situ sensors  1510 , a passenger detector  1520 , a passenger feature detector  1540 , a visual-based reaction cue estimator  1580 , an acoustic-based reaction cue estimator  1590 , a passenger expression detector  1560 , a passenger scenario detector  1570 , and a user reaction generator  1595 . 
     The passenger observation analyzer  1420  is provided to determine the reaction or feedback of the passenger to the current vehicle motion to determine whether the vehicle motion needs to be adjusted. For example, if passenger reaction indicates that the passenger is not happy about the current vehicle motion, an adjustment may be made in motion planning accordingly. The passenger reaction is to be estimated based on different cues, including visual, acoustic, text, or contextual scenarios. 
     In some embodiments, the sensor activator  1500  activates the in-situ sensors  1510  to detect the passenger reaction. The in-situ sensors  1510  comprise a plurality of sensors including visual sensors, acoustic sensors, infrared sensors, . . . , or communication sensors and the like that enable the detection of any expression of the passenger. For instance, the visual sensors included in the in-situ sensors  1510  may comprise a plurality of spatially distributed (within the vehicle) camera devices that are capable of capturing, processing and fusing images of a scene from a variety of viewpoints into some form more useful individual images/videos. For example, the visual sensors may capture a gesture or facial expression of the passenger, which may be used to estimate the reaction of the passenger. The in-situ sensors may be selectively activated. For instance, at night, in order to observe accurately the passenger&#39;s reaction, visual sensor may not work well and in this situation, infrared sensors may be activated instead. 
     As depicted in  FIG. 14C , various physical reactions may be observed and used to analyze passenger&#39;s reaction cues. The passenger detector  1520  receives sensor data and detects the passenger based on, passenger detection models  1530 . The detection may be based on either visual or acoustic information. Thus, the passenger detection models  1530  may include both visual and acoustic models associated with the passenger and can be either individually invoked to detect the passenger based on a single modal data or both invoked to detect the passenger based on both visual and acoustic features. For instance, the passenger detection models  1530  may include a face recognition model which can be used to detect the passenger based on video or pictorial data from one or more visual sensors. The passenger detection models  1530  may also include a speaker based passenger detection model by which the passenger may be recognized based on his/her voice. 
     Upon the detection of the passenger, the sensor data may be continuously fed to the passenger feature detector  1540  to detect various passenger behavior features, which may include both visual and acoustic. For instance, certain body language may be detected that may reveal that the passenger is doing certain things, such as sleeping (napping), reading, yawning, or frequently checking his/her watch. Such detected passenger features may also include acoustic features. For instance, the passenger feature detector  1540  may detect that the passenger is saying “slow down.” Visual and acoustic cues may be simultaneously detected that reveal consistent reaction cues. For example, the passenger may constantly check the watch and say “faster!” 
     The passenger features may be detected based on visual and acoustic feature detection models  1550 . Such models may guide the passenger feature detector  1540  in terms of what feature to detect and provide, for each feature to be detected, a corresponding model that can be used to detect the feature. Those models may be personalized in the sense that what is to be detected may depend on the passenger. For instance, if the passenger is known to be mute, there is no reason to detect acoustic features associated with the passenger. Those feature detection models may be adaptive so that once they are trained and deployed on the vehicle, they may be configured to receive scheduled or dynamic update so that the models are adaptive to the changing situations. 
     The detected passenger visual features are then sent to the visual-based reaction cue estimator  1580 , which may then estimate the passenger&#39;s reaction cues based on such visual cues. For example, if it is detected that the passenger is checking on the watch, the visual based reaction estimator  1580  may the reaction cue may be that the passenger is not happy with the speed of the vehicle and becomes impatient. Such an estimated cue may also be derived based on, e.g., a personalized visual feature model in  1550 , which may be used to determine whether such a behavior (checking watch) is indicative of certain reaction cue associated with this particular passenger (may or may not be depending on the person). 
     Similarly, the detected passenger acoustic features are sent to the acoustic based reaction cue estimator  1590 , which may then estimate the passenger&#39;s reaction cue based on such acoustic features. For example, if it is detected that the passenger is snoring, the acoustic based reaction cue estimator  1590  may estimate that the passenger is comfortable with or at least not unhappy with the current vehicle motion. Such an estimated cue may also be derived based on, e.g., a personalized acoustic feature model in  1550 , which may be used to determine whether such a behavior is indicative of certain reaction cue of this particular passenger. 
     To estimate the reaction of the passenger, the visual based and acoustic based reaction cue estimators  1580  and  1590  may be engaged in estimating the emotional state of the passenger. For example, from the body language observed from the passenger (e.g., restless or seems to be vomiting), it may indicate that the passenger feels uneasy which may be a cue of his/her reaction to the vehicle motion. In addition, the tone of the voice used when the passenger says “faster” or “slow down” may also be used to estimate a level of anxiety of the passenger which is a clue as to how unhappy the passenger is. Such estimated emotional state may be used in evaluating the severity of the reaction that the passenger exhibits in response to the current vehicle motion and may be used to guide whether and/or how to adjust the motion planning. 
     In addition to the observed physical features, other parameters may also be used to estimate whether the current vehicle motion is acceptable. For instance, the source of observation may be an input directly entered by the passenger via some communication interface (e.g., a touch screen display) within the vehicle. The passenger may input via the display interface in the vehicle that he/she wants the vehicle motion to be smoother. This may be detected by the passenger expression detector  1560  via different communication sensors, which could be textual or acoustic. 
     As discussed previously, the scenarios that the passenger is currently in may also influence how the motion should be planned. The passenger scenario detector  1570  is configured to detect any scenario parameters that may be relevant to motion planning. For instance, if it is known that each afternoon between 3:30 pm and 4:30 pm (time of day), the vehicle is used to pick up children from school (task in hand), this may place a restriction on motion planning. That is, the motion planned may need to be based on safety. Once detected, such a restriction may be configured to trump the estimated desire (of the passenger) to be faster in order to ensure safety of the children. Other scenario related factors may also be observed, e.g., the health and age of the passenger. If it is observed (from a passenger model  1535 ) that the passenger is an elderly and suffers from dementia, such scenario parameters may be used to void some detected desire of the passenger. For instance, if the current vehicle motion is already pretty fast and the passenger keeps demanding to be even faster, given the age and known health condition of the passenger, the motion planning module may use such information to make an appropriate motion planning decision. 
     The various passenger reaction cues, detected by the visual/acoustic based reaction cue estimators  1580  and  1590 , the passenger expression detector  1560 , and the passenger scenario detector  1570 , are then send to the user reaction generator  1595 , where the detected different parameters are selected and integrated to generate estimated user reaction and sent to the passenger motion adapter  1460  so that the motion planned by the generic motion planner  1450  may be adapted in accordance with the observed dynamic user reaction to the current vehicle motion. 
       FIG. 16  is a flowchart of an exemplary process for the passenger observation analyzer  1420 , according to an embodiment of the present teaching. To gather observations associated with the passenger, appropriate sensors are activated at  1610 . Information from activated sensors is processed at  1620 . To ascertain the physical behavior of the passenger, the passenger is detected, at  1630 , based on passenger detection models  1530 . Once the identity of the passenger is ascertained, different types of features associated with the passenger may be obtained. At  1640 , any explicit expression from the passenger is detected. The scenario parameters associated with the passenger are detected at  1660 . Such gathered explicit expression from and scenario parameters related to the passengers are then sent to the user reaction generator  1595 . 
     Visual/acoustic features of the passenger are detected at  1650  and are used to estimate, at  1670  and  1680  respectively, the visual and acoustic reaction cues, which are then sent also to the passenger reaction generator  1595 . Different types of information so collected (from  1640 ,  1660 ,  1670 , and  1680 ) are then all used by the passenger reaction generator  1595  to generate, at  1690 , the estimated user reaction. 
     Going back to  FIG. 14 , the estimated user reaction output by the passenger observation analyzer  1420  is sent to the passenger motion adapter  1460  so that the real-time passenger reaction to the current vehicle motion may be considered in determining how to adapt the planned motion based on the dynamic feedback from the passenger. 
       FIG. 17  is a flowchart of an exemplary process for the motion planning module  560 , according to an embodiment of the present teaching. At  1710  and  1720 , real-time data  480  and the self-aware capability parameters are received, respectively. The self-aware capability parameters are processed, at  1730 , and split into intrinsic and extrinsic capability parameters. Based on real time data and the intrinsic capability parameters, the generic motion planner  1450  generates, at  1740 , planned motion for the vehicle. Such planned motion is generated based on generic motion planning models  1440  as well as any applicable sub-category motion planning models  1480 . 
     To personalize the motion planning, the generically planned motion may then be adapted based on personalized information, which may include both known personal preferences and dynamically observed passenger reaction to the current vehicle motion. To achieve that, know passenger preferences are identified, at  1750 , based on the individual passenger models  1430 . In addition, dynamic passenger reaction/feedback is estimated, at  1760 , based on information gathered from different sources/sensors. The personal preferences, either already known or estimated dynamically, are then used to personalize the motion planned, e.g., by adapting, at  1770 , the motion planned based on generic information. Such personalized planned motion is then output, at  1780 , as the planned motion  530 . 
     As discussed with respect to  FIG. 14A , various models are used in motion planning, some being generic, some being semi-generic (sub-category models are semi-generic), and some being personalized. In addition to being personalized and adaptive, the motion planning scheme as disclosed herein also aims at behaving in a manner that is more human-like. Being adaptive to the dynamic reaction of the passenger may be part of it. In some embodiments, the models used by the motion planning module  560  may also be generated to capture human-like behavior so that when they are applied in motion planning, the planned motion  530  will be more human-like. 
       FIG. 18  depicts an exemplary high level system diagram of a motion planning model training mechanism  1800  for generating those models, according to an embodiment of the present teaching. In this illustrated embodiment, the motion planning model training mechanism (MPMTM)  1800  comprises data pre-process portion and a model training portion. The data pre-processing portion comprises a sub-category training data classifier  1820 , an individual training data extractor  1830 , and an observation segmenter  1850 . The model training portion comprises a model training engine  1810  and an independent impact model training engine  1840 , and an independent impact model training engine  1840 . 
     Recorded human driving data  430  are utilized to train models so that the models can capture characteristics related to motion planning that are more human-like. To train the generic motion planning models  1440 , the received recorded human driving data are sent to the model training engine  1810  and the trained models are saved as the generic motion planning models  1440 . To obtain sub-category motion planning models  1480 , the recorded human driving data  430  are classified by the sub-category training data segmenter  1820  into training data sets for the sub-categories and then fed to the model training engine  1810  for training. For each sub-category model, appropriate sub-category training data set is applied to derive the corresponding sub-category model and such trained sub-category models are then saved in  1480 . Similarly, for obtaining individual passenger models  1430  for motion planning, recorded human driving data may be processed to generate different training sets by the individual training data extractor  1830 , each for an individual, and used by the model training engine  1810  to derive individual passenger models that characterize the preferences of the corresponding individuals. 
     In addition to the individual preferences, the individual passenger models  1430  may also include models that characterize impact of vehicle motions on individual passengers observed from the reaction or feedback of passengers. The observed reaction/feedback may be positive or negative and can be used to influence how the motion should be planned in the future for passengers.  FIG. 19  illustrates different types of reactions observed and their roles in model training, according to an embodiment of the present teaching. For example, passenger reaction/feedback that can be used to train impact based models may include negative or position impact. Negative reaction (negative reinforcement) of a passenger to certain planned motion may be captured in a model so that similar motion may be avoided in the future as to this particular passenger. Similarly, positive reaction to a planned motion or positive reinforcement observed may also be captured in the model for future motion planning. Some reaction may be neutral which may also be captured by the individual passenger models. 
     To obtain impact based models for individuals, the real-time data  480 , which capture the passenger characteristics in terms of their behavioral, visual, acoustic cues as well as their conditions (including mental, physical and functional states during vehicle movement), may be segmented based on individuals and such segmented data may then be used to derive models that characterize how certain motions impact passengers. In some embodiments, the mechanism  1800  includes an observation classifier  1850  that segments the real-time data  480  according to individual passengers and fed such segmented training data sets to the independent impact model training engine  1840  to derive individual impact models. Such derived individual impact models are then stored as part of the individual passenger models  1430 . 
     Referring back to  FIG. 5 , the planning module  440  also includes lane planning module  570 , which may be directed to lane following and lane changing as illustrated in  FIG. 20A . Lane following may refer to the behavior to stay in the lane while the vehicle is moving. Lane changing may refer to the behavior to move from one lane the vehicle is currently in to an adjacent lane while the vehicle is moving. Lane planning may refer to the planning of vehicle behavior in terms of either lane following or lane changing. 
       FIG. 20B  illustrates exemplary types of behavior related to lane following, according to an embodiment of the present teaching. As illustrated, there are multiple lanes ( 2010 ,  2020 , and  2030 ) and a vehicle in each lane may follow its lane. Depending on different situations, the lane following behavior of individual vehicles may differ. For instance, as shown, when a vehicle merely tries to stay in a lane without turning, the vehicle may behave to stay in the middle of the lane. This is shown in  FIG. 20B  with respect to the vehicles in lane  2010  and  2020 . This may be referred to as normal behavior  2040 . When a vehicle in lane  2030  needs to turn to the right, e.g., as shown  2050  in  FIG. 20B , the vehicle in lane  2030  may behave differently. For example, instead of remain in the middle of lane  2030 , vehicle in that lane may pull to the right side of the lane before the turn so that the turn is, e.g., safer and easier. Similarly, when a vehicle is turning left, the lane following behavior may differ as well. The lane planning module  570  is configured to capture, e.g., via modeling, lane following behavior in different situations so that the autonomous driving vehicle may be controlled in a natural and human-like way. 
     On the other hand, lane changing may involve behavior of the vehicle when it moves from one lane to an adjacent lane while the vehicle is moving. Different passengers may exhibit different lane changing behaviors. From safety considerations, there may be desirable lane changing behaviors for different situations. Lane planning in terms of lane changing is to plan the vehicle movement with respect to the lanes in a manner that is safe, natural, human-like, and personalized. 
       FIG. 20C  illustrates exemplary types of behavior related to lane changing, according to an embodiment of the present teaching. Illustrated are different lane changing behavior, i.e., changing from a current lane  2020  to the lane left to it (lane  2010 ) and changing from the current lane  2020  to the lane right to it (lane  2030 ). With respect to the lane changing from lane  2020  to lane  2010 , different lane changing behaviors may be characterized in terms of (1) how fast to make the change and (2) in what manner the vehicle is to move to the next lane. For instance, as shown in  FIG. 20B , there are illustrated three plans to move to the lane  2010 , which are left lane changing behavior  1   2060 , left lane changing behavior  2   2070 , and left lane changing behavior  3   2080 , representing respectively different speeds to move to lane  2010 . By behavior  2060 , the vehicle is to move to lane  2010  fastest. By behavior  2080 , the vehicle is to move to lane  2010  slowest. The speed to move to lane  2010  by behavior  2070  is in the middle. Similarly, when a vehicle is to move from lane  2020  to the lane to its right ( 2030 ), there may also be different lane changing behavior, e.g., right lane changing behavior  1   2065 , right lane changing behavior  2   2075 , and right lane changing behavior  2085 , as shown in  FIG. 20B . 
     In addition to the speed by which the vehicle is to move to the next lane, the lane changing behavior may also differ in terms of how the vehicle moves into the next lane. Also as shown in  FIG. 20B , when the vehicle is to move from lane  2020  to lane  2010  by employing left lane changing behavior  1   2060 , there are different behaviors for the vehicle to adopt to ease into lane  2010 , e.g., by following a straight line  2061 , by following curve  2062  (cut in first and then straight out the vehicle), or by following curve  2063  (ease towards the edge of lane  2020  first and watch and then cut in when ready). So, with regard to lane changing, decisions as to vehicle behavior may be made at different levels. 
     Different drivers/passengers may exhibit different lane planning (include both lane following and lane changing) behaviors and in some situations, the same driver/passenger may behave differently under different circumstances. For instance, if there is no one on the street, a driver may decide to cut into the next lane quickly in lane change. When the street is crowded, the same driver may be more careful and decide to take time to gradually ease into the next lane. The lane planning module  570  is configured to learn different human behaviors in different circumstances and use such learned knowledge/models to achieve lane planning in autonomous driving. 
     Smooth and predictable lane following and lane changing behavior is a key aspect of offering human-like driving experience in autonomous vehicles. It may be especially challenging when significant environment noise is present in camera images and/or videos captured during the vehicle operation. Traditional approaches rely on computer vision to detect lanes by detecting drivable area on the fly. Some uses end to end image raw pixels for vehicle control signal prediction. Such conventional approaches fail to utilize the available manual driving data collected so that they usually produce rigid planning and control and are susceptible to environment variances, while ultimately limit the capability to operate the vehicle satisfactorily. 
     The present teaching utilizes lane detection models and lane planning models for lane planning and control. Both models are trained based on large amount of training data, some labeled and some are as collected. For lane detection, lane detection models are obtained using training data with labeled lanes to derive supervised models for lane detection. Such supervised models are to be trained using a large set of training data covering a wide range of environmental conditions to ensure the representativeness and robustness of the trained models. 
     For lane planning, to achieve human-like lane planning behavior, large volume of human driving data are collected and used to train lane control models that, when used for lane planning, are to exhibit human-like behavior in maneuvering the vehicles. Although the lane detection models and the lane planning models are trained separately, in operation, the two sets of models are used in a cascade manner for inference in order to produce robust behavior in diverse types of environment or conditions with human-like operational behavior. In some embodiments, when human driving data are classified according to individual drivers, the present teaching can be configured to further personalize to create personalized human-like lane planning models. With such personalized human-like lane planning models, an autonomous driving vehicle may be operated during lane planning/control in an adaptive manner, depending on who is the passenger in the vehicle. 
       FIG. 21  depicts an exemplary high level system diagram of the lane planning module  570 , according to an embodiment of the present teaching. In this illustrated embodiment, the lane planning module  570  comprises two model training engines  2110  and  2140  for training lane detection models  2120  and lane planning models  2150 , respectively. Such trained models are then used, in lane planning, in a cascaded manner by a driving lane detector  2130  and a driving lane planning unit  2160 . As discussed above, the lane detection models  2120  are supervised models and are trained using training data with labeled lanes. Such supervised training data are processed and used by the driving lane detection model training engine  2110  to obtain the driving lane detection models  2120 . 
     In some embodiments, the lane detection models  2120  may correspond to a generic model, capturing the characteristics of lane detection in different situations. In some embodiments, the lane detection model  2120  may include different models, each of which may be for providing a model to detect lanes in a specific distinct situation. For example, some model(s) may be for detecting lanes in normal road conditions, some may be for detecting lanes when the road is wet, some may be for detecting lanes when the road has glare or reflection, some may even be for estimating lanes when the roads are covered with, e.g., snow or other types of visual obstructing objects. The lane detection models may also provide separate models for different types of vehicle. For example, some vehicles have higher gravity so that cameras capturing the ground image in front of the vehicle may be installed at higher positions relative to the ground. In this case, the lane detection models for such vehicles may be different from the lane detection models for vehicles with cameras installed at a level closer to the ground level. Each type of model may be trained using appropriate labeled training data that are related to the corresponding scenario. 
     To achieve human-like lane planning behavior in autonomous driving, the driving lane planning model training engine  2140  takes recorded human driving data  430  as input and learns human-like behavior in terms of lane planning. As discussed above, such human driving data may be collected from a wide range of drivers/situations/conditions in order for the driving lane planning model training engine  2140  to learn and capture the characteristics of a wide range of human driving behavior in lane planning/control. In some embodiments, the driving lane planning model training engine  2140  may optionally take some supervised training data with labeled lanes as input, e.g., as seeds or some small set of data to drive the learning towards convergence more quickly. 
     Based on the recorded human driving data  430 , the driving lane planning model training engine  2140  may learn and/or train models for both lane following and lane changing. In some embodiments, for each of lane following and lane changing, a generic model in  2150  for generic human behavior may be derived. In some embodiments, the lane planning model training engine  2140  may also learn and/or train multiple models for lane planning, each of which may be for different known situations, e.g., lane following or lane changing for specific subgroups of the general population, or for particular different driving environment scenarios (wet road, dark light, crowded road). Such models for subgroups of the general population may also be stored in  2150 . 
     The human-like lane control models  2150  may also be personalized and stored in  2150 . When multiple models are to be derived via training, lane human driving data that meet the condition associated with each of different model may be extracted and used to train the models. For example, lane planning (including lane following and lane changing) models for lane related behavior exhibited when driving on crowded roads may be learned based on human driving data related to lane driving behavior on crowded roads. The models for lane planning may also be personalized. To achieve personalization, the driving lane planning model training engine  2140  may derive a model for each individual passenger (e.g., with respect to each of lane following and lane changing) based on the passenger&#39;s past driving data. Optionally, information from a personal profile associated with the passenger may also be used during learning in order to obtain a model that is more accurately reflect the preferences of the passenger. 
     Such obtained different types of lane planning/control models may then be stored in the driving lane control model storage  2150 . In some embodiments, different models for different situations may be organized and indexed for easy identification and quick access in real time during the operation of the vehicle. In some embodiments, the driving lane detection model training engine  2110  and the driving lane planning model training engine  2140  may reside remotely from the vehicle and the learning may be performed in a centralized manner, i.e., they may be operating based on training data from different sources and the learning and update may be activated regularly. The trained models may be sent to distributed vehicles. In some embodiments, personalized models for lane planning may be updated locally in each vehicle based on data acquired locally. 
     The training via both  2110  and  2140  engines may be achieved via any learning mechanism including artificial neural networks, deep leaning networks, etc. Depending on the types and number of models to be obtained, each training engine may comprise a plurality of sub training engines, each for a specific (set of) models for some specific purposes and each may be configured and implemented differently in order to deriving the most effective models. Each training engine ( 2110  and  2140 ) may also include, in addition to learning, pre-processing mechanisms (not shown) for process the training data prior to being used by learning mechanism to derive trained models. For example, it may include data segmentation mechanism that segment the received training data into separate sets, each may be used for training a specific model directed for a particular situation, e.g., the driving lane planning model training engine  2140  may be configured to derive a generic model for the general population, a personalized model for the driver/passenger of the vehicle, a model for lane planning in day light condition, a model for lane planning in night light condition, a model for lane planning in wet road condition, and a model for lane planning for snowy day condition. In this case, the pre-processing mechanism may then first group the received recorded human driving data  430  into different groups, each of which for one model planned so that the training engine may then use the appropriate training data group to learn the appropriate model. The models may be continuously updated when the new training data arrive. The update of the models may be performed by re-learning based on all data received (batch mode) or by incremental mode. 
     Once the models, include the lane detection models  2120  and the driving lane control models  2150 , are generated, they are used to plan lane related behavior for an autonomous driving vehicle in a human-like manner and in some instances personalized. As discussed previously, in operation, the obtained driving lane detection models  2120  and the driving lane control models  2150  are applied in a cascade manner. In the illustrated embodiments, when the vehicle is on the road, sensor(s) installed in the vehicle take pictures/videos of the road the vehicle is currently driving on and send such sensor data to the driving lane detector  2130 . In addition to the sensor data, the driving lane detector  2130  may also receive the self-aware capability parameters  510 . Via the self-aware capability parameters, the driving lane detector  2130  may determine various types of information, e.g., road condition, the vehicle&#39;s capabilities, etc., in order to determine how it may proceed in a way that is appropriate. For example, if it is night time of the day, which may be indicated in the extrinsic capability parameters, the driving lane detector  2130  may proceed to invoke a lane detection model that is trained for detecting lanes in dark light situation to achieve reliable performance. 
     Using the appropriately invoked lane detection model(s), the driving lane detector  2130  estimates segments of the lanes from the sensor data and optionally the estimated position of the vehicle. Such estimated lane segments and the vehicle position are then sent to the driving lane planning unit  2160 , where appropriate driving lane planning model(s) may then be applied in a cascade manner for planning the lane control behavior of the vehicle. 
     As discussed previously, lane planning includes both lane following and lane changing. In operation, lane planning is directed to either controlling the vehicle behavior in lane following or the vehicle behavior in lane changing. When the vehicle is in motion, the operation context may provide some indication as to whether lane following or lane changing planning is needed. For instance, if the vehicle needs to exit, it may need first to get into an exit lane from a current lane that does not lead to the exit. In this case, lane changing is implied so that the task involved in lane planning is for lane changing. In some embodiments, the passenger in the vehicle may also provide an explicit lane control decision to indicate lane changing, e.g., by turning on the turn signal. In some embodiments, an indication of lane changing may also be from the vehicle itself, e.g., the engine may experience some problem so that the autonomous driving system may send a lane control decision signal to the driving lane planning unit  2160 , instructing to prepare for lane changing so that the vehicle can move to the emergency lane. In normal situations, the vehicle may assume a default mode of lane following in absence of any indication of entering into a lane changing mode. 
     To perform lane planning, the driving lane planning unit  2160  receives, from different sources, various types of information (e.g., detected lanes, estimated vehicle position, lane planning decision, and self-aware capability parameters  510 ) and proceeds to lane planning accordingly. For example, if the lane control decision signal indicates that the current task is for lane following, models for lane following are to be retrieved and used for planning. If the current task is for lane changing, then models for lane changing are to be used. 
     Similar to the driving lane detector  2130 , the driving lane planning unit  2160  may invoke the generic lane planning model from  2150  for the planning. It may also invoke different lane planning models that are appropriate for the situation in hand in order to enhance the performance. As discussed earlier, the self-aware capability parameters  510  provide both intrinsic and extrinsic capability parameters, which may indicate the weather condition, road condition, etc. which can be used by the driving lane planning unit  2160  to invoke appropriate lane planning models for the planning. For example, if the current task is for lane following with a right turn coming up, personalized human-like models for the passenger in the event of a right turn from the current lane may be retrieved from  2150  and used to plan the vehicle behavior as to how to ease into a position in the current right lane and then make a right turn. 
     On the other hand, if the current task is for lane changing, the lane control decision indicates that it is to change to the lane left to the current one, and the self-aware capability parameters indicate heavy rain and flooded roads, then the driving lane planning unit  2160  may appropriately access lane planning models trained for planning lane changing behavior on very wet roads. In some embodiments, such tasks may also be carried out using generic lane changing models. Based on selected models for the tasks in hand, the driving lane planning unit  2160  generates the planned lane control, which may then be sent to the vehicle control module  450  ( FIG. 4A ) so that the planned lane control behavior can be implemented. 
     The driving lane planning unit  2160  may also perform personalized lane planning. In some embodiments, the passenger currently present in the vehicle may be known, e.g., either via driver/passenger information sent to the driving lane planning unit  2160  or via detection of the passenger (now shown) from the sensor data. Upon receiving such information about the passenger, the driving lane planning unit  2160  may appropriately invoke lane control models suitable for the passenger. Such invoked customized models may be a model for a subgroup that the passenger belongs to or may be a model that is personalized for the passenger. Such customized models may then be used to control how the lane planning is performed in a personalized manner. 
       FIG. 22  is a flowchart of an exemplary process for the lane planning module  570 , according to an embodiment of the present teaching. To obtained lane detection models, training data with labeled lanes are first received at  2210  and such received supervised data are then used to obtain, at  2230  via training, driving lane detection models. On the other hand, to obtain driving lane planning models, recorded human driving data, and optionally together with personalized profile information, are received at  2220  and used to derive, at  2240  via training, lane planning models. Once the models are obtained, they may be dynamically updated based on newly arrived training data (now shown). 
     During operation, when a vehicle is in motion, the sensors on the vehicle acquire sensor data including imagery of the road ahead of the vehicle with lanes present. Such sensor data are received at  2250  and are used to detect, at  2260 , lanes in front of the vehicle based on the lane detection models. The relative position of the vehicle may also be optionally estimated. Such detected lanes and optionally estimated vehicle position may then be sent to the driving lane planning unit  2160 . At the driving lane planning unit  2160 , various types of information received at  2270 , which include lane control decision, detected lanes, and self-aware capability parameters. Such information is used to determine the lane planning models to be used so that the lane planning can be achieved, at  2280 , based on appropriated selected lane planning models. 
     By learning from human driving data, the learned lane planning models capture the characteristics of human behavior in lane planning so that when such models are used in autonomous driving, the vehicle can be controlled in a human-like manner. In addition, by further personalizing lane planning models based on relevant driving data of the passengers/drivers, the lane planning behavior of the vehicle can be controlled in a manner that is familiar and comfortable for the passenger/driver in the vehicle. 
     With reference to  FIGS. 5-22 , details of the planning module  440  on route planning, motion planning, and lane planning are disclosed. The output of the planning module  440  includes the planned route  520  from the route planning module  550 , the planned motion  530  from the motion planning module  560 , and the planned lane control  540  from the lane planning module  570  (see  FIG. 5 ). Such output may be sent to different parts of the autonomous driving vehicle in order to carry out the planned vehicle behavior. For instance, the planned route  520  may be sent to the part of the vehicle that is responsible for guide the vehicle in terms of route control, e.g., such as the built in GPS. The planned motion  530  and the planned lane control  540  may be sent to the vehicle control module  450  (in  FIG. 4 ) so that the planned vehicle behavior as to motion and lane control may be carried out on the vehicle via the vehicle control module  450 . 
     When the motion and lane control planned to achieve human-like behavior, the vehicle control module  450  aims at delivering the planned action. According to the present teaching, the vehicle control module  450  also aims at learning how to control the vehicle according to the knowledge in terms of how the vehicle behaves or responds to different control signals in different situations so that the vehicle can be controlled to achieve the desired effect, including the planned vehicle behavior. Traditional approaches apply machine learning based control and derive vehicle dynamics models from classical mechanics, which often fail to model a variety of situations that occurred in real world. As a consequence, it often leads to poor performance and in some situations, may cause dangerous consequences. Although some conventional approaches is designed to learn the vehicle dynamics models from historic data via, e.g., neural networks, are able to learn the vehicle dynamics models in common scenarios, in some situations, such systems have made predictions that have substantial and unpredicted errors, which in real life can be fatal. 
     The present teaching discloses an approach that enables both achieving accurate simulation and safety of the vehicle performance. Instead of directly learn the vehicle dynamics model from the historic data, classical mechanics model is used as backbone model and learn how to adjust the predicted result from the historic data. In addition, limitation to the adjustment to be made is specified as a way to prevent a prediction result that significantly deviates from the normal situations. 
       FIG. 23A  depicts a system diagram of traditional approach for generating vehicle control signal. To determine vehicle control signals needed for control the vehicle to achieve certain target motion, a vehicle kinematic model  2310  is provided and used by a vehicle kinematic model (VKM) vehicle control signal generator  2320  based on the target motion and information about the current vehicle state. For example, if the current vehicle state is 30 miles per hour and the target motion is reaching 40 miles per hour in the next 5 seconds, the VKM vehicle control signal generator  2320  uses such information to determine, based on the vehicle kinematic model  2310 , what kind of vehicle control is to be applied so that the velocity acceleration can enable the vehicle to achieve the target motion. The approach as depicted in  FIG. 23A  is based on the traditional vehicle kinematic models  2310 , which is merely a mechanical dynamic model. 
       FIG. 23B  depicts a high level system diagram of the vehicle control module  450  in  FIG. 4A , according to an embodiment of the present teaching. The vehicle control module  450  as disclosed herein aims at providing the ability of generating vehicle control signals that can enable human-like driving behavior for autonomous vehicles. As shown, the vehicle control module  450  comprises a human-like vehicle control unit  2340  and a human-like vehicle control (HLVC) model  2330 . To achieve human like autonomous driving, the human-like vehicle control unit  2340  receives recorded human driving data  430  to uses that for learning human like vehicle control and generate the HLVC model  2330  that characterizes vehicle control behavior that is humanlike. 
     With the HLVC model  2330  created, when the human-like vehicle control unit  2340  receives information related to a target motion and the current vehicle state, it generates a human-like vehicle control signal based on the HLVC model  2330  with respect to the real time situation associated with the vehicle (characterized by the real time data  480 ). When additional recorded human driving data  430  are made available, the HLVC model  2330  may be dynamically updated or re-trained so that it captures the characteristics of human vehicle control behavior in a variety of situations. The dynamic update of the HLVC model  2330  may be triggered via a model update signal as shown in  FIG. 23B . The model update signal may be triggered manually or automatically when certain conditions are met, e.g., set up with regular update with a pre-determined internal or when additional data available for update amounts to a certain level. 
     In some embodiments, the HLVC model  2330  may also be personalized. This is illustrated in  FIG. 23C . The HLVC model  2330  in this case may comprise a plurality of HLVC sub-models (e.g.,  2330 - 1 ,  2330 - 2 , . . . ,  2330 -N), each of which may correspond to a sub-population with, e.g., similar characteristics, and trained based on a part of the recorded human driving data  430  that are related to the sub-population. For instance, a HLVC sub-model may be directed to sub-population for people who may prefer to drive cautiously so that the model may be derived based on training data in the recorded human driving data  430  from the driving record of a corresponding sub-population that exhibit cautious driving record. A HLVC sub-model may also be personalized (e.g., a sub-model is for an individual) if appropriate training data are applied to derive a personalized sub-model. 
     Details on the human-like vehicle control unit  2340  is disclosed below with reference to  FIGS. 24-29 .  FIG. 24  depicts an exemplary internal high level architecture of the human-like vehicle control unit  2340 , which comprises a human-like vehicle control model generator  2410  and a human like vehicle control signal generator  2420 . The human like vehicle control model generator  2410  takes recorded human driving data  430  as input and uses that information for learning and training the HLVC model  2330 . Exemplary types of data extracted from the recorded human driving data  430  for training may include, e.g., the vehicle control data applied to the vehicle and the vehicle states, which may include both the vehicles states prior to and after the vehicle control data are applied. 
     Data to be used for deriving the HLVC model  2330  may also include environment data that characterize the surrounding condition under which the vehicle control data yielded the corresponding vehicle state. The environment data may include various types of information, e.g., road condition, whether condition, vehicle type and condition. In some embodiments, the environment data may also include information about the passenger in the vehicle as well as characteristics of the passenger, e.g., gender, age, health situation, preferences, etc. All these different types of information from the human driving data may present some variables that may impact the passenger&#39;s vehicle control behavior. For instance, when the road is wet or slippery, human drivers may exhibit different vehicle control behavior in terms of break the vehicle (e.g., apply pressure on the brake more slowly) than that when the road is not slippery. 
     When the HLVC model  2330  is generated, it can be used by the human-like vehicle control signal generator  2420  to generate a vehicle control signal, when it receives a desired target motion, to yield human-like vehicle control behavior in achieving the desired target motion. To generate a human-like vehicle control signal, the vehicle control signal generator  2420  obtains real time data  480  comprising information about the surrounding of the vehicle at the time of the desired target motion and use such information in invoking the HLVC model  2330  to generate a human-like vehicle control signal. As illustrated in the example above, a target motion may be to accelerate, in 5 seconds, the vehicle to 40 miles per hour starting from the current velocity of 30 miles per hour. The real time data for that moment may indicate that the road the vehicle is on has a deep slope and the road is slippery because it is currently raining. Such real time data is relevant and may be provided as environment data to the HLVC model  2330 . The human-like vehicle control signal generator  2420  may invoke the HLVC model  2330  with such parameters in order to obtain an inferred human-like vehicle control signal that enables the autonomous vehicle to achieve the desired target motion in a manner similar to human driving. 
       FIG. 25  is a flowchart of an exemplary process of the human-like vehicle control unit  2340 , according to an embodiment of the present teaching. To generate the HLVC model  2330 , recorded human driving data are received at  2510  to obtain training data, which are used in a training process, at  2520 , to derive, at  2530 , the HLVC model  2330 . Such generated HLVC model  2330  is then used in operation when a requested target motion is received, at  2550 . Based on the requested target motion, the human-like vehicle control signal generator  2420  obtains, at  2560 , the real time data  480  in order to extract information related to the vehicle at the time, including environment data, current vehicle state, etc. In some embodiments, information related to the passenger (e.g., identification of the passenger or characteristics of the passenger) in the vehicle may also be obtained (not shown) so that such information may be used to personalize the process of generating an appropriate human-like vehicle control signal. 
     The information obtained by the human-like vehicle control signal generator  2420  may then be applied to the HLVC model  2330  to generate, at  2570 , the human-like vehicle control signal in accordance with the HLVC model  2330 . In the event of personalization, one or more specific HLVC sub-models appropriate for the situation may be invoked and used to generate personalized human-like vehicle control signal. During the operation, the human-like vehicle control unit  2340  may check, at  2540 , whether an update triggering signal is present. If an update signal is received, determined at  2540 , the human-like vehicle control model generator  2410  proceeds to step  2510  to gather training data and re-train or adaptively adjust the HLVC model  2330  based on the dynamically collected human driving data. 
       FIG. 26  depicts an exemplary high level system diagram of the human-like vehicle control model generator  2410 , according to an embodiment of the present teaching. In the illustrated embodiment, the human-like vehicle control model generator  2410  comprises a training data processing unit  2610 , a VKM vehicle control prediction engine  2630 , and a vehicle control model learning engine  2640 . The training data processing unit  2610  takes the recorded human driving data  430  as input, processes the input to generate training data  2620  to be used to train the HLVC model  2330 . The recorded human driving data  430  may include environment data  2620 - 1 , vehicle state data  2620 - 2 , . . . , and current vehicle control data  2620 - 3 . The environment data  2620 - 1  may include information such as the road condition such as the slope of the road, angle of the turn, slippage of the surface condition, wetness of the road, etc. The environment data may also include information related to limitations to the vehicle such as speed limit, time of the day, season, location, etc. Such environment data may be used as contextual information in training so that the HLVC model  2330  may learn human like vehicle control behavior in different context conditions such that the trained model characterizes human-like vehicle control behaviors exhibited in different situations. If personalization is needed, the obtained training data may also be classified into different sub-groups of training data (not shown), each of which may be used to train an HLVC sub-model specific for passengers that belong to that group. 
     The vehicle state data  2620 - 2  may include information characterizing the state of the vehicle, including, e.g., position of the vehicle, velocity of the vehicle, roll/pitch/yaw of the vehicle, and steering angle of the vehicle, etc. The vehicle control data  2620 - 3  may provide information characterizing the control applied to the vehicle, such as brake applied with a certain force, steering by turning the steering wheel by a certain angle, or throttle. 
     According to the present teaching, rather than training the HLVC model  2330  to generate the vehicle control signal directly, the present teaching combines or fuses the traditional kinematic model based prediction approach with learning model, created by learning from human driving data, as to how to adjust a vehicle control signal predicted using the traditional kinematic model so that the adjustment yields human-like vehicle control behavior. Such an integrated approach enables not only more accurate vehicle control but also human-like feel as to control of the vehicle. 
     In learning the HLVC model  2330 , the vehicle state data  2620 - 2  and vehicle control data  2620 - 3  are provided to the VKM vehicle control prediction engine  2630  to predict the motion achieved because of the control exercised. The VKM vehicle control prediction engine  2630  performs the prediction based on based on the vehicle kinematic model  2310 , e.g., via traditional mechanical dynamics approach to generate VKM based prediction signal, as shown in  FIG. 26 . The VKM based prediction is then sent to the vehicle control model learning engine  2640  to be combined with other information from the training data  2620  to learn. Based on the outputs of the VKM vehicle control prediction engine  2630  and the training data  2620 , the vehicle control model learning engine  2640  trains the HLVC model  2330  in, e.g., in an iterative process until the HLVC model  2330  converges. Once the model  2330  converges, it is used to derive human-like vehicle control signals given desired target motions. 
     As shown, the vehicle control model learning engine  2640  may be triggered by the model update signal. When it is activated, the vehicle control model learning engine  2640  invokes the training data processing unit  2610  and the VKM vehicle control prediction engine  2630  to initiate the training process. In some embodiment, any subsequent training based on additional human driving data may be performed in a derivative manner or in a batch mode, i.e., re-train the HLVC model  2330 . 
       FIG. 27  is a flowchart of an exemplary process for the human-like vehicle control model generator  2410 , according to an embodiment of the present teaching. Recorded human driving data  430  are first received at  2710 . The received human driving data are processed, at  2720 , to obtain training data. Some of the training data are used, at  2730 , to generate VKM based prediction based on the traditional vehicle kinematic model  2310 . For the learning and fusion based training, various aspects of the training data are identified, at  2740 , and used, at  2750 , to train the HLVC model  2330 . After convergence, the HLVC model  2330  is created at  2760 . 
       FIG. 28  depicts an exemplary high level system diagram of the human-like vehicle control signal generator  2420 , according to an embodiment of the present teaching. As discussed herein, the human-like vehicle control signal generator  2420  aims to generate, for a specified target motion, a human-like vehicle control signal based on the HLVC model  2330  so that when the human-like vehicle control signal is used to control the vehicle, the vehicle exhibits human-like vehicle control behavior. In this embodiment, the human-like vehicle control signal generator  2420  comprises a VKM vehicle control signal inference engine  2810 , a context data determiner  2820 , and an HLVC model based fusion unit  2830 . 
     In operation, upon receiving a target motion, the human-like vehicle control signal, the VKM vehicle control signal inference engine  2810  obtains the current state of the vehicle and generates a VKM based vehicle control signal based on the vehicle kinematic model  2310 . As discussed herein, the use of the traditional approach to generate an inferred vehicle control signal based merely on the vehicle kinematic model  2310  aims at providing initially an inferred vehicle control signal based on purely on mechanical dynamics. To achieve human-like behavior in vehicle control to achieve the target motion, the inferred VKM based vehicle control signal is to be further used as an input to the HLVC model based fusion unit  2830 , where the VKM based vehicle control signal is used as the initial inferred result to be fused with the HLVC based approach so that the VKM based vehicle control signal may be adjusted in accordance with the learned HLVC model  2330 . 
     The HLVC model based fusion unit  2830  may activate, upon receiving the target motion, the context data determiner  2820  to obtain any information related to the surrounding of the vehicle. The context data determiner  2820  receives the real time data  480  and extracts relevant information such as environment data or passenger data, etc. and sends to the HLVC model based fusion unit  2830 . Based on the target motion, the current vehicle state, context information surrounding the vehicle, and the VKM based vehicle control signal inferred using the traditional vehicle kinematic model  2310 , the HLVC model based fusion unit  2830  accesses the HLVC model  2330  based on such input data to obtain a fused human-like vehicle control signal. 
     As discussed herein, the HLVC model  2330  may be created by learning the discrepancies between VKM model based predictions and the observed information from the recorded human driving data  430 . As such, what the HLVC model  2330  captures and learns may correspond to adjustments to be made to the VKM based vehicle control signals to achieve human-like behavior. As discussed previously, as learning process may create overfitting situation, especially when the training data include outliers, to minimize the risks in vehicle control due to adjustment to the VKM based vehicle control signal, the human-like vehicle control signal generator  2420  may also optionally include preventative measures by limiting the adjustments to VKM vehicle control signals based on some fusion constraints  2840 , as shown in  FIG. 28 . In this manner, the human-like vehicle control signal generated as a modified VKM based vehicle control signal can maximize the likelihood as to human behavior yet minimize the risks in vehicle control. 
     In some embodiments, information about the passenger in the vehicle may also be extracted from real time data  480  and can be used to access personalized HLVC sub-model related to the passenger, which may be a HLVC sub-model for a group that the passenger belongs or a completely personalized HLVC sub-model for the passenger). Using such a personalized HLVC sub-model may allow the human-like vehicle control signal generator  2420  to generate personalized human-like vehicle control signal so that the vehicle control carried out based on it can be not only human-like but also to the personal liking of the passenger. 
       FIG. 29  is a flowchart of an exemplary process of the human-like vehicle control signal generator  2420 , according to an embodiment of the present teaching. Target motion information and the vehicle state data are first received at  2910 . Based on the target motion and the vehicle state, the vehicle kinematic model  2310  is accessed, at  2920 , and used to infer, at  2930 , the VKM vehicle control signal. Such inferred control signal based on mechanical dynamic model is sent to the HLVC model based fusion unit  2830 . To yield a fused human-like vehicle control signal, the context data determiner  2820  receives, at  2940 , real time data  480  and extracts, at  2950 , relevant information related to the vehicle. Using the context information as well as the VKM vehicle control signal, the HLVC model based fusion unit  2830  infers, at  2960 , the human-like vehicle control signal based on the HLVC model  2330 . Such inferred human-like vehicle control signal is then output, at  2970 , so that human-like vehicle control to achieve the target motion may be carried out in a human-like manner. 
       FIG. 30  depicts the architecture of a mobile device which can be used to realize a specialized system implementing the present teaching. This mobile device  3000  includes, but is not limited to, a smart phone, a tablet, a music player, a handled gaming console, a global positioning system (GPS) receiver, and a wearable computing device (e.g., eyeglasses, wrist watch, etc.), or in any other form factor. The mobile device  3000  in this example includes one or more central processing units (CPUs)  3040 , one or more graphic processing units (GPUs)  3030 , a memory  3060 , a communication platform  3010 , such as a wireless communication module, storage  3090 , one or more input/output (I/O) devices  3050 , a display or a projection  3020 - a  for visual based presentation, and one or more multi-modal interface channels  3020 - b . The multi-modal channels may include acoustic channel or other media channels for signaling or communication. Any other suitable component, including but not limited to a system bus or a controller (not shown), may also be included in the mobile device  3000 . As shown in  FIG. 30 , a mobile operating system  3070 , e.g., iOS, Android, Windows Phone, etc., and one or more applications  3080  may be loaded into the memory  3060  from the storage  3090  in order to be executed by the CPU  3040 . 
     To implement various modules, units, and their functionalities described in the present disclosure, computer hardware platforms may be used as the hardware platform(s) for one or more of the elements described herein. The hardware elements, operating systems and programming languages of such computers are conventional in nature, and it is presumed that those skilled in the art are adequately familiar therewith to adapt those technologies to the present teachings as described herein. A computer with user interface elements may be used to implement a personal computer (PC) or other type of work station or terminal device, although a computer may also act as a server if appropriately programmed. It is believed that those skilled in the art are familiar with the structure, programming and general operation of such computer equipment and as a result the drawings should be self-explanatory. 
       FIG. 31  depicts the architecture of a computing device which can be used to realize a specialized system implementing the present teaching. Such a specialized system incorporating the present teaching has a functional block diagram illustration of a hardware platform which includes user interface elements. The computer may be a general purpose computer or a special purpose computer. Both can be used to implement a specialized system for the present teaching. This computer  3100  may be used to implement any component or aspect of the present teachings, as described herein. Although only one such computer is shown, for convenience, the computer functions relating to the present teachings as described herein may be implemented in a distributed fashion on a number of similar platforms, to distribute the processing load. 
     The computer  3100 , for example, includes COM ports  3150  connected to and from a network connected thereto to facilitate data communications. The computer  3100  also includes a central processing unit (CPU)  3120 , in the form of one or more processors, for executing program instructions. The exemplary computer platform includes an internal communication bus  3110 , program storage and data storage of different forms, e.g., disk  3170 , read only memory (ROM)  3130 , or random access memory (RAM)  3140 , for various data files to be processed and/or communicated by the computer, as well as possibly program instructions to be executed by the CPU. The computer  2600  also includes an I/O component  3160 , supporting input/output flows between the computer and other components therein such as interface elements  3180  in different media forms. An exemplary type of interface element may correspond to different types of sensors  3180 - a  deployed on the autonomous driving vehicle. Another type of interface element may correspond to a display or a projection  3180 - b  for visual based communication. There may be additional components for other multi-modal interface channels such as acoustic device  3180 - c  for audio based communications and/or component  2680 - d  for signaling based on communication, e.g., signal that causes vibration on a vehicle component such as a car seat. The computer  3100  may also receive programming and data via network communications. 
     Hence, aspects of the methods of the present teachings, as outlined above, may be embodied in programming. Program aspects of the technology may be thought of as “products” or “articles of manufacture” typically in the form of executable code and/or associated data that is carried on or embodied in a type of machine readable medium. Tangible non-transitory “storage” type media include any or all of the memory or other storage for the computers, processors or the like, or associated modules thereof, such as various semiconductor memories, tape drives, disk drives and the like, which may provide storage at any time for the software programming. 
     All or portions of the software may at times be communicated through a network such as the Internet or various other telecommunication networks. Such communications, for example, may enable loading of the software from one computer or processor into another, for example, from a management server or host computer of a search engine operator or other enhanced ad server into the hardware platform(s) of a computing environment or other system implementing a computing environment or similar functionalities in connection with the present teachings. Thus, another type of media that may bear the software elements includes optical, electrical and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical landline networks and over various air-links. The physical elements that carry such waves, such as wired or wireless links, optical links or the like, also may be considered as media bearing the software. As used herein, unless restricted to tangible “storage” media, terms such as computer or machine “readable medium” refer to any medium that participates in providing instructions to a processor for execution. 
     Hence, a machine-readable medium may take many forms, including but not limited to, a tangible storage medium, a carrier wave medium or physical transmission medium. Non-volatile storage media include, for example, optical or magnetic disks, such as any of the storage devices in any computer(s) or the like, which may be used to implement the system or any of its components as shown in the drawings. Volatile storage media include dynamic memory, such as a main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wire and fiber optics, including the wires that form a bus within a computer system. Carrier-wave transmission media may take the form of electric or electromagnetic signals, or acoustic or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media therefore include for example: a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any other optical medium, punch cards paper tape, any other physical storage medium with patterns of holes, a RAM, a PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave transporting data or instructions, cables or links transporting such a carrier wave, or any other medium from which a computer may read programming code and/or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a physical processor for execution. 
     Those skilled in the art will recognize that the present teachings are amenable to a variety of modifications and/or enhancements. For example, although the implementation of various components described above may be embodied in a hardware device, it may also be implemented as a software only solution—e.g., an installation on an existing server. In addition, the present teachings as disclosed herein may be implemented as a firmware, firmware/software combination, firmware/hardware combination, or a hardware/firmware/software combination. 
     While the foregoing has described what are considered to constitute the present teachings and/or other examples, it is understood that various modifications may be made thereto and that the subject matter disclosed herein may be implemented in various forms and examples, and that the teachings may be applied in numerous applications, only some of which have been described herein. It is intended by the following claims to claim any and all applications, modifications and variations that fall within the true scope of the present teachings.