Patent Description:
Systems configured to record, store, and transmit video, audio, and sensor data associated with a vehicle are known. Autopilots for vehicles are known. Typically, an autonomous operation mode of vehicle control may be entered or exited under certain specific and/or pre-determined parameters. Some vehicles may interface/interoperate with external computers (e.g., at an automobile mechanic) where information that is relevant to vehicle operation may be generated and/or processed. <CIT> discloses an information presenting apparatus used in an autonomous vehicle capable of switching between autonomous driving control and manual driving control, the apparatus determining a response action for checking that the driver is ready to take over when the autonomous driving control is switched to the manual driving control. <CIT> discloses a method for controlling a vehicle involving determining period of time for generation of warning signal as function of driver's attention level.

One aspect of the invention relates to a system as defined in claim <NUM> configured to determine when to gauge whether a vehicle operator in an autonomously controlled vehicle is prepared to assume manual control of a vehicle. The system is coupled and/or otherwise related to a vehicle. Some or all of the system is installed in the vehicle and/or otherwise coupled with the vehicle. The system is configured to capture information based on vehicle operation. In some implementations, the system is configured to off-load and/or otherwise generate output signals conveying information related to the operation of the vehicle, the vehicle operator, driving parameters, and/or other relevant information. The system is configured to gauge, measure, derive, estimate, approximate, and/or otherwise determine whether a vehicle operator is ready or prepared to assume control of a vehicle, based on a combination of different types of information. In some implementations, the system controls vehicle operation based on a level of confidence in the vehicle operator and/or the autopilot to handle current driving parameters.

In some implementations or modes of operation, the system may detect vehicle events based on a comparison of the information conveyed by the output signals from the sensors to predetermined (variable and/or fixed) values, threshold, functions, and/or other information. Advantageously, in some implementations or modes of operation, the system identifies vehicle events in real-time or near real-time during operation of the vehicle. As used herein, the term "processor" is used interchangeably with the term "physical processor.

Individual sensors are configured to generate output signals conveying information. The information may include one or more of visual information, motion-related information, position-related information, biometric information, and/or other information. In some implementations, the system determines one or more parameters that are measured, derived, estimated, approximated, and/or otherwise determined based on one or more output signals generated by one or more sensors.

The individual sensors of the system may include, by way of non-limiting example, one or more of an altimeter (e.g. a sonic altimeter, a radar altimeter, and/or other types of altimeters), a barometer, a magnetometer, a pressure sensor (e.g. a static pressure sensor, a dynamic pressure sensor, a pitot sensor, etc.), a thermometer, an accelerometer, a gyroscope, an inertial measurement sensor, global positioning system sensors, a tilt sensor, a motion sensor, a vibration sensor, an image sensor, a camera, an ultrasonic sensor, an infrared sensor, a light sensor, a microphone, an air speed sensor, a ground speed sensor, an altitude sensor, biometric sensors (including but not limited to blood pressure sensor, pulse oximeter, heart rate sensor, Blood Alcohol Level (BAC), etc.), degree-of-freedom sensors (e.g. <NUM>-DOF and/or <NUM>-DOF sensors), a compass, seat pressure sensor, and/or other sensors. As used herein, the term "motion sensor" includes one or more sensors configured to generate output conveying information related to position, location, distance, motion, movement, acceleration, and/or other motion-based parameters. Output signals generated by individual sensors (and/or information based thereon) may be stored and/or transferred in electronic files.

The individual sensors of the system may include image sensors, cameras, depth sensors, remote sensors, and/or other sensors. As used herein, the terms "camera" and/or "image sensor" include any device that captures images, including but not limited to a single lens-based camera, a camera array, a solid-state camera, a mechanical camera, a digital camera, an image sensor, a depth sensor, a remote sensor, a lidar, an infrared sensor, a (monochrome) complementary metal-oxide-semiconductor (CMOS) sensor, an active pixel sensor, and/or other sensors. Different types of individual sensors are configured to capture information, including but not limited to visual information, video information, audio information, geolocation information, orientation and/or motion information, depth information, and/or other information. Information captured by one or more sensors may be marked, timestamped, annotated, and/or otherwise processed such that information captured by other sensors can be synchronized, aligned, annotated, and/or otherwise associated therewith. For example, video information captured by an image sensor may be synchronized with information captured by an accelerometer or other sensor. Output signals generated by individual image sensors (and/or information based thereon) may be stored and/or transferred in electronic files.

In some implementations, an image sensor may be integrated with electronic storage such that captured information can be stored in the integrated embedded storage. In some implementations, the system includes one or more cameras. For example, a camera can include one or more image sensors and electronic storage media. In some implementations, an image sensor is configured to transfer captured information to remote electronic storage media, e.g., through "the cloud.

The one or more servers include one or more processors configured to execute one or more computer program components. The computer program components include one or more of a parameter determination component, an autopilot component, a timing component, a challenge component, a response component, a responsiveness component, a vehicle control component, a confidence component, a record component, a transmission component, and/or other components.

The parameter determination component is configured to determine parameters, e.g., based on output signals from sensors. Parameters include one or more of operating parameters, driving parameters, traffic parameters, road surface parameters, weather parameters, vehicle parameters, operator parameters, environmental parameters, and/or other parameters. The different types of parameters used in this invention need not be mutually exclusive. For example, a parameter representing the current level of precipitation could be both a weather parameter and an environmental parameter. For example, a parameter representing how icy the road is could be both a road surface parameter and an environmental parameter. By way of non-limiting example, other relationships between different types of parameters are described in this disclosure.

Current operating parameters are related to the vehicle, the operation of the vehicle, physical characteristics of the vehicle, and/or other information. Driving parameters are related to traffic parameters, road surface parameters, weather parameters, and/or other information. Vehicle parameters are related to the vehicle, the operation of the vehicle, physical characteristics of the vehicle, and/or other information. Operator parameters are related to the vehicle operator, and/or other information. Environmental parameters are related to the environment outside of the vehicle, visibility, and/or other information.

In some implementations, the parameter determination component is configured to determine a parameter one or more times in an ongoing manner during operation of the vehicle, and/or over periods longer than one day, one week, and/or one month. In some implementations, one or more parameters (e.g., weather parameters) are received from one or more sources external to the vehicle. For example, a source external to the vehicle could include a remote server and/or an external provider.

The autopilot component is configured to operate a vehicle autonomously, e.g., in an autonomous operation mode. As used herein, the term "autonomous" refers to a lack of human intervention or control, at least for some continuous duration.

The timing component is configured to determine moments in time for gauging whether a vehicle operator is ready to assume control of a vehicle. A vehicle operator being ready to assume manual control is referred to as being prepared or as preparedness. In some implementations, the timing component is configured to determine a moment in time to present a challenge to the vehicle operator. Once a challenge has been presented, the vehicle operator may take action to respond to or meet the challenge. The challenge and response are used to gauge a level of responsiveness of the vehicle operator. Preparedness may correspond to responsiveness of the vehicle operator. For example, a highly responsive vehicle operator could be deemed highly prepared to assume manual control of the vehicle. On the other hand, a vehicle operator who is slow to respond to a challenge could be deemed unprepared or insufficiently prepared to assume manual control of the vehicle.

The challenge component is configured to present challenges to vehicle operators. In some implementations, the challenge component is configured to present a challenge to a vehicle operator at a moment in time as determined by the timing component. For example, a challenge may include one or more of a generation of a sound, activation of an indicator, and/or another audible and/or visual notification to which a vehicle operator is expected to respond, in order to confirm he or she is paying attention in general, and, more specifically, ready to assume manual control of the vehicle if needed.

The response component is configured to detect whether a vehicle operator has responded to a challenge presented by the challenge component. In some implementations, a response may include audible feedback (i.e., the vehicle operator could say a particular phrase out loud such that his or her response can be captured through a microphone), a particular gesture or motion, a particular interaction with a user interface (i.e., the vehicle operator can push a button to indicate a response), and/or other types of responses capable of being interpreted and/or recognized through one or more sensors of the vehicle.

The responsiveness component is configured to determine levels of responsiveness of a vehicle operator. In some implementations, determinations by the responsiveness component are based on detections by the response component. Alternatively, and/or simultaneously, determinations by the responsiveness component can be based on (the timing of) presentations by the challenge component. For example, a level of responsiveness may be based on the elapsed time between a presentation of a challenge and detection of a response to that challenge by the vehicle operator.

The vehicle control component is configured to operate the vehicle in one or more modes of operation. In some implementations, the vehicle control component is configured to transition vehicle control between different modes of operation. For example, the vehicle control component may transition vehicle control from an autonomous mode of operation to a manual mode of operation, and/or vice versa. For example, the vehicle control component may transition vehicle control from an autonomous mode of operation to halt vehicle operation, e.g., by pulling the vehicle over or parking the vehicle. In some implementations, operations and/or transitions by the vehicle control component is based on determinations by the responsiveness component, and/or based on the operation of other components.

The confidence component is configured to determine and/or interpret levels of confidence pertaining to one or more modes of vehicle operation. In some implementations, the confidence component is configured to determine and/or interpret an operator confidence. Operator confidence is a level of confidence that the vehicle operator is prepared to assume manual control of the vehicle. In some implementations, the confidence component is configured to determine and/or interpret an automation confidence. Automation confidence is a level of confidence that vehicle control through an autonomous mode of operation is prepared to continue autonomous vehicle operation. One or more levels of confidence could be used by other components, e.g., by the timing component.

The record component is configured to capture, record, store, and/or transmit information, including but not limited to information related to vehicle operation. In some implementations, information related to vehicle operation is used to create vehicle event records. For example, changes in the operation mode of the vehicle can be used to create a vehicle event record. For example, determinations by the responsiveness component can be used to create a vehicle record. For example, determinations by other components may be used to create a vehicle record. Vehicle event records include video information, audio information, data from an engine control module (ECM) system, metadata, information based on sensor-generated output, and/or other information.

The transmission component is configured to transmit information, e.g., to one or more remote servers that are external to a vehicle. For example, the transmission component can transmit vehicle event records. Vehicle event records may be stored locally in a vehicle and/or transmitted from a vehicle to a system, server, and/or a service that is external to the vehicle, including but not limited to a remote server and/or an external provider. In some implementations, a system, server, and/or a service that is external to the vehicle can query and/or request information from a particular vehicle. The transmission component is configured to respond to a query or request by transmitting information as queried and/or requested. In some implementations, the transmission component is configured to facilitate communication of information between vehicles, remote servers, external providers, and/or other systems, servers, and/or services external to vehicles. Communication may be in real-time or near real-time. In some implementations, communication is wireless.

The transmission component is configured to generate and/or determine notifications related to vehicle operation. In some implementations, notifications are intended for drivers of vehicles. For example, the transmission component can receive transmissions, such as notifications for drivers, including but not limited to warnings or requests (for example to reduce speed). In some implementations, notifications are transmitted from a vehicle to a system, server, and/or a service that is external to the vehicle, including but not limited to a remote server and/or an external provider.

As used herein, any association (or relation, or reflection, or indication, or correspondency) involving vehicles, sensors, vehicle events, parameters, parameters, thresholds, functions, notifications, and/or another entity or object that interacts with any part of the system and/or plays a part in the operation of the system, can be a one-to-one association, a one-to-many association, a many-to-one association, and/or a many-to-many association or N-to-M association (note that N and M may be different numbers greater than <NUM>).

As used herein, the term "obtain" (and derivatives thereof) may include active and/or passive retrieval, determination, derivation, transfer, upload, download, submission, and/or exchange of information, and/or any combination thereof. As used herein, the term "effectuate" (and derivatives thereof) may include active and/or passive causation of any effect. As used herein, the term "determine" (and derivatives thereof) may include measure, calculate, compute, estimate, approximate, generate, and/or otherwise derive, and/or any combination thereof.

These and other objects, features, and characteristics of the servers, systems, and/or methods disclosed herein, as well as the methods of operation and functions of the related elements of structure and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this disclosure, wherein like reference numerals designate corresponding parts in the various figures. As used in the specification and in the claims, the singular form of "a", "an", and "the" include plural referents unless the context clearly dictates otherwise.

<FIG> illustrates a system <NUM> configured to transition vehicle control between autonomous operation and manual operation for a vehicle <NUM>. Some or all of system <NUM> is installed in vehicle <NUM>, carried by vehicle <NUM>, and/or otherwise coupled with and/or related to vehicle <NUM>. System <NUM> includes one or more of sensors <NUM>, one or more servers <NUM>, one or more physical processors <NUM>, electronic storage <NUM>, a network <NUM>, one or more external providers <NUM>, and/or other components. In a preferred embodiment, system <NUM> includes a set of sensors <NUM>, and one or more processors <NUM> configured to execute multiple computer program components. One or more sensors <NUM> are configured to generate output signals. The output signals convey information related to vehicle <NUM>, vehicle operation of vehicle <NUM>, a vehicle operator of vehicle <NUM>, operating parameters of vehicle <NUM>, parameters of vehicle <NUM>, and/or other parameters.

Information related to current operating parameters of the vehicle include feedback information from one or more of the mechanical systems of vehicle <NUM>, and/or other information. The mechanical systems of vehicle <NUM> include, for example, the engine, the drive train, the lighting systems (e.g., headlights, brake lights), the braking system, the transmission, fuel delivery systems, and/or other mechanical systems. The mechanical systems of vehicle <NUM> include one or more mechanical sensors, electronic sensors, and/or other sensors that generate the output signals (e.g., seat belt sensors, tire pressure sensors, etc.). In some implementations, at least one of sensors <NUM> is a vehicle system sensor included in an ECM system of vehicle <NUM>.

In some implementations, sensors <NUM> include one or more video cameras, one or more image sensors, and/or one or more microphones, and/or other sensors. Based on an analysis of images and/or sounds captured, system <NUM> can determine, using algorithms, that vehicle <NUM> is moving forward, is in reverse, has maneuvered outside of its lane of traffic, is making a turn, and/or other maneuvers. For example, by way of non-limiting example, driving maneuvers include swerving, a U-turn, freewheeling, over-revving, lane-departure, short following distance, imminent collision, unsafe turning that approaches rollover and/or vehicle stability limits, hard braking, rapid acceleration, idling, driving outside a geo-fence boundary, crossing double-yellow lines, passing on single-lane roads, a certain number of lane changes within a certain amount of time or distance, fast lane change, cutting off other vehicles during lane-change speeding, running a red light, running a stop sign, and/or other driving maneuvers.

In some implementations, information related to current operating parameters of vehicle <NUM> includes information related to the environment in and/or around vehicle <NUM>. The vehicle environment includes spaces in and around an interior and an exterior of vehicle <NUM>. The information may be related to movement of vehicle <NUM>, an orientation of vehicle <NUM>, a geographic position of vehicle <NUM>, a spatial position of vehicle <NUM> relative to other objects, a tilt angle of vehicle <NUM>, an inclination/declination angle of vehicle <NUM>, and/or other information. In some implementations, the output signals conveying information are generated via non-standard aftermarket sensors installed in vehicle <NUM>. Non-standard aftermarket sensors could include, for example, a video camera, a microphone, an accelerometer, a gyroscope, a geolocation sensor (e.g., a GPS device), a radar detector, a magnetometer, radar (e.g., for measuring distance of leading vehicle), and/or other sensors. In some implementations, sensors <NUM> include multiple cameras positioned around vehicle <NUM> and synchronized together to provide a <NUM> degree view of the inside of vehicle <NUM> and/or a <NUM> degree view of the outside of vehicle <NUM>.

Although sensors <NUM> are depicted in <FIG> as five elements, this is not intended to be limiting. Sensors <NUM> include one or more sensors located adjacent to and/or in communication with the various mechanical systems of vehicle <NUM>, in one or more positions (e.g., at or near the front of vehicle <NUM>, at or near the back of vehicle <NUM>, on the side of vehicle <NUM>, on or near the windshield of vehicle <NUM>, facing outward and/or inward, etc.) to accurately acquire information representing the vehicle environment (e.g. visual information, spatial information, orientation information), and/or in other locations. For example, in some implementations, system <NUM> is configured such that a first sensor is located near or in communication with a rotating tire of vehicle <NUM>, and a second sensor located on top of vehicle <NUM> is in communication with a geolocation satellite. In some implementations, sensors <NUM> are configured to generate output signals continuously during operation of vehicle <NUM>. In some implementations, one or more sensors <NUM> are located external to vehicle <NUM>, e.g., as depicted in <FIG>, in proximity of a road <NUM>. In some implementations, sensors external to vehicle <NUM> are configured to generate output signals that convey information that is external to vehicle <NUM> but pertinent to vehicle operation, including but not limited to traffic information, visibility information, road surface information, and/or other information.

As shown in <FIG>, server <NUM> includes one or more processors <NUM> configured to execute one or more computer program components. System <NUM> is configured to gauge, measure, derive, estimate, approximate, and/or otherwise determine whether a vehicle operator is ready or prepared to assume control of vehicle <NUM>, based on a combination of different types of information. In some implementations, system <NUM> controls vehicle operation based on a level of confidence in the vehicle operator and/or the autopilot to handle current driving parameters. The computer program components include one or more of a parameter determination component <NUM>, an autopilot component <NUM>, a timing component <NUM>, a challenge component <NUM>, a response component <NUM>, a responsiveness component <NUM>, a vehicle control component <NUM>, a confidence component <NUM>, a record component <NUM>, a transmission component <NUM>, and/or other components. In a preferred embodiment, system <NUM> is configured to execute at least autopilot component <NUM>, timing component <NUM>, challenge component <NUM>, response component <NUM>, responsiveness component <NUM>, and vehicle control component <NUM>, in order to operate the vehicle autonomously in an autonomous operation mode; determine a first moment in time for presenting a challenge to a vehicle operator to gauge a level of responsiveness of the vehicle operator, wherein determination of the first moment in time is based on the output signals; present, at the first moment in time, the challenge to the vehicle operator to gauge the level of responsiveness of the vehicle operator; detect whether the vehicle operator provides a response to the challenge; determine the level of responsiveness of the vehicle operator based on the detection; and operate the vehicle based on the determined level of responsiveness.

Parameter determination component <NUM> is configured to determine parameters, e.g., current operating parameters and/or vehicle parameters of vehicle <NUM>. Parameter determination component <NUM> can determine one or more of current operating parameters, driving parameters, traffic parameters, road surface parameters, weather parameters, vehicle parameters, operator parameters, environmental parameters, and/or other parameters based on the information conveyed by the output signals from sensors <NUM> and/or other information. The one or more current operating parameters are related to vehicle <NUM>, the operation of vehicle <NUM>, physical characteristics of vehicle <NUM>, and/or other information. Driving parameters are related to traffic parameters, road surface parameters, weather parameters, and/or other information. Vehicle parameters are related to vehicle <NUM>, the operation of vehicle <NUM>, physical characteristics of vehicle <NUM>, and/or other information. Operator parameters are related to the vehicle operator, and/or other information. Environmental parameters are related to the environment outside of vehicle <NUM>, visibility for the vehicle operator, and/or other information. In some implementations, parameter determination component <NUM> is configured to determine one or more of the current operating parameters and/or the vehicle parameters one or more times in an ongoing manner during operation of vehicle <NUM>.

In some implementations, operating parameters include vehicle parameters. For example, vehicle parameters are related to one or more of an acceleration, a direction of travel, a turn diameter, a vehicle speed, an engine speed (e.g., RPM), a duration of time, a closing distance, a lane departure from an intended travelling lane of the vehicle, a following distance, physical characteristics of vehicle <NUM> (such as mass and/or number of axles, for example), a tilt angle of vehicle <NUM>, an inclination/declination angle of vehicle <NUM>, and/or other parameters.

The physical characteristics of vehicle <NUM> may be physical features of vehicle <NUM> set during manufacture of vehicle <NUM>, during loading of vehicle <NUM>, and/or at other times. For example, the one or more vehicle parameters could include a vehicle type (e.g., a car, a bus, a semi-truck, a tanker truck), a vehicle size (e.g., length), a vehicle weight (e.g., including cargo and/or without cargo), a number of gears, a number of axles, a type of load carried by vehicle <NUM> (e.g., food items, livestock, construction materials, hazardous materials, an oversized load, a liquid), vehicle trailer type, trailer length, trailer weight, trailer height, a number of axles, and/or other physical features.

Traffic parameters are related to one or more of the number of other vehicles on the road, the distance to one or more other vehicles, the average distance to a vehicle in front of vehicle <NUM>, average speed of vehicle <NUM>, and/or other parameters that vary with traffic conditions. Road surface parameters are related to one or more of the material that comprises the road, the inclination of the road, the width of the road, the curviness of the road, the wetness of the road, the iciness of the road, and/or other parameters related to road surface. Weather parameters are related to one or more of temperature, humidity, precipitation, wind speed and direction, storminess, visibility, ambient lighting conditions, and/or other parameters related to weather. Operator parameters are related to one or more of the height, weight, reaction time, dexterity, driving record, reaction time, eye-sight, hearing, and/or other measurements or physical characteristics of the vehicle operator. Operator parameters are related to one or more of biometric information of the vehicle operator, including but not limited to heart rate, breathing rate, blood pressure level, and/or other biometric information. Operator parameters are related to one or more of the current position or location of the vehicle operator, the direction of the vehicle operator's face, eyes, or gaze, and/or other operator information. Environmental parameters are related to one or more of the environment outside of the vehicle, visibility, and/or other information.

In some implementations, parameter determination component <NUM> is configured to determine one or more vehicle parameters based on the output signals from at least two different sensors. For example, parameter determination component <NUM> can determine one or more of the vehicle parameters based on output signals from a sensor <NUM> related to the ECM system and an external aftermarket added sensor <NUM>. In some implementations, a determination of one or more of the vehicle parameters based on output signals from at least two different sensors <NUM> could be more accurate and/or precise than a determination based on the output signals from only one sensor <NUM>. For example, on an icy surface, output signals from an accelerometer may not convey that a driver of vehicle <NUM> is applying the brakes of vehicle <NUM>. However, a sensor in communication with the braking system of vehicle <NUM> would convey that the driver is applying the brakes. System <NUM> could determine a value of a braking parameter based on the braking sensor information even though the output signals from the accelerometer fail to convey that the driver is applying the brakes.

Parameter determination component <NUM> is configured to determine vehicle parameters that are not directly measurable by any of the available sensors. For example, an inclinometer may not be available to measure the road grade, but vehicle speed data as measured by a GPS system and/or by a wheel sensor ECM can be combined with accelerometer data to determine the road grade. If an accelerometer measures a force that is consistent with braking, but the vehicle speed remains constant, the parameter component can determine that the measured force is a component of the gravity vector that is acting along the longitudinal axis of the vehicle. By using trigonometry, the magnitude of the gravity vector component can be used to determine the road grade (e.g., pitch angle of the vehicle in respect to the horizontal plane).

In some implementations, one or more of the vehicle parameters are determined one or more times in an ongoing manner during operation of vehicle <NUM>. In some implementations, one or more of the vehicle parameters are determined at regular time intervals during operation of vehicle <NUM>. The timing of the vehicle parameter determinations (e.g., in an ongoing manner, at regular time intervals, etc.) can be programmed at manufacture, obtained responsive to user entry and/or selection of timing information via a user interface and/or a remote computing device, and/or can be determined in other ways. In some implementations, the time intervals of parameter determination are significantly less (e.g., more frequent) than the time intervals at which various sensor measurements are available. In such cases, system <NUM> can estimate vehicle parameters in between the actual measurements of the same vehicle parameters by the respective sensors, to the extent that the vehicle parameters are measurable. This may be established by means of a physical model that describes the behavior of various vehicle parameters and their interdependency. For example, a vehicle speed parameter could be estimated at a rate of <NUM> times per second, although the underlying speed measurements are much less frequent (e.g., four times per second for ECM speed, one time per second for GPS speed). This may be accomplished by integrating vehicle acceleration, as measured by the accelerometer sensor where the measurements are available <NUM> times per second, across time to determine change in speed that is accumulated over time again for the most recent vehicle speed measurement. The benefit of these more frequent estimates of vehicle parameters could include improved operation of other components of system <NUM>, reduced complexity of downstream logic and system design (e.g., all vehicle parameters are updated at the same interval, rather than being updating irregularly and at the interval of each respective sensor), and more pleasing (e.g., "smooth") presentation of vehicle event recorder data through an event player apparatus.

In some implementations, one or more types of information are received by system <NUM> through network <NUM>, e.g., the internet. Network <NUM> could include private networks, public networks, and/or combinations thereof. For example, information related to weather parameters could be received from a particular external provider <NUM> that provides weather information. For example, information related to road surface parameters could be received from a particular external provider <NUM> that provides road parameter information. For example, information related to traffic parameters could be received from a particular external provider <NUM> that provides traffic information.

In some implementations, a value of a current operating parameter that effectuates a particular determination and/or detection varies as a function of a parameter or of other information.

Autopilot component <NUM> is configured to operate vehicle <NUM> autonomously, e.g., in an autonomous operation mode. In some implementations, system <NUM> supports a single autonomous operation mode. In some implementations, system <NUM> supports multiple autonomous operation modes. For example, a vehicle operator could select which mode of operation is preferred. In some implementations, autonomous operation is based on a particular operator-specific destination, such as a destination address, or the nearest coffee shop. Autopilot component <NUM> is configured to use output signals from sensors and/or parameters derived therefrom in order to operate vehicle <NUM> autonomously.

Timing component <NUM> is configured to determine moments in time for gauging whether a vehicle operator is ready to assume control of vehicle <NUM>. For example, timing component <NUM> is configured to determine a moment in time to present a challenge to the vehicle operator. Once a challenge has been presented, the vehicle operator could take action to respond to or meet the challenge. The challenge and response are used to gauge a level of responsiveness of the vehicle operator. Preparedness corresponds to responsiveness of the vehicle operator. In some implementations, preparedness corresponds to an estimate for how long it would take the vehicle operator to assume manual control of vehicle <NUM>.

In some implementations, determinations by timing component <NUM> are based on one or more operator parameters, driving parameters, and/or other parameters, as well as combinations thereof. In some implementations, determinations by timing component <NUM> are based on one or more of biometric information of the vehicle operator, and operator information related to a direction of view of the vehicle operator. In some implementations, determinations by timing component <NUM> are based on traffic information related to current traffic parameters and environmental information related to surface parameters of the road. In some implementations, determinations by timing component <NUM> are based on one or more of biometric information of the vehicle operator, operator information related to a direction of view of the vehicle operator, traffic information related to current traffic parameters, and environmental information related to surface parameters of the road. In some implementations, determinations by timing component <NUM> are based on one or more determination by other modules, such as a determination of a level of responsiveness of the vehicle operator by responsiveness component <NUM>.

Challenge component <NUM> is configured to present challenges to vehicle operators. In some implementations, challenge component <NUM> presents a challenge to a vehicle operator at a moment in time as determined by timing component <NUM>. For example, a challenge could include one or more of a generation of a sound, an activation of an indicator, and/or another tactile, audible, and/or visual notification to which a vehicle operator is expected to respond, in order to confirm he or she is paying attention in general, and, more specifically, ready to assume manual control of vehicle <NUM> if needed.

In some implementations, the output signals convey biometric information of the vehicle operator, and a determination of timing component <NUM> could be based on the biometric information. For example, detected operator drowsiness could prompt more challenges. In some implementations, the output signals convey operator information related to a direction of view of the vehicle operator, and a determination of timing component <NUM> is based on the operator information. For example, distractedness by the operator could prompt more challenges. In some implementations, the output signals convey traffic information related to current traffic parameters, and a determination of timing component <NUM> is based on the traffic information. For example, crowded streets could warrant more challenges. In some implementations, the output signals convey environmental information related to surface parameters of the road, and a determination of timing component <NUM> is based on the environmental information. For example, wet, icy, and/or uneven roads could prompt more challenges.

Response component <NUM> is configured to detect whether a vehicle operator has responded to a challenge presented by challenge component <NUM>. In some implementations, a response includes one or more of audible feedback (i.e. the vehicle operator can say a particular phrase out loud such that his or her response may be captured through a microphone), a particular gesture or motion, a particular interaction with a user interface (i.e. the vehicle operator can push a button or touch a particular object to indicate a response), and/or other types of responses capable of being interpreted and/or recognized through one or more sensors <NUM> of vehicle <NUM>.

Responsiveness component <NUM> is configured to determine levels of responsiveness of a vehicle operator. In some implementations, determinations by responsiveness component <NUM> are used to determine whether the current mode of operation should be maintained or be changed, e.g., from autonomous to manual operation. In some implementations, determinations by responsiveness component <NUM> are based on detections by response component <NUM>. Alternatively, and/or simultaneously, determinations by responsiveness component <NUM> could be based on (the timing of) presentations by challenge component <NUM>. For example, a level of responsiveness could be based on the elapsed time between a presentation of a challenge and detection of a response to that challenge by the vehicle operator. In some implementations, the elapsed time is used as an estimate for how long it would take the vehicle operator to assume manual control of vehicle <NUM>.

In line with the invention, operation of responsiveness component <NUM> is based on comparisons with one or more thresholds. For example, the level of responsiveness of the vehicle operator is compared with a threshold of adequate responsiveness. For example, if the threshold is met, system <NUM> maintains an autonomous mode of operation. For example, if the threshold is not met, system <NUM> transitions to another mode of operation and/or halt or park vehicle <NUM>. Alternatively, and/or simultaneously, if the threshold is not met, system <NUM> could present additional and/or different challenges that are used to determine whether the current mode of operation should change or stay the same. In line with the invention, the threshold is adjusted dynamically. For example, under more challenging driving conditions, the required level of responsiveness may be higher than compared to less challenging driving conditions. For example, the current level of responsiveness can be used to determine the next moment in time for presenting a challenge and/or the next threshold of adequate responsiveness that the vehicle operator is required to meet in order to maintain the autonomous operation mode.

In some implementations, a value of a threshold level varies as a function of one or more of responsiveness, a parameter, and of other information. For example, a threshold level could increase after one or more challenges. A vehicle operator may be expected to have an increased level of preparedness after one or more challenges have been presented, in particular if some of those challenges were prompted by conditions indicating a reduction in operator confidence.

In some implementations, the required level of responsiveness is adjusted based on measurements and/or determinations spanning a period extending beyond <NUM> minutes, or an hour, or spanning a day, week, month, year, or more. For example, adjustments could be based on a particular vehicle operator's driving history. For example, in some implementations, adjustments are based on the driving history for multiple drivers.

In some implementations, responsiveness is based on more than one challenge-response pair. For example, the past <NUM>, <NUM>, <NUM>, <NUM>, or more such pairs may be aggregated to determine a current level of responsiveness.

Vehicle control component <NUM> is configured to operate vehicle <NUM> in one or more modes of operation. In some implementations, vehicle control component <NUM> is configured to transition vehicle control between different modes of operation. For example, vehicle control component <NUM> transitions vehicle control from an autonomous mode of operation to a manual mode of operation, and/or vice versa. For example, vehicle control component <NUM> transitions vehicle control from an autonomous mode of operation to halt vehicle operation, e.g., by pulling vehicle <NUM> over or parking vehicle <NUM>. In some implementations, operations and/or transitions by vehicle control component <NUM> are based on determinations by responsiveness component <NUM>, and/or based on the operation of other components.

By way of non-limiting example, <FIG> illustrates a top-view of a scenario in which a system similar or the same as system <NUM> of <FIG> is used. At a moment indicated by "t=<NUM>," vehicle <NUM> is driving on a road in an autonomous operation mode of control, in a direction indicated by a dotted arrow. At a subsequent moment indicated by "t=<NUM>," a system of vehicle <NUM> presents a challenge to the operator of vehicle <NUM> to gauge the level of responsiveness of the vehicle operator. At a subsequent moment indicated by "t=<NUM>," the system detects a response by the vehicle operator to the presented challenge. At a subsequent moment indicated by "t=<NUM>," the system determines that the level of responsiveness of the vehicle operator is not adequate to maintain the autonomous vehicle operation. As a result, vehicle <NUM> is guided to the nearest parking place. At a subsequent moment indicated by "t=<NUM>," the system parks vehicle <NUM> in a parking spot and halts autonomous operation of vehicle <NUM>.

Confidence component <NUM> is configured to determine levels of confidence pertaining to one or more modes of vehicle operation. In some implementations, confidence component <NUM> is configured to determine a metric representing a level of operator confidence, also referred to as an operator confidence. Operator confidence is a level of confidence that the vehicle operator is prepared to assume manual control of vehicle <NUM>. In some implementations, operator confidence could be based on the level of responsiveness of the vehicle operator as determined by responsiveness component <NUM>. In some implementations, operator confidence is based on a prediction of how responsive a particular vehicle operator will be. Alternatively, and/or simultaneously, operator confidence could be based on a prediction of the level of responsiveness required to transition vehicle control to the vehicle operator. In some implementations, predictions are based on traffic parameters, weather parameters, and/or other parameters. For example, driving at a higher speed corresponds to a requirement of a higher level of responsiveness, whereas driving at a lower speed corresponds to a requirement of a lower level of responsiveness.

In some implementations, confidence component <NUM> is configured to determine an automation confidence. Automation confidence is a level of confidence that vehicle control should continue autonomous vehicle operation. In some implementations, automation confidence is based on one or more predictions. For example, automation confidence could be based on a prediction of changes in a traffic parameter and/or other parameter. One or more levels of confidence could be used by other components, e.g., by timing component <NUM>. In some implementations, lower levels of confidence correspond to more and/or more frequent challenges to the vehicle operator.

By way of non-limiting example, <FIG> illustrates four scenarios, depicted by graphs <NUM>, <NUM>, <NUM>, and <NUM>, in which a system similar or the same as system <NUM> of <FIG> is used to determine a moment in time for presenting a challenge to a vehicle operator. In these graphs, the horizontal axis represents the passage of time. In graph <NUM>, a heart rate <NUM> of a vehicle operator is depicted. At a moment indicated by reference <NUM>, heart rate <NUM> crosses a threshold heart rate <NUM>. For example, a low heart rate may indicate that a vehicle operator is falling asleep or has fallen asleep. At moment <NUM>, the system may be configured to determine that it is time to present a challenge to the vehicle operator.

In graph <NUM>, an operator confidence <NUM> of a vehicle operator is depicted. At a moment indicated by reference <NUM>, operator confidence <NUM> crosses a threshold confidence level <NUM>. For example, a low level of operator confidence may indicate that a vehicle operator is less responsive than required, and/or that the vehicle operator is predicted to be insufficiently responsive. At a moment indicated by 47a, the threshold level <NUM> may be changed and/or adjusted. For example, such an adjustment may be prompted by an increase in vehicle speed, a decrease in visibility, a decrease in the heart rate of the vehicle operator, and/or by other events or occurrences. At moment <NUM>, the system may be configured to determine that it is time to present a challenge to the vehicle operator.

In graph <NUM>, an automation confidence <NUM> of a system for autonomous vehicle operation is depicted. At a moment indicated by reference <NUM>, automation confidence <NUM> crosses a threshold confidence level <NUM>. For example, a low level of automation confidence may indicate a prediction of increasingly challenging traffic conditions. At a moment indicated by 51a, the threshold level <NUM> may be changed and/or adjusted. For example, such an adjustment may be prompted by an increase in vehicle speed, a decrease in visibility, an increase in traffic density, and/or by other events or occurrences. At moment <NUM>, the system may be configured to determine that it is time to present a challenge to the vehicle operator.

In graph <NUM>, an operator confidence <NUM> of a vehicle operator and an automation confidence <NUM> of a system for autonomous vehicle operation are depicted at the same time. The operator confidence <NUM> is compared to threshold operator confidence level <NUM>. The automation confidence <NUM> is compared to threshold automation confidence level <NUM>. As depicted, the system may be configured to determine that it is time to present a challenge to a vehicle operator if a combination of two events occur: responsive to both the operator confidence <NUM> and the automation confidence <NUM> breaching their respective threshold levels, a challenge will be presented to the vehicle operator. As depicted in graph <NUM>, at a moment indicated by 57a, the automation confidence level <NUM> crosses the threshold automation confidence level <NUM>. Since the operator confidence level <NUM> is sufficient (i.e., has not breached the threshold operator confidence level <NUM>), no determination is made that a challenge should be presented at that time. However, at a moment indicated by 57b, the automation confidence level <NUM> again crosses the threshold automation confidence level <NUM>. At this time, the operator confidence level <NUM> is no sufficient (i.e., it has breached the threshold operator confidence level <NUM> at moment 56a). Accordingly, at moment 57b, a determination is made that a challenge should be presented to the vehicle operator.

Referring to <FIG>, record component <NUM> is configured to capture, record, store, transmit, and/or process information, including but not limited to information related to vehicle operation. In some implementations, information related to vehicle operation is used to generate and/or create vehicle event records. Vehicle event records include one or more of video information, audio information, data from an ECM system, metadata, timing information, information based on sensor-generated output, and/or other information. For example, changes in the operation mode of vehicle <NUM> could be used to create a vehicle event record. In particular, information regarding transitions from autonomous to manual operation may be captured, recorded, stored, transmitted, and/or otherwise processed. In some implementations, record component <NUM> is configured to capture information conveyed by the output signals proximate in time to presentation of one or more particular challenges and/or responses.

Vehicle event records are generated and/or stored locally in vehicle <NUM> and/or transmitted from vehicle <NUM> to system <NUM>, server <NUM>, and/or to a service that is external to the vehicle, including but not limited to a remote server and/or external provider <NUM>. In some implementations, vehicle event records are generated and/or stored remotely, i.e., not locally at vehicle <NUM>. In some implementations, system <NUM>, server <NUM>, and/or a service that is external to vehicle <NUM> queries and/or requests information from a particular vehicle <NUM>. Record component <NUM> is configured to respond to a query or request by transmitting information as queried and/or requested. In some implementations, record component <NUM> is configured to facilitate communication of information between particular vehicles, remote servers, external providers, and/or other systems, servers, and/or services external to the particular vehicles. Such communication may be in real-time or near real-time. Such communication may include wireless communication.

Transmission component <NUM> is configured to transmit information, e.g., to one or more remote servers that are external to vehicle <NUM>. For example, transmission component <NUM> transmits vehicle event records. Vehicle event records are stored locally in vehicle <NUM> and/or transmitted from vehicle <NUM> to system <NUM>, server <NUM>, and/or a service that is external to vehicle <NUM>, including but not limited to a remote server and/or an external provider <NUM>. In some implementations, a system, server, and/or a service that is external to the vehicle queries and/or requests information from vehicle <NUM>. Transmission component <NUM> is configured to respond to a query or request by transmitting information as queried and/or requested. In some implementations, transmission component <NUM> is configured to facilitate communication of information between vehicles, remote servers, external providers, and/or other systems, servers, and/or services external to vehicles. Communication may be in real-time or near real-time. In some implementations, communication is wireless.

Transmission component <NUM> is configured to generate and/or determine notifications related to vehicle operation. In some implementations, notifications are intended for drivers of vehicles. For example, transmission component <NUM> receives transmissions, such as notifications for drivers, including but not limited to warnings or requests (for example to reduce speed). In some implementations, notifications are transmitted from vehicle <NUM> to a system, server, and/or a service that is external to the vehicle, including but not limited to a remote server and/or an external provider <NUM>.

In some implementations, system <NUM> includes a user interface configured to provide an interface between system <NUM> and users through which the users can provide information to and receive information from system <NUM>. This enables information to be communicated between a user and one or more of processor <NUM>, sensors <NUM>, vehicle <NUM>, and/or other components of system <NUM>.

Examples of interface devices suitable for inclusion in a user interface include a keypad, buttons, switches, a keyboard, knobs, levers, a display screen, a touch screen, speakers, a microphone, an indicator light, an audible alarm, a printer, a tactile feedback device, and/or other interface devices.

It is to be understood that other communication techniques, either hard-wired or wireless, are also contemplated by the present disclosure as a user interface. Information may be loaded into system <NUM> wirelessly from a remote location, from removable storage (e.g., a smart card, a flash drive, a removable disk, etc.), and/or information to be communicated between a user and one or more of processor <NUM>, sensors <NUM>, vehicle <NUM>, and/or other components of system <NUM>.

It is to be understood that other communication techniques, either hard-wired or wireless, are also contemplated by the present invention as a user interface. Information may be loaded into system <NUM> wirelessly from a remote location, from removable storage (e.g., a smart card, a flash drive, a removable disk, etc.), and/or other sources that enable the user(s) to customize the implementation of system <NUM>. Other exemplary input devices and techniques adapted for use with system <NUM> include, but are not limited to, an RS-<NUM> port, RF link, an IR link, modem (telephone, cable, and/or other modems), a cellular network, a Wi-Fi network, a local area network, and/or other devices and/or systems. In short, any technique for communicating information with system <NUM> is contemplated by the present invention as a user interface.

Electronic storage <NUM> comprises electronic storage media that electronically stores information. The electronic storage media of electronic storage <NUM> may comprise one or both of system storage that is provided integrally (i.e., substantially non-removable) with system <NUM> and/or removable storage that is removably connectable to system <NUM> via, for example, a port (e.g., a USB port, a firewire port, etc.) or a drive (e.g., a disk drive, etc.). Electronic storage <NUM> may comprise one or more of optically readable storage media (e.g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.), electrical charge-based storage media (e.g., EEPROM, RAM, etc.), solid-state storage media (e.g., flash drive, etc.), and/or other electronically readable storage media. Electronic storage <NUM> may store software algorithms, recorded video event data, information determined by processor <NUM>, information received via a user interface, and/or other information that enables system <NUM> to function properly. Electronic storage <NUM> may be (in whole or in part) a separate component within more of a digital processor, an analog processor, a digital circuit designed to process information, an analog circuit designed to process information, a state machine, and/or other mechanisms for electronically processing information. Although processor <NUM> is shown in <FIG> as a single entity, this is for illustrative purposes only. In some implementations, processor <NUM> may comprise a plurality of processing units. These processing units may be physically located within the same device (e.g., a vehicle event recorder), or processor <NUM> may represent processing functionality of a plurality of devices operating in coordination.

Processor <NUM> is configured to execute components <NUM>-<NUM> by software; hardware; firmware; some combination of software, hardware, and/or firmware; and/or other mechanisms for configuring processing capabilities on processor <NUM>. It should be appreciated that although components <NUM>-<NUM> are illustrated in <FIG> as being co-located within a single processing unit, in implementations in which processor <NUM> comprises multiple processing units, one or more of components <NUM>-<NUM> may be located remotely from the other components. The description of the functionality provided by the different components <NUM>-<NUM> described herein is for illustrative purposes, and is not intended to be limiting, as any of components <NUM>-<NUM> may provide more or less functionality than is described. For example, one or more of components <NUM>-<NUM> may be eliminated, and some or all of its functionality may be provided by other components <NUM>-<NUM>. As another example, processor <NUM> may be configured to execute one or more additional components that may perform some or all of the functionality attributed below to one of components <NUM>-<NUM>.

<FIG> illustrates a method <NUM> to transition vehicle control between autonomous operation and manual operation. The operations of method <NUM> presented below are intended to be illustrative. In some implementations, method <NUM> is accomplished with one or more additional operations not described, and/or without one or more of the operations discussed. Additionally, the order in which the operations of method <NUM> are illustrated (in <FIG>) and described below is not intended to be limiting. In some implementations, two or more of the operations may occur substantially simultaneously.

In some implementations, method <NUM> may be implemented in one or more processing devices (e.g., a digital processor, an analog processor, a digital circuit designed to process information, an analog circuit designed to process information, a state machine, and/or other mechanisms for electronically processing information). The one or more processing devices may include one or more devices executing some or all of the operations of method <NUM> in response to instructions stored electronically on one or more electronic storage mediums. The one or more processing devices may include one or more devices configured through hardware, firmware, and/or software to be specifically designed for execution of one or more of the operations of method <NUM>.

Referring to <FIG> and method <NUM>, at an operation <NUM>, output signals are generated by a set of sensors conveying information related to vehicle operation. In some embodiments, operation <NUM> is performed by a set of sensors the same as or similar to sensors <NUM> (shown in <FIG> and described herein).

At an operation <NUM>, the vehicle is operated autonomously in an autonomous operation mode. In some embodiments, operation <NUM> is performed by an autopilot component the same as or similar to autopilot component <NUM> (shown in <FIG> and described herein).

At an operation <NUM>, a first moment in time is determined for presenting a challenge to a vehicle operator to gauge a level of responsiveness of the vehicle operator. Determination of the first moment in time is based on the output signals. In some embodiments, operation <NUM> is performed by a timing component the same as or similar to timing component <NUM> (shown in <FIG> and described herein).

At an operation <NUM>, at the first moment in time, the challenge is presented to the vehicle operator to gauge the level of responsiveness of the vehicle operator. In some embodiments, operation <NUM> is performed by a challenge component the same as or similar to challenge component <NUM> (shown in <FIG> and described herein).

At an operation <NUM>, whether the vehicle operator provides a response to the challenge is detected. In some embodiments, operation <NUM> is performed by a response component the same as or similar to response component <NUM> (shown in <FIG> and described herein).

At an operation <NUM>, the level of responsiveness of the vehicle operator is determined based on the detection. In some embodiments, operation <NUM> is performed by a responsiveness component the same as or similar to responsiveness component <NUM> (shown in <FIG> and described herein).

At an operation <NUM>, the vehicle is operated based on the determined level of responsiveness. In some embodiments, operation <NUM> is performed by a vehicle control component the same as or similar to vehicle control component <NUM> (shown in <FIG> and described herein).

Claim 1:
A system (<NUM>) configured to transition vehicle control between autonomous operation and manual operation, the system (<NUM>) configured to couple with a vehicle (<NUM>), the system (<NUM>) comprising:
a set of sensors (<NUM>) configured to generate (<NUM>) output signals conveying information related to vehicle operation; and
one or more processors (<NUM>) configured to:
operate (<NUM>) the vehicle (<NUM>) autonomously in an autonomous operation mode;
determine (<NUM>) a first moment in time for presenting a challenge to a vehicle operator to gauge a level of responsiveness of the vehicle operator, wherein determination of the first moment in time is based on the output signals;
present (<NUM>), at the first moment in time, the challenge to the vehicle operator to gauge the level of responsiveness of the vehicle operator;
detect (<NUM>) whether the vehicle operator provides a response to the challenge;
determine (<NUM>) the level of responsiveness of the vehicle operator based on elapsed time between presentation of the challenge and the detection of the response; and
operate (<NUM>) the vehicle based on the determined level of responsiveness,
wherein the level of responsiveness of the vehicle operator is compared with a threshold of adequate responsiveness that is adjusted dynamically.