System and method for detecting severe road events

The present technology is effective to cause at least one processor to collect sensor data from at least one sensor on an autonomous vehicle, wherein the sensor data includes a plurality of measurements from the at least one sensor, identify, from the sensor data, at least one measurement from the plurality of measurements that is outside a threshold measurement for the at least one sensor and is indicative of an impact incident, send the sensor data to a remote computing system, and receive, in response to the sending of the sensor data that is indicative of the impact incident, routing instructions from the remote computing system.

TECHNICAL FIELD

The present technology relates to detecting severe road events and more particularly to identifying measurements in sensor data that indicate impact incidents.

BACKGROUND

An autonomous vehicle is a motorized vehicle that can navigate without a human driver. An exemplary autonomous vehicle includes a plurality of sensor systems, such as, but not limited to, a camera sensor system, a lidar sensor system, a radar sensor system, amongst others, wherein the autonomous vehicle operates based upon sensor signals output by the sensor systems. Specifically, the sensor signals are provided to an internal computing system in communication with the plurality of sensor systems, wherein a processor executes instructions based upon the sensor signals to control a mechanical system of the autonomous vehicle, such as a vehicle propulsion system, a braking system, or a steering system.

Over the lifetime of a vehicle, the vehicle will likely encounter an impact incident, that is severe enough that it may cause damage to the vehicle. Skilled human drivers can identify the impact using their own senses. Furthermore, inspections for the damage to the vehicle is costly, time-consuming, and inefficient.

DETAILED DESCRIPTION

Various examples of the present technology are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations may be used without parting from the spirit and scope of the present technology. In some instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more aspects. Further, it is to be understood that functionality that is described as being carried out by certain system components may be performed by more or fewer components than shown.

Over the lifetime of a vehicle, the vehicle will likely encounter an impact incident that is severe enough that it may have caused damage to the vehicle. For example, a vehicle might traverse over a pothole that may result in a bent tie-rod, flat tire, bent rim, etc. Skilled human drivers can identify the impact using their own senses. Some skilled human drivers may also identify the damage to the vehicle. Other human drivers will need to bring the vehicle to a mechanic or other inspection facility to identify the damage to the vehicle. These inspections for the damage to the vehicle is costly, time-consuming, and inefficient.

Autonomous vehicles eliminate the need for human drivers, and therefore vehicles are often without any personnel that can assess the functioning of the autonomous vehicle at unscheduled times. Autonomous vehicles may utilize regularly scheduled vehicle inspections to check for needed services or repairs, but such scheduled inspections are not well suited to identifying service needs resulting from acute events such as impact incidents. Autonomous vehicles also have many sensors that can, in some specific situations, identify instances where unplanned service is needed, but these sensors may not provide sufficient data to accurately determine when a vehicle needs service. In fact, impact evets are likely to cause changes in some sensor readings, but the vehicle might not need service. In other words, relying on sensor data that identifies an impact event and/or that some change in base line sensor readings has changed can result in falsely identifying that service is needed. Avoiding extra servicing is important since these inspections are costly due to the manual labor involved and reduce vehicle availability. Conversely, sensor data might also fail to identify that a service is needed, which can result in the autonomous vehicle traversing roads while damaged, which may result in further damage to the autonomous vehicle and/or danger to passengers and/or pedestrians.

Accordingly, the disclosed technology addresses the need for a more advanced method and system to gather data from sensors of the autonomous vehicle, analyze the data, and determine whether a severe road event has occurred that requires servicing the vehicle or whether the vehicle can continue operations without servicing following an impact event.

FIG.1illustrates environment100that includes an autonomous vehicle102in communication with a remote computing system150.

The autonomous vehicle102can navigate about roadways without a human driver based upon sensor signals output by sensor systems104-106of the autonomous vehicle102. The autonomous vehicle102includes a plurality of sensor systems104-106(a first sensor system104through an Nth sensor system106). The sensor systems104-106are of different types and are arranged about the autonomous vehicle102. For example, the first sensor system104may be a camera sensor system, and the Nth sensor system106may be a lidar sensor system. Other exemplary sensor systems include radar sensor systems, global positioning system (GPS) sensor systems, inertial measurement units (IMU), infrared sensor systems, laser sensor systems, sonar sensor systems, linear and/or rotary potentiometers, and the like.

The autonomous vehicle102further includes several mechanical systems that are used to effectuate appropriate motion of the autonomous vehicle102. For instance, the mechanical systems can include but are not limited to, a vehicle propulsion system130, a braking system132, and a steering system134. The vehicle propulsion system130may include an electric motor, an internal combustion engine, or both. The braking system132can include an engine brake, brake pads, actuators, a regenerative braking system in an electric vehicle, and/or any other suitable componentry that is configured to assist in decelerating the autonomous vehicle102. The steering system134includes suitable componentry that is configured to control the direction of movement of the autonomous vehicle102during navigation.

The autonomous vehicle102further includes a safety system136that can include various lights and signal indicators, parking brake, airbags, etc. The autonomous vehicle102further includes a cabin system138that can include cabin temperature control systems, in-cabin entertainment systems, etc.

The autonomous vehicle102additionally comprises an internal computing system110that is in communication with the sensor systems104-106and the systems130,132,134,136, and138. The internal computing system includes at least one processor and at least one memory having computer-executable instructions that are executed by the processor. The computer-executable instructions can make up one or more services responsible for controlling the autonomous vehicle102, communicating with remote computing system150, receiving inputs from passengers or human co-pilots, logging metrics regarding data collected by sensor systems104-106and human co-pilots, etc.

The internal computing system110can include a control service112that is configured to control the operation of the vehicle propulsion system130, the braking system132, the steering system134, the safety system136, and the cabin system138. The control service112receives sensor signals from the sensor systems104-106as well communicates with other services of the internal computing system110to effectuate operation of the autonomous vehicle102. In some embodiments, control service112may carry out operations in concert one or more other systems of autonomous vehicle102.

The internal computing system110can also include a constraint service114to facilitate safe propulsion of the autonomous vehicle102. The constraint service114includes instructions for activating a constraint based on a rule-based restriction upon operation of the autonomous vehicle102. For example, the constraint may be a restriction upon navigation that is activated in accordance with protocols configured to avoid occupying the same space as other objects, abide by traffic laws, circumvent avoidance areas, etc. In some embodiments, the constraint service can be part of the control service112.

The internal computing system110can also include a communication service116. The communication service can include both software and hardware elements for transmitting and receiving signals from/to the remote computing system150. The communication service116is configured to transmit information wirelessly over a network, for example, through an antenna array that provides personal cellular (long-term evolution (LTE), 3G, 5G, etc.) communication.

In some embodiments, one or more services of the internal computing system110are configured to send and receive communications to remote computing system150for such reasons as reporting data for training and evaluating machine learning algorithms, requesting assistance from remoting computing system or a human operator via remote computing system150, software service updates, ridesharing pickup and drop off instructions etc.

The internal computing system110can also include a latency service118. The latency service118can utilize timestamps on communications to and from the remote computing system150to determine if a communication has been received from the remote computing system150in time to be useful. For example, when a service of the internal computing system110requests feedback from remote computing system150on a time-sensitive process, the latency service118can determine if a response was timely received from remote computing system150as information can quickly become too stale to be actionable. When the latency service118determines that a response has not been received within a threshold, the latency service118can enable other systems of autonomous vehicle102or a passenger to make necessary decisions or to provide the needed feedback.

The internal computing system110can also include a user interface service120that can communicate with cabin system138in order to provide information or receive information to a human co-pilot or human passenger. In some embodiments, a human co-pilot or human passenger may be required to evaluate and override a constraint from constraint service114, or the human co-pilot or human passenger may wish to provide an instruction to the autonomous vehicle102regarding destinations, requested routes, or other requested operations.

As described above, the remote computing system150is configured to send/receive a signal from the autonomous vehicle102regarding reporting data for training and evaluating machine learning algorithms, requesting assistance from remote computing system150or a human operator via the remote computing system150, software service updates, rideshare pickup and drop off instructions, etc.

The remote computing system150includes an analysis service152that is configured to receive data from autonomous vehicle102and analyze the data to train or evaluate machine learning algorithms for operating the autonomous vehicle102. The analysis service152can also perform analysis pertaining to data associated with one or more errors or constraints reported by autonomous vehicle102.

The remote computing system150can also include a user interface service154configured to present metrics, video, pictures, sounds reported from the autonomous vehicle102to an operator of remote computing system150. User interface service154can further receive input instructions from an operator that can be sent to the autonomous vehicle102.

The remote computing system150can also include an instruction service156for sending instructions regarding the operation of the autonomous vehicle102. For example, in response to an output of the analysis service152or user interface service154, instructions service156can prepare instructions to one or more services of the autonomous vehicle102or a co-pilot or passenger of the autonomous vehicle102.

The remote computing system150can also include a rideshare service158configured to interact with ridesharing application170operating on (potential) passenger computing devices. The rideshare service158can receive requests to be picked up or dropped off from passenger ridesharing app170and can dispatch autonomous vehicle102for the trip. The rideshare service158can also act as an intermediary between the ridesharing app170and the autonomous vehicle wherein a passenger might provide instructions to the autonomous vehicle102to go around an obstacle, change routes, honk the horn, etc.

FIG.2shows an example environment200, in which a severe road event is detected. More specifically, an autonomous vehicle102is traversing a street202that has a pothole204. The autonomous vehicle102heading leftwards has driven over the pothole204and may have been damaged. As the autonomous vehicle102drives over the pothole204, the sensor systems104-106detect measurements of various sensors and components of the autonomous vehicle102. For example, the autonomous vehicle102may have suspension sensors104-106that detect vertical displacement of the autonomous vehicle102from the ground. In some embodiments, the suspension sensors104-106may also detect changes in the vertical displacement of the autonomous vehicle102over a period of time, such that a large change in vertical displacement in a short period of time may indicate that the autonomous vehicle102has driven over a pothole or an object, such as a manhole cover. In some embodiments, the autonomous vehicle102may continue to receive additional sensor data from other sensor systems104-106after driving over the pothole204. For example, an alignment sensor may determine that the autonomous vehicle102is no longer driving in a straight line after driving over the pothole204. As another example, the sensor systems104-106may monitor acceleration and/or gravitational force equivalents (g-force) of the autonomous vehicle102as the autonomous vehicle traverses over the pothole204. The acceleration of the autonomous vehicle102may abruptly decline while the autonomous vehicle102traverses over the pothole204then return to normal. Similarly, the autonomous vehicle102may detect a sudden change in g-force in the direction that the autonomous vehicle102is traversing over the pothole204. Thus, the sensor systems104-106may further an accelerometer and an inertial measurement unit IMU) sensor.

The autonomous vehicle102, via communication service116, may be in communication with a remote computing system150. Accordingly, the autonomous vehicle102sends, via communication service116, to the remote computing system the sensor data collected from the sensor systems104-106. The remote computing system104-106may then return routing instructions to the autonomous vehicle102. For example, if the autonomous vehicle102and/or the remote computing system150determine, based on the measurements in the sensor data, that the autonomous vehicle102has encountered a severe road event, the remote computing system150may send routing instructions to the autonomous vehicle102to guide the autonomous vehicle102to an inspection facility and/or a repair facility.

It is further contemplated that the autonomous vehicle102is an autonomous vehicle in a fleet of autonomous vehicles102. As shown, the autonomous vehicle102driving rightwards does not encounter the severe road event. Thus, the autonomous vehicle102may provide sensor data that indicates a baseline measurement that the measurements of the damaged vehicle may be measured or compared against.

It is further contemplated that the sensor systems104-106of the autonomous vehicle102may detect vandalism to the autonomous vehicle102. For example, a vandal may throw a rock at and shatter windows of the autonomous vehicle102. A microphone or other glass breakage sensor may detect that the windows of the autonomous vehicle102has been damaged. The autonomous vehicle102may then communicate the sensor data to the remote computing system150and receive routing instructions to a repair facility in response to the communication.

FIG.3shows an example method300implemented by a remote computing system150for detecting a severe road event. Specifically, method300addresses that remote computing system150can collect data from autonomous vehicles in a fleet of autonomous vehicles and can use this data to identify and/or learn of impact events experienced by respective vehicles in the fleet. Thereafter, the respective vehicles experiencing impact events can be inspected by a technician or mechanic, who can enter data regarding whether a repair was needed, the nature of the repair, and/or a rating of the severity of the damage. Using this data, an algorithm can be created (human programmed heuristics, a machine learning algorithm, etc.) and send the algorithm to the autonomous vehicles in the fleet so that the individual autonomous vehicles can make determinations regarding the severity of future impact events.

The method300begins with the remote computing system150receiving302sensor data from autonomous vehicles102in a fleet of autonomous vehicles. The sensor data may indicate an impact incident experienced by a respective autonomous vehicle in the fleet and indicating measurement from sensors on the autonomous vehicle from before, during, and after the impact event.

The remote computing system150then receives304maintenance data from a user device associated with a mechanic after the mechanic has evaluated the respective autonomous vehicle in the fleet following the impact incident. The maintenance data indicates any necessary maintenance, the nature of the maintenance, and/or a rating of the severity of the damage.

The remote computing system150then creates306an algorithm from the sensor data and the maintenance data. The algorithm can receive sensor data following an impact incident and classify the impact incident as one that likely requires vehicle maintenance or one that does not likely require vehicle maintenance.

In some embodiments, the algorithm may also evaluate, based on the sensor data and the maintenance data, when an autonomous vehicle might need routine maintenance based on a number and severity of impact incidents and miles driven. In some embodiments, this evaluation may be a second algorithm. In other words, the remote computing system may create a second algorithm to evaluate when an autonomous vehicle might need routine maintenance that takes into account a number of impact events (that did not require servicing of the autonomous vehicle) encountered by a respective autonomous vehicle and the severity of those events. In some embodiments, the second algorithm may be for use by the remote computing system150.

In some embodiments, the algorithm may be a machine learning algorithm that takes various inputs, such as the sensor data, the maintenance data, and a classification of a severity of a respective impact event, where the data is collected from a fleet of autonomous vehicles. The inputs are then given to a machine learning model to train the neural network to receive current sensor data indicating an impact event and output a respective classification of the severity of the impact event and determine whether a repair is needed. In some embodiments, the algorithm may also be trained based on miles driven, a number and severity of impact incidents, and necessary maintenance timing and needs, such that when inputted with miles driven and a number and severity of impacts, the algorithm may determine whether the autonomous vehicle may need unscheduled maintenance following an impact event. Similarly, the second algorithm may be a machine learning algorithm that takes the inputs to determine whether the autonomous vehicle may need routine maintenance.

As understood by those of skill in the art, machine-learning based classification techniques can vary depending on the desired implementation. For example, machine-learning classification schemes can utilize one or more of the following, alone or in combination: hidden Markov models, recurrent neural networks (RNNs), convolutional neural networks (CNNs); Deep Learning networks, Bayesian symbolic methods, general adversarial networks (GANs), support vector machines, image registration methods, and/or applicable rule-based systems. Where regression algorithms are used, they can include but are not limited to: a Stochastic Gradient Descent Regressors, and/or Passive Aggressive Regressors, etc.

The remote computing system150then sends308the algorithm to the autonomous vehicles in the fleet of autonomous vehicles102to determine whether an impact event likely requires unscheduled vehicle maintenance.

After the fleet of autonomous vehicles102is running the algorithm that can classify impact events as ones likely requiring maintenance or not, a respective autonomous vehicle102can encounter an impact event and use the algorithm to determine whether the respective autonomous vehicle102likely requires unscheduled maintenance as a result of the impact event. If the algorithm determines that the respective autonomous vehicle102likely needs maintenance, the respective autonomous vehicle102can notify the remote computing system150of the impact event.

In some embodiments, the remote computing system may also send308the second algorithm to the fleet of autonomous vehicles for the respective autonomous vehicle to evaluate then that it might need maintenance. Similarly, after the fleet of autonomous vehicles is running the second algorithm to evaluate when a respective autonomous vehicle102may likely require routine maintenance, the respective autonomous vehicle102may determine, based on miles driven, a number and severity of impact incidents, and necessary maintenance timing and needs, a need for routine maintenance.

The remote computing system150then receives310a communication from the respective autonomous vehicle indicating that an impact event has been detected that likely requires maintenance.

In some embodiments, the remote computing system150may then request312further sensor data. The further sensor data may assist the remote computing system150in distinguishing potential false positives and/or provide additional confirmation that the impact event likely requires maintenance for the respective autonomous vehicle.

The remote computing system150then determines314that the respective autonomous vehicle does require maintenance. In some embodiments, the determination314may be based on a severity determination of the impact event. Accordingly, the impact events may be classified based on severity.

The remote computing system150then instructs316the respective autonomous vehicle to navigate to a particular service station at a time based on the severity determination. More specifically, the remote computing system150may instruct316autonomous vehicles102to the particular service station in an order of severity. In other words, autonomous vehicles102with more severe impact events and/or damage may be instructed316to navigate to the particular service station immediately, while an autonomous vehicle with a less severe impact event may be able to schedule service for a later time. Any data produced from servicing the respective autonomous vehicle102can also be fed into the machine learning model created at306to improve the model.

FIG.4shows an example method400implemented by an autonomous vehicle102for detecting a severe road event.

It is contemplated that the autonomous vehicle102may receive, from the remote computing system150, an algorithm that is trained from a collection of sensor data received from a plurality of autonomous vehicles102in a fleet of autonomous vehicles102including the autonomous vehicle102. The sensor data received from the plurality of autonomous vehicles is labeled with classifications of at least a severity of a respective impact event and any repair that was needed in response to the respective impact event. For example, the algorithm may be the algorithm created by the remote computing system150discussed above.

The method400begins with the autonomous vehicle102collecting402sensor data from at least one sensor on the autonomous vehicle102. The sensor data includes a plurality of measurements from the at least one sensor. For example, the sensor may be a suspension sensor, an accelerometer, and/or an IMU sensor that collects sensor data including roll, yaw, pitch, and vertical displacement measurements. It is further contemplated that the sensor data may be later processed and calibrated based on a condition of a lane that the autonomous vehicle102is traversing. For example, the sensor data may be later processed and calibrated to consider whether the lane is more bumpy than other lanes.

The autonomous vehicle102then identifies404at least one measurement that is outside a threshold measurement for the at least one sensor. The at least one measurement may be indicative of an impact incident. For example, the suspension sensor may have a threshold measurement of 1 inch of vertical displacement. Thus, when the autonomous vehicle102drives over a pothole, the suspension sensor may identify that vertical displacement has increased over 1 inch of vertical displacement from the ground. In other words, the autonomous vehicle identifies that the vertical displacement is outside or exceeds the threshold vertical displacement of 1 inch from the ground. In some embodiments, the threshold measurement may be based upon or calibrated based on the condition of the lane that the autonomous vehicle102is traversing. It is further contemplated that the threshold measurement may be changed based on sensor data collected from previous trips and current trips from the autonomous vehicle102and the fleet of autonomous vehicles102.

In some embodiments, the autonomous vehicle102may initiate406specific actions that identify issues in a component of the autonomous vehicle102. In some embodiments, the specific actions can be in response to receiving312a request for further sensor data inFIG.3. In some embodiments, the component of the autonomous vehicle102may be a component for which the sensor system104-106collects data. For example, the component may be a suspension system and the sensor may be a suspension sensor measuring vertical displacement. Thus, the autonomous vehicle102may initiate406specific actions to allow the at least one sensor to collect data indicating the measurements. For example, the autonomous vehicle102may, when safe (e.g. when no other vehicles are nearby), rapidly accelerate or break hard to identify a suspension component that compresses more than an expected and/or threshold value or measurement.

In some embodiments, the autonomous vehicle102may analyze408data from at least one second sensor to determine that the impact incident resulted in a change in a baseline reading. For example, a suspension sensor on the front right portion of an autonomous vehicle may identify vertical displacement exceeding a threshold vertical displacement. A wheel rotation sensor on the front right wheel may collect data indicating wheel rotation speed of the front right wheel. The wheel rotation sensor may identify that the front right wheel, which is proximate to the suspension sensor is rotating at a different speed from other wheels of the autonomous vehicle, all of which maintain the previous baseline reading. Thus, the wheel rotation sensor indicates that the impact incident resulted in a change in the baseline reading of the wheel rotation sensor.

The autonomous vehicle then sends410the sensor data to the remote computing system150. In some embodiments, the sending410of the sensor data is dependent on the outcome of the analyzing408of the data from the at least one second sensor.

As shown, in some embodiments, after identifying404at least one measurement that is outside of the threshold measurement and is indicative of an impact incident, the autonomous vehicle102may immediately send410the sensor data to the remote computing system150without initiating406specific actions and/or analyzing408data from at least one second sensor.

In some embodiments, the autonomous vehicle102may collect 412 additional sensor data and determine that the additional sensor data indicates a deviation from the sensor data previously collected in relation to the impact incident and is greater than a threshold deviation. More specifically, the additional sensor data is collected after the impact incident occurred, such that the additional sensor data indicates a second impact event from driving over a second hazard in the road. For example, the autonomous vehicle102may continue traversing the lane after hitting the pothole and encounter a second pothole. The suspension sensor of the autonomous vehicle102may continue to collect 412 additional sensor data that indicates increased vertical displacement after hitting the second pothole.

In some embodiments, the additional sensor data may also indicate that the deviation is greater than a threshold deviation. For example, the suspension sensor of the autonomous vehicle102may have a threshold deviation of 0.25″ from a baseline vertical displacement from the ground. After hitting the pothole, the suspension sensor may indicate that the suspension component now sags and deviates 0.5″ from the baseline vertical displacement from the ground. In some embodiments, the threshold deviation may be based upon or calibrated for the condition of the lane that the autonomous vehicle102is traversing.

As shown, in some embodiments, after identifying404at least one measurement that is outside of the threshold measurement and is indicative of an impact incident, the autonomous vehicle102may immediately collect 412 additional sensor data and determine that the additional sensor data indicates a deviation without initiating406specific actions, analyzing408data from at least one second sensor, and/or sending the sensor data410to the remote computing system.

In some embodiments, the autonomous vehicle102may then report414the additional sensor data to the remote computing system150. In some embodiments, the reporting414of the additional sensor data is dependent on the outcome of the analyzing the data from the at least one second sensor. In some embodiments, the reporting414may also include the sensor data originally collected402by the autonomous vehicle102.

The autonomous vehicle102may then receive416routing instructions from the remote computing system150. Depending on whether the autonomous vehicle102requires unscheduled servicing and a severity of the impact event, the routing instructions may guide the autonomous vehicle102to a particular service station at a specific time. In some embodiments, when it is determined that unscheduled servicing is not yet required, the autonomous vehicle102may receive routing instructions that guide the autonomous vehicle102to continue driving without stopping at a particular service station.

Collectively, the methods illustrated inFIG.3andFIG.4provide efficiencies in managing a fleet of autonomous vehicles by first allowing a specific autonomous vehicle to self-assess the severity of an impact incident. In many cases, the specific autonomous vehicle will determine that the impact incident is not of any concern and can continue with operations. However, in some embodiments, the specific autonomous vehicle might determine that the impact event was of a character that was sufficiently likely that the autonomous vehicle would need service, either immediately, or soon. In such embodiments, the specific autonomous vehicle can communicate with the remote computing system150to further assess any damage, confirm whether servicing is needed, and schedule servicing either immediately or in the future, depending on the necessity of the repair.

In some embodiments, the present technology can also be used to avoid impact events. The received sensor data may be interpreted by a remote computing system150to place and characterize a road hazard on a map. In other words, data regarding impact events can also be provided to a mapping service on remote computing system150which can locate persistent hazards, such as potholes, on a map used by the autonomous vehicles to navigate. This information can be used to instruct the autonomous vehicles to utilize lanes that avoid the persistent hazard, to drive in a position of a lane that avoids the persistent hazard, or to minimize the impact of the persistent hazard.

Additionally, the present technology can be used to monitor the evolution of a persistent hazard. If remote computing system150already had data identifying the road hazard, the remote computing system150can use the sensor data to determine that the road hazard is getting more severe. For example, the present technology can analyze data from autonomous vehicles encountering the persistent hazard over time and note how the hazard is evolving (e.g. a pothole might be getting bigger).

FIG.5shows an example of computing system500, which can be for example any computing device making up internal computing system110, remote computing system150, (potential) passenger device executing rideshare app170, or any component thereof in which the components of the system are in communication with each other using connection505. Connection505can be a physical connection via a bus, or a direct connection into processor510, such as in a chipset architecture. Connection505can also be a virtual connection, networked connection, or logical connection.

Example system500includes at least one processing unit (CPU or processor)510and connection505that couples various system components including system memory515, such as read-only memory (ROM)520and random access memory (RAM)525to processor510. Computing system500can include a cache of high-speed memory512connected directly with, in close proximity to, or integrated as part of processor510.

Processor510can include any general purpose processor and a hardware service or software service, such as services532,534, and536stored in storage device530, configured to control processor510as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor510may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

To enable user interaction, computing system500includes an input device545, which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech, etc. Computing system500can also include output device535, which can be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems can enable a user to provide multiple types of input/output to communicate with computing system500. Computing system500can include communications interface540, which can generally govern and manage the user input and system output. There is no restriction on operating on any particular hardware arrangement, and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

The storage device530can include software services, servers, services, etc., that when the code that defines such software is executed by the processor510, it causes the system to perform a function. In some embodiments, a hardware service that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor510, connection505, output device535, etc., to carry out the function.