Intelligent logging

An on-board computing system for determining an opportune time to log data into a first memory. A sensor system collects data of a vehicle's environment. A controller of the on-board computing system logs the data to a first memory when it determines an opportune time to log data to the first memory. The controller holds data in a second memory if it determines it is not an opportune time to log data into the first memory. The controller resumes logging data to the first memory when an opportune time presents itself.

BACKGROUND

The present disclosure relates to systems, components, and methodologies for logging data through sensors in a vehicle. More particularly, the present disclosure relates to systems, components, and methodologies that improve logging data through sensors in a vehicle in situations such as a high traffic density environment.

SUMMARY

According to the present disclosure, systems, components, and methodologies are provided for logging data in an intelligent manner to reduce the computational load on an on-board computing system.

In illustrative embodiments, an on-board computing system of an autonomous vehicle assesses the criticality of a situation before determining whether or not to log data to a first memory or hold data in a second memory until a more opportune time presents itself to log data to a first memory. This is because logging data to a first memory that may be embodied as a hard drive is orders of magnitude slower than holding data in a second memory that may be embodied as RAM. The second memory in this case can be accessed hundreds of times faster than the first memory. Therefore, in potential critical situations when a faster cycle of processing is needed, the second memory holds the data in order to speed up the process of data logging during that duration in order to address the potential critical situation.

The assessment of criticality of the situation may be based upon a determination if a number of objects in the near vicinity of the autonomous vehicle exceed a predetermined threshold value. The assessment of criticality of the situation may also be based upon a determination if a predicted time to collision of the autonomous vehicle with another object falls below a predetermined threshold value. If there is a determined potential critical situation, the on-board computing system holds data in the second memory to reduce the computational load and allow the on-board computing system to react faster to a potential critical situation.

DETAILED DESCRIPTION

FIG. 1depicts an autonomous vehicle10driving on a roadway104having four lanes104a,104b,104c, and104d. Several neighboring vehicles106,108,110, and112are driving in proximity of the autonomous vehicle10, wherein neighboring vehicle112carries an exposed cargo114.

The autonomous vehicle10may include an on-board computing system200(depicted inFIG. 2and to be described in more detail below). The on-board computing system200may include a front camera212and a rear camera214that may capture image data12of the proximity of the autonomous vehicle10. Thus, for example, the front camera212may capture image data12of the neighboring vehicles110and112, both of which may be located generally forward of the vehicle10. Similarly, the rear camera214may capture image data12of the neighboring vehicles106and108, both of which may be located generally rearward of the vehicle10.

The on-board computing system200may use image data12of the neighboring vehicles106,108,110, and112to develop an assessment of criticality of a given situation to determine if is an opportune time to log data12into a first memory208or hold data12in a second memory206. The assessment of criticality of the situation may include identifying objects in the near vicinity of the autonomous vehicle10to see if the number of objects exceeds a predetermined threshold value. The assessment of criticality of the situation may also include evaluating a time to collision with a neighboring vehicle106,108,110, or112to determine if the time to collision falls below a predetermined threshold value. In the illustrative embodiment, if it is deemed not to be an opportune time to log data12into a first memory208based on the assessment of criticality because the predetermined threshold value was exceeded, then second memory206may hold data12until an opportune time presents itself, i.e., the threshold is not exceeded. The on-board computing system200may also compare the amount of space on second memory206and the assessment of criticality of the situation to determine if it is necessary to log data12into first memory208.

As a result, the on-board computing system200may provide improved safety and efficiency. With respect to safety, the on-board computing system200enables the autonomous vehicle10to log data12in a situation where it is less likely to collide with a neighboring vehicle106,108,110, or112. With respect to efficiency, the on-board computer system200logs data12in situations that are less computationally intensive on the processor204to process the data12and store the data12into first memory208.

FIG. 2is a diagrammatical view of the illustratively embodied on-board computing system200in accordance with the present disclosure. The on-board computing system200may include the controller202having a processor204, first memory208, and a second memory206. In accordance with a main embodiment, the on-board computing system200may also include a sensor system210that includes the previously described cameras212,214, Lidar216, radar218, and other sensors220. The illustratively embodied on-board computing system200may also include a steering and acceleration/braking system222and a human machine interface224, side mirror adjustment system with proximity detectors, a headlight control system with proximity detectors, a window control system with proximity detectors, an information and entertainment system with proximity detectors, a climate control system with proximity detectors, a gear and power train adjustment system with proximity detectors, an audio control system, and a multifunction display control system. Lidar technology collects data using remote sensing technology to measure distance by illuminating a target with a laser and analyzing the reflected light.

In other, additional and/or optional embodiments, the other sensors may include, for example, microphones, air and particulate detector, etc.

In accordance with disclosed embodiments, controller202may be electrically coupled to the sensor system210and the electrical systems220,222,224,226,228,230,232, and234. The electrical connections can be made using any mechanism known in the art, such as a communication bus.

The illustratively embodied on-board computing system200may use the controller202to process the electrical systems222and224and the sensor system210to send data12to the controller202to log into first memory208or hold data12in second memory206until the processor204can log data12in first memory208.

FIG. 3depicts a highway300having an on-ramp302connecting with lanes306of highway300. An autonomous vehicle10entering highway300from an on-ramp302may capture data12regarding secondary vehicles304on highway300. The data12may be used to assess the criticality of the situation to determine if it is an opportune time to log the data12into first memory208or hold the data12in second memory206until an opportune time presents itself. For example, if the amount of secondary vehicles304exceeds a predetermined threshold value, then the autonomous vehicle10may stop logging data12regarding its environment.

In the illustrative embodiment ofFIG. 3, data12may include information regarding thermal emissions311,312, light emissions313,314, audio emissions315, and radio emissions316,317from secondary vehicle304, as illustrated inFIG. 4. These emissions311-317may be logged by the autonomous vehicle10to analyze parameters indicative to activity in a vehicle environment (e.g. the surrounding thermal data, the surrounding light data, the surrounding audio data, etc.). In some embodiments, data12regarding multiple secondary vehicles304in a group of adjacent secondary vehicles304may be captured to analyze the parameters indicative of the activity in the vehicle environment (e.g. the surrounding thermal data, the surrounding light data, the surrounding audio data, etc.). The data12may be used to determine the distance, relative velocity, relative acceleration, and predicted time to collision with the secondary vehicles304in order to assess the criticality of the situation to determine if it is an opportune time to log data12into the first memory208or hold data12in the second memory206until an opportune time presents itself as described in the disclosure.

In the illustrative embodiment, a notification16of traffic ahead may be displayed to a user of the autonomous vehicle10if the probability that adjacent secondary vehicles304are in a traffic jam, as determined by the on-board computing system200, reaches or exceeds a predetermined threshold limit as shown inFIG. 5. In some embodiments, a prompt18may be displayed to the user to activate an autonomous driving function of vehicle10which may operate when vehicles10,304are in a traffic jam. In some embodiments, identification of a traffic jam may prompt autonomous vehicle10to send location data12to a server for mapping traffic patterns. Other uses for identification of traffic jams are also contemplated.

FIG. 6depicts a residential roadway400having lanes406. Autonomous vehicle10driving on roadway400may capture data12regarding secondary vehicles404positioned alongside a curb402of right-side lane406. An opening408for autonomous vehicle10to park in may also be identified as part of an auto-park function of autonomous vehicle10if it is determined that secondary vehicles404are also parked. The data12may be used to determine the distance, relative velocity, relative acceleration, and predicted time to collision with the secondary vehicles404in order to assess the criticality of the situation to determine if it is an opportune time to log the data12into first memory208or hold the data12in second memory206until an opportune time presents itself. For example, if the amount of secondary vehicles404exceeds a predetermined threshold value then the autonomous vehicle10may stop logging data12of its environment. However, in an illustrative embodiment, the on-board computing system200may determine that it is safe to log data12in first memory208if there is a low potential for a collision as a result of the secondary vehicle404.

As shown inFIG. 7, data12captured by autonomous vehicle10may indicate that secondary vehicle404lacks any active-status indicators, making the probability unlikely that secondary vehicles404are a possible object to collide with. As a result, the on-board computing system200may log data12to first memory208after assessing the criticality of the situation. Location data12of autonomous vehicle10may indicate that the autonomous vehicle10is travelling on residential roadway400, as depicted inFIG. 8, which may confirm that the probability of secondary vehicles404being part of a traffic jam is low and more likely that the secondary vehicles404are actually parked.

In the illustrative embodiment, a notification17that pedestrians may be present may be displayed to a user of autonomous vehicle10if it is determined by the on-board computing system200that secondary vehicles404are parked on a residential or other non-controlled access roadway as shown inFIG. 8. A detection that a number of pedestrians may be present may cause the on-board computing system200to stop logging data12in first memory208and hold data in second memory206as a result of the number of objects in the vicinity of the autonomous vehicle exceeding a predetermined threshold value. In some embodiments, a prompt19may be displayed to the user to activate the auto-park function of vehicle10to guide autonomous vehicle10into opening408. In some embodiments, access to the autonomous driving function of autonomous vehicle10may be blocked or prohibited if it is determined that autonomous vehicle10is on a residential or other non-controlled access roadway.

In an illustrative embodiment, the autonomous driving function of a primary vehicle may use Lidar or radar based cruise control to maintain spacing from other secondary vehicles on the roadway. However, such a Lidar or radar based system may not be able to distinguish objects smaller than a vehicle, such as a pedestrian or bicycle user. In such an embodiment, access to the autonomous driving function may be blocked, or prohibited, if the primary vehicle is on a non-controlled access roadway where pedestrians are likely to be present.

The on-board computing system200may include certain components for detecting and analyzing characteristics of secondary vehicles304,404. A sensor system210may be provided on autonomous vehicle10and configured to capture data12including emissions311-317of secondary vehicles304,404. In an illustrative embodiment, sensor system210may include the cameras212,214for obtaining image data12regarding light emissions313,314and image data12regarding thermal emissions311,312, such as through infrared signals for example, and a radio receiver for obtaining signal data12regarding radio emissions316,317coming from secondary vehicles304,404. Sensor system210may be coupled to autonomous vehicle10in an area where a wide range of views are visible, such as, for example, a bumper, hood, roof, side mirror, rear-view mirror, front fascia, or dashboard13of autonomous vehicle10, etc.

In some embodiments that include optional sensors, audio emissions315include passenger voices, such as indicated at315inFIG. 4, sounds from an entertainment system, engine noise, and braking noise, to name a few. In some embodiments, radio emissions316,317include distance sensor signals, ultrasonic parking signals, blind spot radar, and adaptive cruise control radar such as indicated at317, wi-fi signals, BLUETOOTH™ signals, cellphone signals, and entertainment system signals, such as indicated at316, to name a few. In some embodiments, light emissions313,314may include tail or brake light emissions, such as indicated at313, headlamp or turn signal emissions, such as indicated at314, and internal cabin light emissions, etc. In some embodiments, thermal emissions311,312may include brake heat emissions, such as indicated at311, engine heat emissions, such as indicated at312, cabin heat emissions, and exhaust heat emissions, etc.

Moreover, in accordance with such embodiments, a microphone may be used to detect audio emissions315from particular neighboring vehicles. Typical driving patterns of secondary vehicles304or404may be informed by audio emissions315. For example, if a neighboring vehicle304or404may be emitting sounds suggesting engine trouble, the on-board computing system200may determine that the neighboring vehicle304or404could suddenly decelerate or pull over. If a neighboring vehicle may be emitting sounds suggesting loud music, the on-board computing system200may determine that a driver of the secondary vehicle304or404may be distracted and that the secondary vehicle304or404may drive erratically. The on-board computing system200may use the audio emissions315to assess the criticality of the situation and determine that a collision may occur with a secondary vehicle304or404. If the on-board computing system200determines that the time to collision falls below a predetermined threshold then the on-board computing system200may stop logging data12to first memory208and hold the data12in second memory206until a more opportune to log data12presents itself.

In accordance with embodiments including such optional sensors, an air and particulate detector may be used to measure air composition through, for example, olfactory analysis, akin to how a human may smell odors in the air representing impurities. If it is determined that a particular secondary vehicle304or404may be emitting excessive exhaust, the on-board computing system200may avoid that neighboring vehicle304or404. The air and particulate detector may be of any type suitable for performing chemical or olfactory analysis to detect impurities typically present in air on roadways. The on-board computing system200may use the data12collected from the air and particulate detector to avoid needing to collect more data regarding excessive exhaust from a secondary vehicle304or404. Therefore, the computational load of the on-board computing system200could be reduced.

The on-board computing system200may use the controller202to pre-process signals generated by the sensor system210. For example, the controller202may apply filters to signals transmitted by the sensor system210to remove noise and isolate meaningful data using signal processing techniques. This could reduce the computational load of logging data12to first memory208when the on-board computing system200deems it an opportune time to log data12.

FIGS. 9A-9Cdepict an autonomous vehicle in different driving scenarios.FIG. 9Adepicts the autonomous vehicle10following a neighboring vehicle902at a following distance904to the neighboring vehicle902. The following distance904may be used to assess the criticality of the situation and determine if it is an opportune time to log data12to first memory208. The following distance may be used in an evaluation in a potential time to collision assessment. If the on-board computing system200determines that the time to collision falls below a predetermined threshold amount then the on-board computing system200may stop logging data12to first memory208. If it is determined that a collision is eminent then the autonomous vehicle10may continue to log data12to first memory208.

FIG. 9Bdepicts the autonomous vehicle10on a roadway901in a middle lane901b, and suggests that the autonomous vehicle may switch to the left lane901aor the right lane901c. There are also neighboring vehicles908and910in lanes901cand901a. The assessment of the criticality of the situation of the autonomous vehicle may determine that a time to collision falls below a predetermined threshold value when switching to either lanes901a,901c. In addition, neighboring vehicle908or910may be driving aggressively and may accelerate and decelerate quickly. The on-board computing system200may determine that a potential time to collision may fall below a threshold value because of the aggressive driving nature or a neighboring vehicle908,910. As a result of the time to collision falling below a threshold value, the autonomous vehicle10may stop logging data12to first memory208and hold the data12in second memory206until a more opportune time presents itself.

FIG. 9Cdepicts the autonomous vehicle10at an intersection912, and suggests that the autonomous vehicle10is attempting to execute a left turn. The on-board computing system200may assess the criticality of the situation by analyzing all of the objects at the intersection and determine that the number of objects in the nearby vicinity exceeds a predetermined threshold value. In addition, pedestrians may be detected at the intersection912. The pedestrians may not obey traffic laws and as a result cause the autonomous vehicle10to react in a quick manner to avoid colliding with the pedestrians. Furthermore, the neighboring vehicles914,916,918, and920may drive in an aggressive manner. This may cause the on-board computing system200to assess the criticality of the situation and determine the time to collision falls below a predetermined threshold value. A vehicle918may be carrying exposed cargo922, and the autonomous vehicle10may predict that some of the exposed cargo922may fall off of the vehicle918and be a hazard to avoid. As a result of these scenarios, the on-board computing system200may stop logging data12to first memory208and hold data12in second memory206until a more opportune time presents itself.

FIG. 10is a flow diagram1000illustrating a methodology for operation of an intelligent logging system in accordance with the present disclosures. The illustrative methodology begins with operation1005, in which a controller202of the on-board computing system200queries sensors212,214,216,218and220of the sensor system210. After receiving sensor data12, controller202proceeds with operation1010and processes sensor data12. In operation1015, the controller202plans actions in response to the sensor data12. In this illustrative embodiment, these actions may be related to the functions of the autonomous vehicle10. In operation1020, the on-board computing system200may determine if there is a critical situation in which a predetermined threshold value has been exceeded. The assessment of the criticality of a situation may be based upon the number of objects in the vicinity of an autonomous vehicle10or a time to collision as described above. If there is a critical situation as a result of a predetermined threshold value being exceeded then operation may continue to operation1025and data12may be held in second memory206. If not, operation1030may be executed and data12may be logged into first memory208.

In operation1025, there may be an evaluation to see if the data12held in second memory206is reaching a critical amount and preventing the controller202from processing functions for other electrical systems. The sensor data12may be held in second memory206if it is determined that storage in second memory206has not reached a critical amount and operation returns to operation1005to restart the process. The frequency of the processing may be increased to improve the reaction time of the autonomous vehicle10to potential critical situations. Until operation proceeds to1030, data may be held in second memory206. If the controller202determines that the data12stored on second memory206has exceeded a determined threshold value then operations may proceed to operation1030, and the on-board computing system200may start logging data12to first memory208to reduce the computational load on the on-board computing system200. After data12is logged to first memory208, operation1035may be executed and the on-board computing system200may wait out the rest of the loop and then return to operation1005to restart the process.

The technical problem that arises when logging data12into an autonomous vehicle10is the gathering of sensor data12becomes computationally intensive. The sensor data12is important for system debugging and development. As such, the excessive data12logging can place a large burden on the on-board computing system200and negatively impact performance because file I/O is performance-intensive for computers. It may even prevent the autonomous vehicle10from operating in certain complex scenarios, where massive data12gathering is required to properly develop and debug the system in those situations.

Certain conventional solutions to this problem have been to record less data12(downsampling or simply leaving out data12). Other conventional solutions to the problem are to invest in additional computing power dedicated for logging data12.

To the contrary, the proposed logging innovation approaches the problem in such a way as to provide an improved technical solution by maintaining the amount of data12logged using fewer computational resources. Sensor data12is held in second memory206until an assessment of the criticality of the situation determines that it is an opportune time to log data12into the first memory208. The on-board computing system200may assess the calculated performance impact of logging data12to the first memory208and use that to determine if it is an opportune time to log data12to the first memory208or hold data12in the second memory206. In critical situations (e.g. lots of surrounding cars, cyclists, pedestrians) where faster cycle times are required, the system may choose to avoid logging data12to the first memory208until an opportune time presents itself. The described logging data12approach reduces the computational resources needed while maintaining the level of data12logged for system debugging and development.

Although certain embodiments have been described and illustrated in exemplary forms with a certain degree of particularity, it is noted that the description and illustrations have been made by way of example only. Numerous changes in the details of construction, combination, and arrangement of parts and operations may be made. Accordingly, such changes are intended to be included within the scope of the disclosure, the protected scope of which is defined by the claims.