APPARATUS AND METHOD FOR DEBUGGING AUTONOMOUS DRIVING APPLICATION WITH CONDITIONAL BREAKPOINT

A method for debugging an autonomous driving application includes setting a line breakpoint in a line of source code of the application; setting a trigger condition relative to simulation data of a simulator executing the application; running the application by a debugger for debugging the application; and pausing running of the application by the debugger based on the line breakpoint being reached and the trigger condition being hit.

BACKGROUND

1. Technical Field

The present disclosure relates to debugging an autonomous driving application, and in particular, relates to debugging an autonomous driving application by triggering a breakpoint in a debugger based on a condition being satisfied.

2. Description of Related Art

A given autonomous driving simulator uses various simulation models to test a particular application for use in an autonomous driving vehicle, such as an autonomous driving stack that controls an ego vehicle or a perception application that detects surrounding objects from a camera image. A related art autonomous driving simulation system includes a core simulator, a vehicle simulator, and one or more sensor simulators. Via the core simulator, the virtual world of the simulation can be defined, as well as the actors and their movements. The vehicle simulator simulates an autonomous driving vehicle (ego vehicle) and includes a vehicle controller to control a position of the vehicle based on surrounding objects. The sensor simulator simulates various sensors of the ego vehicle, and includes a camera sensor model for capturing images of the ego vehicle's surroundings in the virtual world. While the core simulator is run to test an application, log data is generated by the simulator. This log data can be subsequently replayed by a log player to debug the application in conjunction with a debugger.

Typically, a user can use the debugger to set breakpoints at particular lines of the source code, i.e., line breakpoints. Upon debugging the application, e.g., executing the application in a debug mode, the debugger suspends the simulation once the execution reaches the line on which the breakpoint is set. This allows the user to perform any of various debugging steps, such as analyzing variable values, program parameters, and the call stack, as well as stepping through and/or running back the application.

SUMMARY

In the context of a core simulator for an autonomous driving simulation, however, large amounts of serial data are generated. This could be in the range of tens of thousands of frames for one execution of the simulation. When replaying, the debugger will stop at the line breakpoint at each enumeration, i.e., at every frame of the serial data. This results in a tedious, inconvenient and impractical debugging, particularly where the user is interested in analyzing program parameters or variables at a specific frame (e.g., a frame right before a collision occurs or a particular item fails). To reach that specific frame, the user will have to step through the breakpoint in each frame prior. Further, if the user were to manually break the execution at the frame of interest, it is nearly impossible to precisely time the break at the exact line of code at which the user desires to analyze the log data.

The present disclosure provides a method and device for debugging an autonomous driving application by triggering a breakpoint in a debugger based on a condition being satisfied.

An aspect of the disclose provides a method for debugging an autonomous driving application. The method may include setting a line breakpoint in a line of source code of the application; setting a trigger condition relative to simulation data of a simulator executing the application; running the application by a debugger for debugging the application; and pausing running of the application by a debugger based on the line breakpoint being reached and the trigger condition being satisfied.

The simulation data may be log data generated by the simulator, and the trigger condition may be a log breakpoint in a time line of the log data.

The setting the trigger condition may include setting the trigger condition via a log player for replaying the log data.

The pausing the running of the application may include, based on the log breakpoint being reached, pausing a replaying of the log data by the log player and sending a notification to the debugger; and based on the notification being received by the debugger, pausing the running of the application by the debugger at the line breakpoint.

In response to the notification being received by the debugger, the debugger may begin checking for the line breakpoint.

The method may farther include replaying of the log data by the log player. The replaying of the log data by the log player comprises sending log data to the debugger and checking the replayed log data for the log breakpoint.

The log breakpoint may be set at a timeline or a play bar of the log player.

The trigger condition may be a simulation condition in the simulator, and the setting the trigger condition may include setting the trigger condition in the simulator.

The pausing the running of the application may include, based on the simulation condition being satisfied while executing the application by the simulator, pausing the simulator and sending a notification to the debugger; and based on the notification and the line breakpoint being reached, pausing the running of the application by the debugger.

In response to the notification being received by the debugger, the debugger may begin checking for the line breakpoint.

The method may further include executing the application by the simulator. Executing the application by the simulator may include checking for the simulation conditions.

The simulation condition may be one of a time to a collision between an ego vehicle and an object being less than a particular threshold, a failure of a particular model or object, a distance between the ego vehicle and objects or a lane line being less than a predetermined value, an ego vehicle speed being greater than a particular value, and an ego vehicle acceleration being greater than a particular value.

Another aspect of the disclosure provides a device for debugging an autonomous driving application. The device may include a memory store at least one instruction; and a processor configured to execute the at least one instruction to: set a line breakpoint in a line of source code of the application; set a trigger condition relative to simulation data of a simulator executing the application; run the application by a debugger for debugging the application; and pause running of the application by a debugger based on the line breakpoint being reached and the trigger condition being satisfied.

The simulation data may be log data generated by the simulator, and the trigger condition may be a log breakpoint in a time line of the log data; and the setting the trigger condition may include setting the trigger condition via a log player for replaying the log data.

The processor may be configured to execute the at least one instruction to: based on the log breakpoint being reached, pause a replaying of the log data by the log player and sending a notification to the debugger; and based on the notification being received by the debugger, pause the running of the application by the debugger at the line breakpoint.

The processor may be configured to execute the at least one instruction to, in response to the notification being received by the debugger, control the debugger to begin checking for the line breakpoint.

The processor may be configured to execute the at least one instruction to replay the log data by the log player. The replaying of the log data by the log player may include sending log data to the debugger and checking the replayed log data for the log breakpoint.

The log breakpoint may be set at a timeline or a play bar of the log player.

The trigger condition may be a simulation condition in the simulator, and the setting the trigger condition may include setting the trigger condition in the simulator.

The processor may be configured to execute the at least one instruction to: based on the simulation condition being satisfied while executing the application by the simulator, pause the simulator and sending a notification to the debugger; and based on the notification and the line breakpoint being reached, pause the running of the application by the debugger.

The processor may be configured to execute the at least one instruction to, in response to the notification being received by the debugger, control the debugger to being checking for the line breakpoint.

The processor may be configured to execute the at least one instruction to execute the application by the simulator. Executing the application by the simulator may include checking for the simulation conditions.

The simulation condition may be one of a time to a collision between an ego vehicle and an object being less than a particular threshold, a failure of a particular model or object, a distance between the ego vehicle and objects or a lane line being less than a predetermined value, an ego vehicle speed being greater than a particular value, and an ego vehicle acceleration being greater than a particular value.

Another aspect of the disclosure provides a non-transitory computer-readable medium storing instructions. The instructions may include one or more instructions that, when executed by one or more processors of a device, cause the one or more processors to: set a line breakpoint in a line of source code of an autonomous driving application; set a trigger condition relative to simulation data of a simulator executing the application; run the application by a debugger for debugging the application; and pause running of the application by a debugger based on the line breakpoint being reached and the trigger condition being satisfied.

DETAILED DESCRIPTION

A debugging method according to the present disclosure provides a conditional breakpoint for debugging an autonomous driving application. To provide precise timing for breaking the execution at the code of interest, a line breakpoint may be set in the source code. Meanwhile, to prevent repetitive breaks at each frame of the log data, the debugger according to an embodiment may not be triggered to stop execution unless both the line breakpoint and an additional trigger condition are hit. The additional trigger condition may be set in relation to simulation data, either in log data via a log player or in a simulation execution itself.

FIG.1is a flowchart of an overview of a method of debugging an application using conditional breakpoints according to an embodiment.

Referring toFIG.1, at operation110, a line breakpoint may be set in a source code of the application. For example, a line breakpoint may be set at entry to a function so the application may be analyzed at that point. In some embodiments, multiple line breakpoints may be set throughout the source code at positions where analysis may be performed.

At operation120, a trigger condition is set relative to simulation data of a simulator running the application. In an embodiment, the trigger condition may be set in log data (e.g., via a log player) produced by the simulation. For example, one or more log breakpoints may be set at a timeline or play bar of the log player, thereby allowing the particular frame at which the user desires to analyze the code or the application to be easily selected. In another embodiment, the trigger condition may be set via the simulator, based on a condition detected (or to be detected) in the simulator. For example, simulation conditions may be set as a time of collision between an ego vehicle and an object being less than a particular threshold, a failure of a particular model or object, a distance between an ego vehicle and objects or a lane line being less than a predetermined value, an ego vehicle speed being greater than a particular value, and/or an ego vehicle acceleration being greater than a particular value. As a result, the user can easily select a desired point of time in the simulation to debug the application.

At operation130, the application may be run by a debugger (e.g., the application may be executed in a debug mode). The debugger may be in communication with the simulator running the application and/or a log player playing log data produced by the simulator while running the application. In some embodiments, the debugger may run concurrently with the simulator and/or the log player.

At operation140, the running of the application by the debugger may be paused (break) based on the line breakpoint being reached and the trigger condition being satisfied. For example, as the debugger is running, multiple line breakpoints may be reached. However, the debugger will pass line breakpoints without pausing execution if the trigger condition has not been satisfied. Once the trigger condition is satisfied, the bugger is notified by the simulator or the log player (e.g., by receiving a pause call), and the debugger will pause the next time it reaches a line breakpoint in the source code.

In this way, some implementations herein permit efficient debugging by only breaking at relevant points based on a condition being satisfied. That is, as a result of the conditional breakpoint, a precise line of code can be set for debugging at a specific frame of a simulation or log data without repeatedly stopping execution at each frame of the serial data.

FIG.2is a diagram of an example environment200in which systems and/or methods, described herein, may be implemented. As shown inFIG.2, the environment200may include a simulator210, a debugger220, and a log player230in communication with each other. According to an embodiment, the components of the environment200may be implemented as software programs running on a single device such as a personal computer, workstation computer, a server device, or other types of computation and/or communication devices. In other embodiments, the components may operate on multiple devices that are interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

The number and arrangement of devices and networks shown inFIG.2are provided as an example. In practice, there may be additional components, fewer components, different components, or differently arranged components than those shown inFIG.2. For example, in various embodiments, one of the simulator and the log player may be omitted.

FIG.3is a diagram of example components of a device300for performing the method100of debugging according to an embodiment. The device300may correspond a single device that performs the method100or one of multiple devices that perform the method100. The device300may include or implement at least one of the simulator210, the debugger220, and the log player230illustrated inFIG.2. As shown inFIG.3, the device300may include a bus310, a processor320(or at least one processor), a memory330(or at least one memory), a storage component340, an input component350, an output component360, and a communication interface370.

The bus310may include a component that permits communication among the components of the device300. That is, the bus310is a communication system or subsystem that permits the transfer of data and information between the components of the device300.

The processor320may be implemented in hardware, firmware, or a combination of hardware and software. The processor320may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. In some embodiments, the processor320includes one or more processors capable of being programmed to perform a function.

The memory330stores information and/or instructions for use by the processor320. The memory330may include at least one of a random access memory (RAM), a read only memory (ROM), and/or another type of dynamic or static storage device (e.g., a flash memory, a magnetic memory, and/or an optical memory) for storing information and/or instructions.

The input component350may include a component that permits the device300to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, and/or a microphone). Additionally, or alternatively, the input component350may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and/or an actuator). The output component360may include a component that provides output information from the device300(e.g., a display, a speaker, one or more light-emitting diodes (LEDs), an interface for connecting to an external display or speaker, etc.).

The communication interface370may include a transceiver-like component (e.g., a transceiver and/or a separate receiver and transmitter) that enables the device300to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface370may permit the device300to receive information from another device and/or provide information to another device. For example, the communication interface370may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, or the like.

Software instructions may be read into the memory330and/or the storage component340from another computer-readable medium or from another device via the communication interface370. When executed, software instructions stored in the memory330and/or the storage component340may cause the processor320to perform one or more processes described herein. Additionally, or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, implementations described herein are not limited to any specific combination of hardware circuitry and software.

The number and arrangement of components shown inFIG.3are provided as an example. In practice, the device300may include additional components, fewer components, different components, or differently arranged components than those shown inFIG.3. Additionally, or alternatively, a set of components (e.g., one or more components) of the device300may perform one or more functions described as being performed by another set of components of device300.

FIG.4is a diagram of an example process400for debugging an application according to an embodiment. According to some embodiments, the application may be an autonomous driving application. In some implementations, one or more operations ofFIG.4may be performed by the device300. In some implementations, one or more operations ofFIG.4may be performed by another device or a group of devices separate from or including the device300.

As shown inFIG.4, the process400may include a log player230communicating with a debugger220. Log data generated by a simulator210that simulates a vehicle executing the application in a particular scenario may be input into the log player230and the source code from the application may be input into the debugger220. For example, a simulator210on the device300may execute an autonomous driving application and may generate log data, which is subsequently replayed by the log player230.

Once the log data has been input into the log player230and the source code of the autonomous driving application is input into the debugger220, at time T1, both the log player230and debugger220are set to free run.

At time T2(or prior), one or more trigger conditions may be set in the log player230as log breakpoints. For example, log breakpoints may be set at a timeline or play bar of the log player230. Thus, the particular frame at which the user desires to analyze the code or the application can be easily selected.

Also at time T2(or a different time), one or more line breakpoints may be set in the source code being run by the debugger220. The line breakpoints may be set at points in the code that may benefit from analysis. For example, a line breakpoints may be set at a location of the code corresponding to the onset of a function of the application.

At time T3, the log player230may begin running the log from the simulation and sending log data to the debugger220. For example, log data generated by a virtual platform, such as an autonomous driving simulator executing an autonomous driving application, may be run on the log player230and be sent to the debugger220, from the log player230, for use in debugging the source code of the application.

The debugger220may begin executing the application based on the received log data to debug the application. The debugger220passes any line breakpoints in the source code without suspending the execution, if the log player230has not indicated that a trigger condition has been satisfied.

At time T4, the log player230checks the log data for the trigger conditions (log breakpoints) that have been set at time T2as the log is running. In some embodiments, the log player may begin checking for trigger conditions when the log player230begins running the log.

At time T5, the log player may determine that a trigger condition has been satisfied. For example, the log player may reach a point on a timeline that was set as a trigger condition. Based on the trigger condition being satisfied, the log player pauses running the log and notifies the debugger220that the trigger condition has been satisfied.

Upon (or based on) receiving the notification indicating that the trigger condition has been satisfied, the debugger220may begin checking for line breakpoints in the source code. As such, the debugger220may continue to execute the application until a line breakpoint in the source code is reached. When the breakpoint in the source code is reached, the debugger220pauses execution to allow a user to debug the application. Prior to receiving the notification indicating the trigger condition has been satisfied, the debugger220may not check for line breakpoints or may otherwise disregard the line breakpoints.

FIG.5is a diagram of an example process500for debugging an application according to another embodiment. According to some embodiments, the application may be an autonomous driving application. In some implementations, one or more operations ofFIG.5may be performed by the device300. In some implementations, one or more operations ofFIG.5may be performed by another device or a group of devices separate from or including the device300.

As shown inFIG.5, the process500may include a simulator210communicating with a debugger220. According to an embodiment, the simulator210may be a virtual platform provided to simulate operation of an ego vehicle in a particular scenario while executing an autonomous driving application of the ego vehicle, and the debugger220may execute the application for debugging (e.g., in a debug mode).

Once the source code of the autonomous driving application is input into the simulator210and debugger220, at time T1, both the simulator210and debugger220are set to free run.

At time T2(or a different or prior time), one or more trigger conditions may be set in the simulator210as simulation conditions. For example, simulation conditions may be set as a time of collision between an ego vehicle and an object being less than a particular threshold, a failure of a particular model or object, a distance between an ego vehicle and objects or a lane line being less than a predetermined value, an ego vehicle speed being greater than a particular value, and/or an ego vehicle acceleration being greater than a particular value.

Also at time T2(or a different time), one or more line breakpoints may be set in the source code being run by the debugger220. The line breakpoints may be set at points in the code that may benefit from analysis. For example, a line breakpoints may be set at a point of the source code corresponding to the onset of a function of the application.

At time T3, the simulator210may begin running the simulation based on the application. For example, the simulator210may simulate operation of an ego vehicle based on an autonomous driving application.

At time T4, the simulator210may check for the trigger conditions (simulation conditions) that have been set at time T2(or prior) as the simulation is running. In some embodiments, the simulator210may begin checking for the trigger conditions when the simulator210begins running.

At time T5, the simulator210may determine that a trigger condition has been satisfied. For example, the simulator210may determine that an ego vehicle speed is greater than a particular value. Based on the trigger condition being satisfied, the simulator210may pause the simulation and notify the debugger220that the trigger condition has been satisfied.

Upon (or based on) receiving the notification indicating the trigger condition has been satisfied, the debugger220may check for line breakpoints in the source code. As such, the debugger220may continue to execute the application until a line breakpoint in the source code is reached. When the breakpoint in the source code is reached, the debugger220may pause execution to allow a user to debug the application. Prior to receiving the notification indicating the trigger condition has been satisfied, the debugger220may not check for line breakpoints or may otherwise disregard the line breakpoints.

As described above, methods and systems according to embodiments provide a multi-trigger breakpoint for debugging target automotive applications in an electronic control unit (ECU) of an automobile. As a result of the multi-trigger breakpoint, a precise line of code can be set for debugging at a specific frame of simulation or log data, without repeatedly stopping execution at each frame. Precision and efficiency of debugging is therefore improved over related art debugging systems and methods.