Operating system aided code coverage

A method, system, and computer program product for operating system (OS) aided code coverage are provided. The method includes reading context information associated with a software process in response to a context switching event in an OS, the OS initiating the reading of the context information and controlling scheduling of the software process. The method further includes determining coverage information for code implementing the software process as a function of the context information in response to the OS reading the context information, and storing the coverage information as coverage data.

BACKGROUND

The present invention relates to software verification, and more specifically, to operating system aided code coverage.

Code coverage is a measure used in software testing, describing the degree to which source code of an application program has been exercised. Code coverage may quantify what portions of the application program were executed by a given test or series of tests. Collection of code coverage data is typically performed through directly instrumenting the application by adding additional code to monitor application parameters during execution. While running the instrumented application program (or at its termination) typically a coverage file is created. This approach also introduces overhead to the application execution due to the additional instrumentation code. Additional application-level support for a file system may also be required to store coverage information. For example, the application under test may be modified to write data gathered via instrumentation, injecting further changes into the application.

Instrumenting an application may not be effective or feasible for a number of scenarios, such as performing coverage at system level tests, in real-time applications, or where storage is inaccessible to the application for logging coverage information. System level tests may not support modification to add instrumentation, since the application is configured as it will be received by the customer (e.g., final release version) which makes modifying the application for instrumentation unacceptable. In a real-time application, the additional overhead associated with instrumentation may place too great of an additional burden on the system to maintain timing and memory allocation constraints for the application.

A less intrusive technique may employ sampling of hardware performance counters using special performance monitoring hardware to detect which part of the code was executed. Sampling hardware performance counters, however, is not always available, especially in embedded systems. Hardware performance counters add to the complexity and expense of processing system hardware, and may introduce new hardware failure modes should the performance monitoring hardware fail. Moreover, sampling of hardware performance counters introduces additional overhead and intervenes in normal execution of the program since the sampling is typically monitored by special kernel or user space application.

SUMMARY

According to one embodiment of the present invention, a method for operating system (OS) aided code coverage is provided. The method includes reading context information associated with a software process in response to a context switching event in an OS, the OS initiating the reading of the context information and controlling scheduling of the software process. The method further includes determining coverage information for code implementing the software process as a function of the context information in response to the OS reading the context information, and storing the coverage information as coverage data.

A further embodiment is a system for OS aided code coverage. The system includes system memory and a processing unit coupled to the system memory. The processing unit executes an OS that includes scheduling logic to control scheduling of a software process, context switch logic to read context information associated with the software process in response to a context switching event, and coverage logic to determine coverage information for code implementing the software process as a function of the context information read by the context switch logic, and storing the coverage information as coverage data to the system memory.

Another embodiment is embodiment is a computer program product for OS aided code coverage. The computer program product includes a storage medium readable by a processing unit and storing instructions for execution by the processing unit for implementing a method. The method includes reading context information associated with a software process in response to a context switching event in an OS, the OS initiating the reading of the context information and controlling scheduling of the software process, where the software process is an executable instance of an application program. The method further includes determining coverage information for code implementing the software process as a function of the context information in response to the OS reading the context information, and storing the coverage information as coverage data.

DETAILED DESCRIPTION

The invention as described herein provides operating system aided code coverage. Using an operating system (OS), applications may be run and managed as processes, tasks, threads, or the like, which are collectively referred to as “processes” herein. The OS schedules the processes in and out for execution on shared processing circuitry, e.g., a CPU. In order to pause execution of one process, switch to another process, and later resume execution of the first process, context information is stored by the OS prior to switching from the first process to the second process, enabling the first process to resume execution as if it had run without interruption. This is referred to as a “context switch”. During the context switch, the OS may save the complete state of the process, including states of various registers, such as a program counter (PC), stack pointer, condition codes, general purpose registers, floating-point registers, address registers, status registers, memory management information, and the like. In an exemplary embodiment, coverage logic is added to the OS to access the context information captured during context switching. This information can be used to log coverage information of the applications associated with the processes. In addition to application state information captured directly as part of the context, the OS can also check other system parameters and associate them with the current state of the application execution to log various inter-application state data.

The coverage logic of the OS can perform coverage data collection on single thread and multi-thread applications, e.g., where one process or multiple processes represent execution instances of a single application. Additionally, coverage can be achieved across multiple applications running simultaneously under the control of the OS. In an exemplary embodiment, the OS supports pseudo-context switching. Pseudo-context switching refers to scheduling one or more processes for context examination without performing an actual context switch. This enables context information to be examined at a higher rate than the rate of process switching. The OS can perform a pseudo-context switch just for coverage purposes. That is, scheduling the process and examining the PC or another relevant portion of the context without saving it and resuming execution of the process without switching to another process. Thus, in exemplary embodiments, existing OS logic is leveraged and expanded upon to monitor applications without modifying the applications and without adding monitoring hardware.

The coverage logic in the OS can aid in testing code through logging data to determine various types of coverage. For example, the coverage logic can log PC information over a period of time as coverage data, which can then be post processed to compare the log of PC information against a symbol table or map file for the application to determine specific sections of code that were being executed when the context was captured. This information may indicate execution of specific functions, lines of sources code, paths, branches, entry, and exit of the functions in the application, providing coverage information. Additional coverage information can be derived from the context information, as well as other resources available to the OS. For instance, the coverage logic can determine stack usage, heap usage, queue usage, and other memory usage information based checking allocation request parameters against registers containing pointer values (e.g., the stack pointer) to determine how much of the resources have been consumed. The OS may use other resource tracking techniques known in the art to determine resource consumption at context switching time. Further details regarding operating system aided code coverage are provided herein.

FIG. 1and the following discussion are intended to provide a general description of an exemplary data processing system that can be adapted to implement exemplary embodiments of the present invention. While exemplary embodiments of the present invention will be described in the general context of an operating system that runs application programs in conjunction with a personal computer, those skilled in the art will recognize that exemplary embodiments may also be implemented in combination with other program modules such as, for example, platform software modules, user-written software modules (such as spreadsheet templates, word processor macros, graphics scripts, etc.), routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that exemplary embodiments of the present invention may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like, as well as in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Referring now toFIG. 1, there is depicted an exemplary data processing system15that may be utilized to implement exemplary embodiments of the present invention. For discussion purposes, the data processing system15is described as having features common to a personal computer, such as a desktop or portable computer. As used herein, however, the terms “data processing system,” “computer,” and the like, are intended to mean essentially any type of computing device or machine that is capable of receiving, storing, and running a software product, including such devices as communication devices (for example, pagers, telephones, electronic books, electronic magazines and newspapers, etc.) and personal and home consumer devices (for example, handheld computers, Web-enabled televisions, home automation systems, multimedia viewing systems, gaming consoles, etc.).

Data processing system15, as provided inFIG. 1, is configured as a personal computer that generally includes a processing unit4, a system memory50, and a system bus5that couples system memory50to processing unit4. The system memory50may include read-only memory, such as flash memory6, as well as volatile read-write memory, such as random access memory (RAM)8. Flash memory6is an electrically erasable programmable read only memory (EEPROM) module that includes a basic input/output system (BIOS)12. The BIOS12contains the basic routines that facilitate transfer of information between elements within the data processing system15, such as during start-up.

Data processing system15may further include a hard disk drive20, a magnetic disk drive44(which can be used to read from or write to a removable disk31), and an optical disk drive46(which can be used to read a CD-ROM disk33or read or write to other optical media). Hard disk drive20, magnetic disk drive44, and optical disk drive46are communicatively coupled to system bus5by a hard disk drive interface22, a magnetic disk drive interface32, and an optical disk drive interface34, respectively. The drives and their associated computer-readable media provide nonvolatile storage for data processing system15. Although the description of computer-readable media above refers to a hard disk, a removable magnetic disk and a CD-ROM disk, it should be appreciated that other types of media that are readable by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, and the like, may also be used in exemplary computer operating environments.

A number of program modules may be stored in the drives20,44,46, and RAM8, including an operating system (OS)14, application program modules16(such as, for example, word processors, development applications, data management applications, and Web applications), and program data18. A user may enter commands and information into data processing system15through a keyboard46and a mouse48. Other input devices (not shown) may include, for example, a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices may be connected to processing unit4through a serial port interface39that is coupled to system bus5, but may be connected by other interfaces, such as a game port or a universal serial bus (USB). A monitor24or other type of display device is also connected to system bus5via an interface, such as a video adapter36. In addition to the monitor, the exemplary computer operating environment may also include other peripheral output devices (not shown), such as speakers or printers. Although only a single system bus5and processing unit4are depicted, it will be understood that other bus architectures are also included within the scope of the invention, such as multiple buses, multiple processors, and multi-core modules.

Data processing system15may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer49. Remote computer49may be, for example, a server, a router, a peer device, or another common network node, and may include many or all of the elements described in relation to data processing system15. The logical connections depicted inFIG. 1include a local area network (LAN)51and a wide area network (WAN)53.

When used in a LAN networking environment, data processing system15is connected to LAN51through a network interface42. When used in a WAN networking environment, data processing system15may include a modem44or other means for establishing communications over WAN53, such as the Internet. Modem44, which may be internal or external to data processing system15, may be connected to system bus5via serial port interface39. In a networked environment, program modules depicted relative to data processing system15, or portions thereof, may be stored in the remote memory storage device. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used. For instance, networking can be achieved via wired, wireless, and/or optical communication interfaces.

Turning now toFIG. 2, further details of the OS14ofFIG. 1are depicted for an exemplary multi-process environment. In the example depicted inFIG. 2, the OS14interfaces with process1202and process2204, which represent instances of one or more of the application programs16ofFIG. 1. For instance, processes202and204can represent a common application launched multiple times or entirely separate applications. The OS14uses scheduling logic206to determine when the processes202and204should be executed or otherwise interrupted. The scheduling logic206may respond to an event208such as an interrupt or a system call. Alternatively, the scheduling logic206may include timing functions to trigger an internal event and determine when to switch between or interrupt the processes202and204. The scheduling logic206can trigger context switch logic210to store and restore context information for the processes202and204. For example, upon determining that it is time to switch between executing process1202and process2204, the context switch logic210stores process1context212into context storage214and reads process2context216from the context storage214, assuming that process2context216was previously stored to the context storage214. The process2context216values are restored, which enables execution of process2204as if it had not been interrupted to run process1202. The values read from and written to process1context212and process2context216include state information, such values of registers218, which may include a PC, stack pointer, condition codes, general purpose registers, floating-point registers, address registers, status registers, memory management information, and the like.

In an exemplary embodiment, the OS14utilizes coverage logic220to read context information and determine coverage information for code implementing the processes202and204as a function of the context information. This assists in code coverage data collection, which may be helpful during development or post-development test activities. The coverage logic220can access stored context information in the context storage214or access the context information gathered by the context switch logic210. For example, collecting snapshots of the PC on each context switching event indicates which locations in the application associated with process1202or process2204was being executed prior to the context switching event, e.g., event208. The coverage information is stored as coverage data222, which may be held temporarily in RAM8or be written to one of the drives20,44, and/or46ofFIG. 1. The full context of the processes202and204need not be saved or examined completely by the coverage logic220for the coverage logic220to function.

Parameters224can also be examined for coverage information. The parameters224may include inter-process state data, for instance, data that is common across processes202and204. The coverage logic220associates the parameters224with the context information to determine the coverage information. The parameters224may also include one or more threshold limits to perform a coverage task. For example, a coverage task could be that process1202does not reach 70% of stack usage at any time during execution. Comparing the stack usage against the threshold limit at each invocation of the coverage logic220enables the coverage logic220to determine whether the threshold limit was exceeded, as well as capture a snapshot of other context data when the threshold is exceeded. The resulting data can be stored in the coverage data222. The parameters224may also indicate a desired coverage level and whether the coverage logic220should be enabled/disabled under various conditions. Enabling and disabling of the coverage logic220can be driven using supervisor commands, a configuration file, or automatically based on system loading (e.g., percentage utilization of the processing unit4ofFIG. 1).

In an exemplary embodiment, a post-processing application226can be used to further analyze the coverage data222. The post-processing application226may be one of the application programs16ofFIG. 1. For instance, if the coverage data222includes PC values, the post processing application226can look up address values in a symbol table228that maps locations to specific functions and/or portions of code to determine which specific sections of code were executed. The symbol table228may be a kind of program data18ofFIG. 1. While only one symbol table228is depicted inFIG. 2, it will be understood that multiple symbol tables or similar mapping files can be used to map the coverage data222to source code associated with processes202and/or204.

FIG. 3depicts a process switching diagram in accordance with exemplary embodiments. Arrow302illustrates the progression of time from the top towards the bottom ofFIG. 3. Initially in this example, process1202is executing304and process2204is idle306. An interrupt or system call308, such as event208, triggers scheduling logic206ofFIG. 2. The scheduling logic206may determine that it should switch between executing process1202and process2204in response to the interrupt or system call308. The scheduling logic206can trigger a context switching event and utilize the context switch logic210ofFIG. 2to pause process1202and save process1context212to the context storage214ofFIG. 2, as depicted in block310. This makes process1202transition to idle312and process2204also remains idle306as the OS14performs OS operations314. The OS operations314may include running the coverage logic220to examine the process1context212to determine coverage information. After the coverage information is saved as coverage data222and any additional OS operations314are performed, the context switch logic210restores process2context216for process2204as depicted in block316. Process2204switches to executing318and process1202remains idle312.

In response to another interrupt or system call320, such as event208, the scheduling logic206ofFIG. 2is triggered again. The scheduling logic206determines that it should switch between executing process2204and process1202in response to the interrupt or system call320. The scheduling logic206triggers a context switching event and utilizes the context switch logic210ofFIG. 2to pause process2204and save process2context216to the context storage214ofFIG. 2, as depicted in block324. This makes process2204transition to idle322and process1202also remains idle312as the OS14performs OS operations326. The OS operations326may include running the coverage logic220to examine the process2context216to determine coverage information. After the coverage information is saved as coverage data222and any additional OS operations326are performed, the context switch logic210restores process1context212for process1202as depicted in block328. Process1202switches to executing330and process2204remains idle322. The sequence can be repeated until one or both of the processes202and204terminate.

FIG. 4depicts an example of coverage logic220as part of pseudo-context switching in a single process environment.FIG. 4includes elements as previously described in reference toFIGS. 1-3. For example,FIG. 4includes the OS14with scheduling logic206, context switch logic210, coverage logic220and parameters224. The OS14can access registers218and context storage214. Coverage information as determined by the coverage logic220is stored as coverage data222, which may be accessible to the post-processing application226in conjunction with the symbol table228.FIG. 4illustrates an example of a single process402executing under control of the OS14. Since there is only the single process402, actual context switching need not be performed by the context switch logic210. In response to an event404, the scheduling logic206can pause execution of the process402and trigger the context switch logic210to perform pseudo-context switching. Pseudo-context switching makes context information available but avoids storing the context information to the context storage214. Context information may be available to the coverage logic220in the form of a data object in RAM8ofFIG. 1, or it can be directly accessible via the registers218and/or the parameters224. Thus, the coverage logic220is able to operate in a similar fashion as described in reference toFIGS. 2 and 3to determine coverage information for the process402.

Although pseudo-context switching has been described in reference to a single process402, it will be understood that pseudo-context switching can be applied to a multiple process environment, such as that depicted inFIGS. 2 and 3. For example, the event208that triggers a switch between processes202and204may occur at a relatively slow rate. The OS14can also be sensitive to other types of events, such as event404, that can occur at a higher rate to increase the frequency of coverage information collection without unnecessarily saving and switching context on each type of event.

Turning now toFIG. 5, a process500for operating system aided code coverage will now be described in accordance with exemplary embodiments, and in reference toFIGS. 1-4. At block502, context switch logic210reads context information associated with a software process (e.g., processes202,204or402) in response to a context switching event. The OS14initiates the reading of the context information and controls scheduling of the software process, for instance, using the scheduling logic206in conjunction with the context switch logic210. The context switch logic210can pause execution of the software process in response to a scheduling event (e.g., event208or404) received at the scheduling logic206. The context switching event may be triggered in response to pausing the execution of the software process. The context switch logic210may store the context information associated with the software process in response to the context switching event. The context switching event may be an actual context switch or a pseudo-context switch, where the pseudo-context switch performs the determination of coverage information and resumes execution of the software process without switching to a second software process.

At block504, the coverage logic220determines coverage information for code implementing the software process as a function of the context information in response to the OS14reading the context information. The context information associated with the software process can include one or more state values of one or more of the registers218, including one of more of a program counter, a stack pointer, condition codes, general purpose registers, floating-point registers, address registers, status registers, and memory management information. The parameters224accessible to the OS14, including inter-process state data, can be checked and associated with the context information to determine the coverage information. Additionally, the coverage logic220can perform coverage tasks, such as reading the parameters224to establish one or more threshold limits and comparing the context information against the one or more threshold limits.

At block506, the coverage logic220stores the coverage information as coverage data222. The context switch logic210may restore previously stored context information associated with a second software process (e.g., process2204) and resume execution of the second software process subsequent to storing the coverage information. Performance of the determining and storing of the coverage information may be configurable. The post-processing application226can perform post-processing of the coverage data222with respect to symbol table228to identify portions of the code that were executed, as well as output other coverage analysis results.

Technical effects include collecting code coverage data collection data at the OS level using data associated with context switching. This supports software development verification without modifying the application under test or requiring special purpose monitoring hardware. Piggy backing on existing context switching, which is already performed by the OS, enables code coverage without extra overhead. Applying coverage logic at the OS level also allows coverage analysis in a post-development environment, such as at a customer site. OS level code coverage data collection may be safer than at the application level, since more resources are available and accessible. This avoids potential errors that can occur at the application level, such as writing to a file without proper permission or attempting to write to a file that does not exist. The coverage logic of the OS may be turned on/off of by supervisor commands, for example, changing a process running level from at coverage level to regular level and vise versa. This can be done automatically based on system load.