Generating test coverage bin based on simulation result

A solution for generating functional coverage bins for testing a device is disclosed. A method includes: receiving information of a failing test generated from a random simulation performed on the device; tracing a first sequence of signal events that happened in the failing test; correlating the signal events to coverage bins to generate a sequence of coverage bins; creating cross coverage event sequence bins based on the sequence of coverage bins; and outputting the created coverage event sequence bins for testing the device.

BACKGROUND OF THE DISCLOSURE

1. Technical Field

The disclosure relates generally to integrated circuit design, and more particularly to testing an integrated circuit design.

2. Background Art

Functional coverage indicates the thoroughness of a testing of an integrated circuit device. As integrated circuit design becomes larger and more complex, the testing process consumes time and resources, which may substantially hamper the introduction of new products into the market. As such, efforts need to be made to improve the efficiency of testing through, for example, optimizing functional coverage specification/design. Functional coverage bins (hereinafter “coverage bins”), used herein according to the generally accepted meaning(s) in the testing art, are generally used as a metric for determining whether all of the required stimuli have been applied to the device under test (DUT). A coverage bin is a part of a coverage definition of a testing program code (adapted to a simulation language, e.g., a Verilog simulator), which declares a specific behavior (or status) of a state variable. Each coverage bin may include a unique bin name.

Conventionally, functional coverage and/or the coverage bins in the testing program must be manually specified in the testing design stage. However, some failure tests may not be related to a particular stimulus event, but a sequence of them. The stimuli sequence is very difficult to determine when writing the coverage bins because dependencies among various stimulus events are not known beforehand.

Conventional approaches provide solutions to generate coverage bins based on product design specifications. In addition, there are solutions which process coverage data and determine whether existing coverage bins are satisfied/covered in a test. However, no solution exists which generates coverage bins from constrained random simulation test result(s).

SUMMARY OF THE DISCLOSURE

In a first embodiment, there is a method for generating functional coverage bins for testing a device. The method comprises: receiving information of a failing test generated from a random simulation performed on the device; tracing a first sequence of signal events that happened in the failing test; correlating the signal events to coverage bins to generate a sequence of coverage bins; creating cross coverage event sequence bins based on the sequence of coverage bins; and outputting the created coverage event sequence bins for testing the device

In a second embodiment, there is a system for generating functional coverage bins for testing a device. The system comprises: means for receiving information of a failing test generated from a random simulation performed on the device; means for tracing a first sequence of signal events that happened in the failing test; means for correlating the signal events to coverage bins to generate a sequence of coverage bins; and means for creating cross coverage event sequence bins based on the sequence of coverage bins.

In a third embodiment, there is a computer program product for generating functional coverage bins for testing a device. The computer program product comprises: computer usable program code stored in a computer readable medium, which when executed by a computer system enables the computer system to: receive information of a failing test generated from a random simulation performed on the device; trace a first sequence of signal events that happened in the failing test; correlate the signal events to coverage bins to generate a sequence of coverage bins; create cross coverage event sequence bins based on the sequence of coverage bins; and output the created coverage event sequence bins for testing the device.

Other aspects and features of the present disclosure, as defined solely by the claims, will become apparent to those ordinarily skilled in the art upon review of the following non-limited detailed description of the disclosure in conjunction with the accompanying figures.

DETAILED DESCRIPTION OF THE DISCLOSURE

The current disclosure provides a solution(s) for automatically generating coverage bins based on constrained random simulation test results. The following detailed description of embodiments refers to the accompanying drawings, which illustrate specific embodiments of the disclosure. Other embodiments having different structures and operations do not depart from the scope of the present disclosure.

FIG. 1shows an illustrative system10. System10includes a computer environment100for, e.g., generating coverage bins to test a device12. Device12is also associated to computer100through messages14. Messages14may include a result(s) of random simulation test performed on device12. Computer environment100may process the simulation test results and generate coverage bins for further testing device12. To this extent, computer environment100includes a computer infrastructure102that can perform the various processes described herein for generating coverage bins. In particular, computer infrastructure102is shown including a computing device104that comprises a coverage bin generation system132, which when executed by computer device104enables computing device104to perform the process(es) described herein.

Computing device104is shown including a memory112, a processing unit (PU)114, an input/output (I/O) interface116, and a bus118. Further, computing device104is shown in communication with an external I/O device/resource120and a storage system122. In general, PU114executes computer program code, such as coverage bin generation system132, that is stored in memory112and/or storage system122. While executing computer program code, PU114can read and/or write data to/from memory112, storage system122, and/or I/O interface116. Bus118provides a communications link between each of the components in computing device104. I/O interface116can comprise any device that enables a user to interact with computing device104or any device that enables computing device104to communicate with one or more other computing devices. External I/O device/resource120can be coupled to the system either directly or through I/O interface116.

In any event, computing device104can comprise any general purpose computing article of manufacture capable of executing computer program code installed thereon. However, it is understood that computing device104and coverage bin generation system132are only representative of various possible equivalent computing devices that may perform the various processes of the disclosure. To this extent, in other embodiments, computing device104can comprise any specific purpose computing article of manufacture comprising hardware and/or computer program code for performing specific functions, any computing article of manufacture that comprises a combination of specific purpose and general purpose hardware/software, or the like. In each case, the program code and hardware can be created using standard programming and engineering techniques, respectively.

Similarly, computer infrastructure102is only illustrative of various types of computer infrastructures for implementing the disclosure. For example, in an embodiment, computer infrastructure102comprises two or more computing devices that communicate over any type of wired and/or wireless communications link, such as a network, a shared memory, or the like, to perform the various processes of the disclosure. When the communications link comprises a network, the network can comprise any combination of one or more types of networks (e.g., the Internet, a wide area network, a local area network, a virtual private network, etc.). Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters. Regardless, communications between the computing devices may utilize any combination of various types of transmission techniques.

Coverage bin generation system132includes a data collecting unit140; an operation controlling unit142; an event tracing unit143; a correlating unit144; a cross coverage event sequence bins (CCESB) creating unit146including a sequence limiter148and an expert rule applier150; a screening unit152; and other system components158. Other system components158may include any now known or later developed parts of a computer system required for coverage bin generation system132but not individually delineated herein and understood by those skilled in the art. As should be appreciated, components of computer infrastructure102and/or coverage bin generation system132may be located at different physical locations or at the same physical location.

Memory112and/or storage system122may store a database of functional coverage bins for testing device12and other coverage bins, for example, coverage bins proved not to be related to a device12failure. In addition, coverage bin generation system132may be stored and/or deployed in memory112and/or storage system122.

Inputs to computer infrastructure102, e.g., through external I/O device/resource120and/or I/O interface116, may include message14and/or other inputs200. Other inputs200may include any available information required for the operation of coverage bin generation system132. For example, other inputs200may include the existing coverage bins for testing device12and/or the design specification of device12. As described herein, message14may include results of a random simulation test(s) performed on device12. The results may include a passing test(s) and/or a failing test(s). A failing test refers to a simulation test under which device12fails to meet a design specification/expectation. A passing test refers to a simulation test under which device12meets the design specification(s)/expectation(s). The input data may be collected by data collecting unit140and may be analyzed by coverage bin generation system132. Outputs of computer infrastructure102may include newly generated coverage bins, which may be used in, for example, further testing of device12. For example, the testing program may be revised accordingly to make sure that the newly generated coverage bins are hit/covered in a further test. The operation of coverage bin generation system132will be described herein in detail.

2. Coverage Bin Generation System

FIG. 2shows one embodiment of the operation of coverage bin generation system132. Referring now toFIGS. 1-2, collectively, in process S1, data collecting140collects data required for the operation. For example, results of a constrained random simulation test conducted on device12(usually referred to as device under test “DUT”) may be collected. The results may include information regarding whether device12passes or fails the test and other information, e.g., state events (behaviors) of state variables that occur following the application of the constrained random stimuli onto device12. It should be appreciated that data collecting unit140does not necessarily receive random simulation testing data from an outside source (as exemplarily shown inFIG. 1). Coverage bin generation system132may include a simulation component that may conduct a constrained random simulation on device12, and data collecting unit140may just obtain the data from the simulation component.

In process S2, operation controlling unit142determines whether a random simulation test is a passing test or a failing test. For a failing test (i.e., “Yes” at S2), operation controlling unit144directs the operation to process S3to process the test result(s). For a passing test (i.e., “No” at S2), operation controlling unit142directs the operation to process S6to process the test result(s).

In process S3, event tracing unit143traces a (first) sequence of signal events that happened in the falling test. A signal event refers to a detected behavior of a state variable of device12following the application of the stimulus in the failing test. Any solution may be used in the tracing. It should be appreciated that the information regarding the sequence of signal events may already be included in the collected information in process S1. To this extent, the tracing may include rearranging the already provided event sequence information or may include just taking/accepting the provided event sequence information. That is, event tracing unit143may not necessarily actually “trace” an event sequence if the event sequence is already available.

In process S4, correlating unit144correlates the signal events to coverage bins. Any solution may be used to implement the correlation. For example, correlating unit144may correlate a signal event with coverage bins that cross the signal event and the time thereof. For example, the coverage bin may include an event which represents that the signal event happens at a time with a certain value. Correlating unit144may also correlate a signal event with a coverage bin(s) that is related to the signal in the respective sequence. The time referenced in the correlating may be absolute time, cycle based time or a time relative to a time window. In addition, because the signal events are traced as a sequence in process S3, the correlated coverage bins are also related to one another as a sequence. That is, each coverage bin is recorded with respect to other coverage bins in a sequence. It should be appreciated that the correlating also includes the situation that a coverage bin is generated to correspond to a signal event.

In process S5, CCESB creating unit146creates cross coverage event sequence bins (CCESBs). A CCESB is a sub-sequence of coverage bins obtained through changing the sequence of coverage bins obtained in process S4(a CCESB may be the sequence of the coverage bins itself). To this extent, process S5may include multiple sub-processes. In process S5-1, CCESB creating unit146may obtain a pool of coverage bin sub-sequences that correspond to all possible permutations of the sequence of coverage bins. For example, assume the sequence of coverage bins includes coverage bins A-B-C-D in the listed order. The sub-sequences may include A-B, A-C, A-D, A-B-C, A-B-D, A-C-D, B-C-D, B-C, B-D, . . . , and A-B-C-D, i.e., all possible permutation combinations of coverage bins A, B, C, D in the sequence order.

According to an embodiment, sequence limiter148may set a maximum length for a sub-sequence. For example, following the above example, sequence limiter148may set a length limit of 2 coverage bins. Applying this length limit, sequence limiter148may filter the pool of sub-sequences so as to include only sub-sequences A-B, A-C, A-D, B-C, B-D and C-D. In addition, sequence limiter148may also set a limit on the maximum number of sub-sequences (in the pool) to be further processed.

In sub-process S5-2, expert rule applier150applies a provided expert rule to further eliminate irrelevant coverage bin sub-sequences. The expert rule may identify an irrelevant coverage bin sub-sequence using any standard. For example, the expert rule may identify an irrelevant sub-sequence as including coverage bins not related by value. For example, the expert rule may find that coverage bin sub-sequence C-D does not affect connection logic and thus is irrelevant. The expert rule may also identify an irrelevant coverage bin sub-sequence as including coverage bins not related by a specific (specified) order. For example, the expert rule may determine that coverage bins A after B, i.e., A-B, is possible, but B after A is not, i.e., irrelevant. The expert rule may be provided by an outside source (e.g., device12designer) or may be created by expert rule applier150based on device12design and testing requirements/specifications. Applying the expert rule, expert rule applying unit150may eliminate irrelevant coverage bin sub-sequences and the residual sub-sequences in the pool will be the CCESBs that deserve further processing.

In process S6, optionally, a passing test may be processed similarly as in processes S3-S5to obtain CCESBs in the passing test. That is, a sequence of signal events (second sequence) will be traced, correlated to coverage bins, and CCESBs will be obtained from the sequence of the related coverage bins.

In process S7, screening unit152further filters the CCESBs obtained in process S5(failing test) by eliminating CCESBs that are not unique to a failing test. The underlying reason is that if a CCESB exists in a passing test, the CCESB will not cause a failure in the failing test. Any solution/method may be used in the screening. For example, the CCESBs generated from the failing test operation (S3-S5) may be compared with the CCESBs generated from the passing test operation (S6), and if a CCESB exists in both the failing test and the passing test, the CCESB will be eliminated as being not unique. For another example, a passing test may be run on device12with the CCESBs generated from the failing test processing (S3-S5), and if a CCESB is hit/covered in the test, the CCESB is eliminated as being not unique.

In process S8, the residual CCESBs after the screening/filtering of process S7will be output to a user or an automatic testing component to be covered in the testing program code. As such the CCESBs will be hit/covered in the further testing of device12to, e.g., make sure that the failures causes by the CCESBs are avoided/cured.

The following exemplary operation further illustrates the processes. Assume that a device12include the following example DUT interfaces:

clock

and the coverage bins may be prepared as follows:

write all address locations with all data valuescross write (chip_select,addr,data)

read all address locationscross read (chip_select,addr)

Batch simulation may be run on device12to get collection of passing & failing tests (i.e., seeds) as follows:

test wrote addr 4 of bank 0 with A, then read addr 4 of bank 0 and the read data matched expect (PASS)

test wrote addr 4 of bank 0 with A, then read addr 3 of bank 1 and the read data matched expect (PASS)

test wrote addr 4 of bank 0 with A, then read addr 3 of bank 0 and the read data did not match expect (FAIL).

For each failing test, data trace may be generated and correlated to coverage bins to get correlation information that contains when each bin was hit relative to time (by cycle). Then, every coverage bin may be recorded with respect to the other coverage bin to generate a coverage bin sequence:

An expert rule(s) may be provided as follows:

accesses to bank 0 have no affect on access to bank 1 (and visa-vera).

And sequence limitations may be N=2 (max number of events in sequence) and M=4 (number of sequences to analyze). Applying the sequence limitation to change the coverage bin sequence, the sub-sequences may be created as:

Applying the expert rules, the following CCESBs are deemed irrelevant and are eliminated:

And CCESBs 1, 4, and 5 above are used for further testing device12.

While shown and described herein as a method and system for generating functional coverage bins for testing a device, it is understood that the disclosure further provides various alternative embodiments. For example, in an embodiment, the disclosure provides a program product stored on a computer-readable medium, which when executed, enables a computer infrastructure to generate functional coverage bins for testing a device. To this extent, the computer-readable medium includes program code, such as coverage bin generation system132(FIG. 1), which implements the process described herein. It is understood that the term “computer-readable medium” comprises one or more of any type of physical embodiment of the program code. In particular, the computer-readable medium can comprise program code embodied on one or more portable storage articles of manufacture (e.g., a compact disc, a magnetic disk, a tape, etc.), on one or more data storage portions of a computing device, such as memory112(FIG. 1) and/or storage system122(FIG. 1), and/or as a data signal traveling over a network (e.g., during a wired/wireless electronic distribution of the program product).

It should be appreciated that the teachings of the present disclosure could be offered as a business method on a subscription or fee basis. For example, a system10(FIG. 1), a computing device104comprising coverage bin generation system132(FIG. 1) could be created, maintained and/or deployed by a service provider that offers the functions described herein for customers. That is, a service provider could offer to generate functional coverage bins for testing a device as described above.