IMPROVING COVERAGE IN FUNCTIONAL VERIFICATION BY COORDINATED RANDOMIZATION OF VARIABLES ACROSS MULTIPLE CLASSES

A description of stimuli used for functional verification of a circuit design is received. The description includes classes of variables and the variable include random variables. A coverage model for the functional verification of the circuit design is also received. The coverage model includes coverage targets that are functions of the variables. A processing device generates stimuli for multiple iterations of the functional verification, as follows. Context values, which include values of the random variables for the stimuli, are maintained. The values of the random variables in an individual class are randomized, and the randomization of the random variables in the individual class is biased to hit the coverage targets given the context values for the random variables outside the individual class. Whether the coverage targets are hit by the generated stimuli is determined.

TECHNICAL FIELD

The present disclosure relates to functional verification of circuit designs and, more particularly, to improving the generation of enough stimuli to adequately verify the design.

BACKGROUND

Functional verification is a process for determining whether a circuit design functions as intended. Coverage refers to the extent to which different stimuli applied to the circuit design exercise (or cover) the intended or specified functionality. Coverage closure is the process of developing a set of stimuli that covers enough test cases to adequately test the circuit design.

However, one challenge of coverage closure is the ability to generate stimuli that exercise rarely occurring functionality of the circuit. In a constraint-random verification setting, some stimuli are modeled as random variables. The values of these stimuli are randomly selected, leading to a certain distribution of which test cases are exercised. Commonly occurring test cases are hit (exercised) frequently by randomly generated stimuli, and moderately common test cases are hit with moderate frequency. However, some test cases may be hit only rarely. The infrequency of these hits consumes a disproportionate number of processing cycles to reach coverage closure.

SUMMARY

In some aspects, a method includes the following. A description of stimuli used for functional verification of a circuit design is received. The description includes classes of variables and the variable include random variables. A coverage model for the functional verification of the circuit design is also received. The coverage model includes coverage targets that are functions of the variables. A processing device generates stimuli for multiple iterations of the functional verification, as follows. Context values, which include values of the random variables for the stimuli, are maintained. The values of the random variables in an individual class are randomized, and the randomization of the random variables in the individual class is biased to hit the coverage targets given the context values for the random variables outside the individual class. Whether the coverage targets are hit by the generated stimuli is determined.

In another aspect, a system includes a compiler and a verification testbench. The compiler receives a coverage model for functional verification of a circuit design. The coverage model includes coverage targets that are functions of variables for stimuli used for the functional verification. The variables include random variables. From the coverage model, the compiler determines and stores context connectivity information that identifies which coverage targets depend on which random variables. The verification testbench performs multiple stages of constrained random verification of the circuit design. Each stage is for a selected set of coverage targets and a selected class of variables. For each stage, the context connectivity information is accessed to identify which random variables the selected set of coverage targets depends on. Context values from prior stages for values of random variables outside the selected class are accessed. The values of random variables in the selected class are randomized for multiple iterations, but the randomization is biased to hit the coverage targets given the context values for the random variables outside the selected class. The context values are updated.

Other aspects include components, devices, systems, improvements, methods, processes, applications, computer readable mediums, and other technologies related to any of the above.

DETAILED DESCRIPTION

Aspects of the present disclosure relate to improving coverage in functional verification by coordinated randomization of variables across multiple classes. In one approach to verification, stimuli are applied to a circuit design. The operation of the circuit is simulated or otherwise analyzed, and the resulting behavior is compared to the desired behavior to determine whether the circuit will operate properly. Different stimuli exercise different test cases that may be referred to as coverage targets. If a stimulus exercises a test case, it is said to hit or cover that coverage target. Coverage closure is the process of developing sufficient stimuli to hit all desired coverage targets.

In constraint-random verification, some of the stimuli are represented as random variables. The values of these variables are randomly selected to generate different stimuli. The variables may be grouped into classes, with the random variables randomized one class at a time. In other words, the random variables of various classes are randomized individually during the course of simulation.

However, generating a set of stimuli that covers all coverage targets may take a long time using this approach. Some coverage targets may depend on random variables from different classes, but these random variables from different classes are not randomized together. Only the variables in the current class are randomized, without considering the current values of the variables that are outside the current class. This may result in many combinations of random variables that do not increase the coverage, particularly when trying to hit coverage targets that occur only rarely given the default distributions for the randomization.

In one aspect, coverage closure may be accelerated by biasing the randomization. Rather than using the default distribution and constraints, the randomization may be biased to increase the chance of hitting coverage targets by considering the current coverage (e.g., which coverage targets are not yet hit) and the values of other variables that also affect these coverage targets. The randomization may be biased towards increasing the chance of hitting certain coverage targets, given the values of these other variables.

The conditions within which the randomization occurs may be referred to as the context. The values of other variables may be referred to as context values, and the other variables may be referred to as context variables. Which variables are context variables depends on the class that is being randomized and the coverage targets that are being considered.

In one implementation, variables of different classes are randomized one class at a time in different stages of the constraint-random verification, and data structures are used to store and pass the context between these stages. Before run-time, a compiler (e.g. implemented using a processing device performing instructions) may analyze a coverage model to determine which coverage targets depend on which random variables, and this context connectivity information may be stored in a database. At run-time, the values of the random variables and the coverage (e.g., holes in the coverage) may be tracked as the constraint-random verification progress. At each stage, which random variables are relevant to the current coverage targets may be determined from the context connectivity information, and the current values for the random variables outside the currently randomized class may be determined from the tracked context values. The randomization of the selected class may then be biased to hit holes in the coverage targets, given the context values for the variables outside the class.

Technical advantages of the present disclosure include, but are not limited to, the following. Automated coverage closure enables teams to accelerate and improve the quality of verification, and the overall design process. Biasing the randomization increases the chance of generating stimuli that will hit coverage holes. This reduces the number of stimuli generated to reach coverage closure or other coverage goals. This reduces the overall time required to generate sufficient stimuli to reach coverage closure. It also reduces the overall time required for functional verification using the stimuli. With fewer stimuli, the associated processor, memory and data bandwidth requirements are also reduced. Fewer processor cycles are required to simulate the test cases using fewer stimuli, less memory is required to store the fewer stimuli and the results of their simulations, and less data bandwidth is required to move all of this data around.

FIG.1is a flow diagram for generating test stimuli, in accordance with some embodiments of the present disclosure. The flow generates stimuli that will be used in functional verification of a circuit design. The flow receives a description of the inputs (stimuli) for the functional verification, which are defined by variables. The variables115are grouped into classes110. InFIG.1, class1 includes variables {var1, var2, . . . , rand1, rand2, . . . } where rand* are random variables117, and so on for class2, class3, etc.

The classes are descriptions of different objects or constructs used in the circuit design. For example, there may be a class defined for packets. It might have a command field, an address, a sequence number, a time stamp, and a packet payload. In addition, there are various actions that can be done with a packet: initialize the packet, set the command, read the packet's status, or check the sequence number. Each packet is different, but as a class, packets have certain intrinsic properties that can be captured in the class definition. The class definition includes the variables used in the description of the class. In other words, a class is a user-defined data type that encapsulates data and functions related to that data.

The verification process has coverage targets to be reached, which are defined in a coverage model120. The coverage model120defines the coverage targets130as a function of the variables115. Some of the variables115may be random variables117. The values of random variables are randomly selected according to some probability distribution, subject to constraints on the values.

For the verification process, values of the random variables are randomly selected according to the probability distribution for that variable and subject to the constraints on that variable.FIG.1shows multiple iterations150to generate stimuli for the verification. The flow inFIG.1biases the randomization to improve verification coverage. At160, the context of the verification is tracked. This includes maintaining the current values of random variables. At170, the values of some random variables are randomized. Different random variables or class(es) of random variables may be randomized during different iterations. The remaining random variables which are not randomized for the current iteration are context variables for that iteration. The randomization at170is biased at175to hit coverage targets, given the values of the context variables. For example, if there are holes in the coverage produced by the previous stimuli, then the randomization may be biased at175towards hitting those holes, thus increasing the overall coverage.

At180, the current coverage is determined. This includes determining which coverage targets are hit by the newly generated stimuli. At185, if coverage closure is not yet achieved (e.g., based on a threshold closure target), then more iterations are run, and more stimuli are generated.FIG.1shows a single loop of iterations150, but the process may be implemented using multiple loops. For example, steps180and185may not be checked after every new stimulus is generated. Rather, a group of stimuli may be generated and then steps180and185are performed once for the entire group.

FIG.2shows parts of a verification testbench and coverage model, in accordance with some embodiments of the present disclosure. In this SystemVerilog testbench example, the random variables r1 and r2 are data members of classes C1 and C2 respectively, as defined by lines210. These random variables are connected to the coverage target CR1 of covergroup CG, by lines220. In this specific example, the coverage target CR1 is the cross-product of the random variables r1 and r2.

In the procedural code270A,B of this example, the class objects C1_obj (of class C1) and C2_obj (of class C2) are randomized and the covergroup CG is sampled post each randomization. However, the two classes are randomized separately, at different stages in the verification. Lines270A implement the randomization of random variable r1, and lines270B implement the randomization of random variable r2. Without some sharing of context, each randomization will proceed without knowing that the random variables r1 and r2 are both connected to the coverage target CR1, resulting in slower coverage closure. With sharing of context, the randomizations may be biased to accelerate coverage closure. The bias based on shared context may be implemented in the code of the randomize( ) methods.

Different types of bias may be implemented. Consider a simple example where random variables v1 and v2 are integer numbers constrained to fall within the range [0:10]. Let a coverage target CT be the sum of v1 and v2, so target CT has possible range of [0:20]. Previously generated stimuli covered values of CT from [0:15], so there is currently a coverage hole of (15:20] for CT. Assume that the two variables v1 and v2 are in different classes, so that only one of the two will be randomized during any stage. Let v1 be the randomized variable and let v2=8 for the current stage.

If there is no context sharing, then v1 will be randomized over the range [0:10]. However, lower values of v1 will not fill any of the coverage hole and will unnecessarily increase the time required for coverage closure. With context sharing and knowledge of the coverage holes in CT, v1 may be constrained to the range (7,10] so that any random values will hit some hole in the coverage. In this example, the bias was implemented by temporarily modifying the constraint on v1, changing its value range from [0:10] to (7,10]. The modification is temporary because different conditions in other stages may result in different constraints.

In an alternative approach, rather than modifying the constraint, the probability distribution for the randomization may be temporarily modified. The randomization for v1 uses a uniform distribution over [0:10]. This may be modified to skew towards the high end of the range, thus increasing the chance of hitting uncovered targets. In some cases, there may be multiple coverage targets that may interact in different ways. They may have overlapping requirements on the randomized variables, or they may have conflicting requirements. Modifying the probability distribution is one way to address multiple, possibly conflicting, requirements.

FIGS.3A and3Bare another flow diagram for generating test stimuli, in accordance with some embodiments of the present disclosure. This flow includes two parts: a compilation that occurs prior to run-time of the verification testbench shown inFIG.3A, and then the run-time of the verification shown inFIG.3B.

InFIG.3A, the compiler receives the coverage model320, for example as defined by classes “C1” “C2” and “coverage” inFIG.2. At325, the compiler analyzes the coverage model to determine which coverage targets depend on which random variables. In the example ofFIG.2, the coverage target “CR1” depends on the random variables “r1” and “r2.” This information connects different coverage targets to different contexts. It will be referred to as context connectivity information329. The dependencies of the coverage targets on the random variables may be one-to-many, one-to-one, and/or many-to-one. The context connectivity information329may be stored in a database for use at run-time.

The verification run-time is shown inFIG.3B. In addition to the context connectivity information329, the run-time also accesses values of random variables from prior stages (context values362) and extent of the current coverage364. In this example, the current coverage364is represented by a coverage holes scoreboard that tracks holes in the current coverage. The scoreboard may be stored as a database.

The verification is run in stages340. Each stage performs constrained random verification for a particular set of coverage targets and randomizing a specific class of random variables. The coverage targets and randomized class may change from stage to stage.

Each stage340proceeds as follows. At342, the coverages holes scoreboard364is accessed to determine coverage targets for the current stage. At344, the context connectivity information329is accessed to determine which random variables make up the context for the selected set of coverage targets. In the example ofFIG.2, if the current stage includes coverage target “CR1,” then the context connectivity information indicates that “CR1” depends on random variables “r1” and “r2.” The coverage target cg1.CR1 has the context {C1::r1, C2::r2}. Assume that class “C1” is the randomized class for the current stage, then “r1” is a randomized variable and “r2” is a context variable for this stage. At346, the values for the context variables are retrieved from the context values database362.

At350, multiple iterations of the verification are performed, using randomized values for variables in the randomized class. Continuing the above example, the value of “r1” is randomized. However, the randomization is biased to hit the selected coverage targets (e.g., holes in the coverage of “CR1”), given the context values for the context variables (e.g., the value of “r2” retrieved from the context values database362).

At355, the values of “r1” in the context values database are updated. Although not being randomized, the values of “r2” may also change and those values are also updated. Context values may change as a result of randomization of the class. They may also change as a result of assignments to variables in the course of the verification. At355, the coverage holes scoreboard364is also updated.

At340, the process is then repeated for the next stage. Assume that class “C2” is the randomized class for the next stage. Then “r2” will be a randomized variable and “r1” will be a context variable. The flow may perform stages sequentially, stepping through the classes one class at a time. Alternatively, different stages may be performed in parallel, with each stage updating the various databases362,364as the stage progresses.

Individual stages may be performed by a SystemVerilog constraint solver. The constraint solver treats the randomized variables as random and context variables as state variables that are not randomized. It solves for values of the random variables that hit the specified coverage target.

Consider the following example.

class TFoo;rand bit [3:0] a;constraint cb {a inside {[0:10]};}endclass
The random variable “a” is 4-bit wide variable with value range [0:15]. The inside constraint on “a” dictates that the valid value range for this variable is [0:10]. The constraint solver gathers this information and then solves for an appropriate value for “a”. Since the constraint solver is agnostic to the fact that the random variables of a class are connected to a coverage target, the generated solver solutions might not suffice in terms of efficient closure of coverage target.

The context values database362may be organized in different ways. In one approach, it is organized by class. For each class, the database maintains the context values for random variables outside the class. It may also be organized by coverage target. For each coverage target, the database maintains the context values for random variables on which the coverage target depends.

FIGS.4A and4Bshow experimental results for generating test stimuli, in accordance with some embodiments of the present disclosure. Both figures plot the percentage of coverage as a function of the number of iterations (number of stimuli generated).

The example ofFIG.4Ahas two SystemVerilog classes which contain random variables and are connected to one covergroup. Each random variable is 6 bits in size and can take 64 possible values. A cross product can take 64×64=4096 values. However, each random variable is independently randomized as they are in separate classes. Curve410shows coverage closure using the approach described herein. Each value of the cross product is hit within 4098 cumulative randomizations of the classes. Without this technique, on average it can take more than 20,000 cumulative randomizations as shown by curve411. Not having the context available and not considering it during randomization results in redundancy in the value generation during the randomization process.

FIG.4Bshows an example using three classes that are all connected to one covergroup. The covergroup has approximately 256,000 bins. A bin is a value or range of values for a coverage target. Curve420is for the approach described herein. It achieves coverage closure in approximately 265,000 iterations. In contrast, the approach of curve421without context sharing has achieved only 61% coverage at 300,000 iterations and will require more than 1,500,000 iterations to achieve complete closure.

FIG.5illustrates an example set of processes500used during the design, verification, and fabrication of an article of manufacture such as an integrated circuit to transform and verify design data and instructions that represent the integrated circuit. Each of these processes can be structured and enabled as multiple modules or operations. The term ‘EDA’ signifies the term ‘Electronic Design Automation.’ These processes start with the creation of a product idea510with information supplied by a designer, information which is transformed to create an article of manufacture that uses a set of EDA processes512. When the design is finalized, the design is taped-out534, which is when artwork (e.g., geometric patterns) for the integrated circuit is sent to a fabrication facility to manufacture the mask set, which is then used to manufacture the integrated circuit. After tape-out, a semiconductor die is fabricated536and packaging and assembly processes538are performed to produce the finished integrated circuit540.

During netlist verification520, the netlist is checked for compliance with timing constraints and for correspondence with the HDL code. During design planning522, an overall floor plan for the integrated circuit is constructed and analyzed for timing and top-level routing.

During analysis and extraction526, the circuit function is verified at the layout level, which permits refinement of the layout design. During physical verification528, the layout design is checked to ensure that manufacturing constraints are correct, such as DRC constraints, electrical constraints, lithographic constraints, and that circuitry function matches the HDL design specification. During resolution enhancement530, the geometry of the layout is transformed to improve how the circuit design is manufactured.

A storage subsystem of a computer system (such as computer system600ofFIG.6) may be used to store the programs and data structures that are used by some or all of the EDA products described herein, and products used for development of cells for the library and for physical and logical design that use the library.

The example computer system600includes a processing device602, a main memory604(e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), a static memory606(e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device618, which communicate with each other via a bus630.

The computer system600may further include a network interface device608to communicate over the network620. The computer system600also may include a video display unit610(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device612(e.g., a keyboard), a cursor control device614(e.g., a mouse), a graphics processing unit622, a signal generation device616(e.g., a speaker), graphics processing unit622, video processing unit628, and audio processing unit632.

The data storage device618may include a machine-readable storage medium624(also known as a non-transitory computer-readable medium) on which is stored one or more sets of instructions626or software embodying any one or more of the methodologies or functions described herein. The instructions626may also reside, completely or at least partially, within the main memory604and/or within the processing device602during execution thereof by the computer system600, the main memory604and the processing device602also constituting machine-readable storage media.