Abstract:
A system and method are provided that use pass/fail test results to prioritize electronic design verification review issues. It may prioritize either generated properties or code coverage items or both. Thus issues, whether generated properties or code coverage items, that have never been violated in any passing or failing test may be given highest priority for review, while those that have been violated in a failing test but are always valid in passing tests may be given lower priority. Still further, where end-users have marked one or more properties or code coverage items as already-reviewed, the method will give these already-reviewed issues the lowest priority. As a result, both properties and code coverage items may be generated together in a progressive manner starting earlier in development and significant duplication of effort is avoided.

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
       [0001]    This application claims priority under 35 U.S.C. 119(e) from prior U.S. provisional application 62/041,661, filed Aug. 26, 2014. 
     
    
     TECHNICAL FIELD 
       [0002]    The invention relates to integrated circuits verificatin, e.g. by means of simulation, and more particularly to systems, methods and computer program products for prioritizing electronic design verification review issues. 
       BACKGROUND ART 
       [0003]    Electronic chip designers continue to develop electronic chips of ever increasing complexity using more and more transistors. Verifying the behavior of an electronic chip has becoming increasingly difficult and time-consuming. A considerable amount of engineering time is spent running and analyzing simulation results. 
         [0004]    Design verification teams spend many months developing, running and analyzing simulation tests and their results. The verification teams typically first develop functional tests, also known as directed tests. These functional tests are designed to test expected behavior as described in a functional specification. The functional tests check that the design works in all modes and configurations. When the verification teams first run the functional tests they typically show errors in the design and errors in the test. After some period of time, after correcting the design and test errors, the design and verification teams will agree one or more regression test suites. The verification teams will run daily and weekly regression tests. 
         [0005]    After functional test verification, the verification team will typically start random testing. Random tests typically check scenarios with different sets of pseudo-random test inputs. Random testing usually shows fewer errors than functional testing. 
         [0006]    After random testing, the verification team typically starts code coverage analysis. They want to ensure that all lines of RTL code are exercised. Verification teams typically use simulators to generate code coverage reports using the previously developed tests for data input. They analyze these reports and try to ensure all lines of RTL code are exercised. The verification teams often find “dead-code”, code that isn&#39;t needed, and they find situations that the functional tests didn&#39;t test but should have tested. The code coverage project phase generally finds few design errors but is considered a necessary step. The code coverage reports provide a large volume of detailed information. Verification teams and design engineers find it time-consuming and tedious to review the code coverage issues. 
         [0007]    Electronic design automation (EDA) tools are making increased use of verification properties to augment simulation and reduce the verification cost. A verification property declares a condition in the design. If a property always holds true, we call them an assertion. For example, a property “overflow==1′b0” should always hold for any correct FIFO design. On the other hand, a property can capture possible behavior allowed by the design; we call such a property a cover property. For example, a property “full==1′b1” can be a typical coverage property on the same FIFO design. For the given above two examples, we typically write them as:
       assert overflow==1′b0   cover full==1′b1       
 
         [0010]    Users can specify verification properties in an RTL file or in a separate constraint file. They are typically written in specific language including System Verilog Assertions (SVA) or Property Specification Language (PSL). Many EDA tools can parse/generate verification properties. Some EDA tools can generate verification properties by analyzing RTL statements. Other EDA tools can generate verification properties by analyzing simulation test results. 
         [0011]    Atrenta Inc.&#39;s Bugscope® EDA tool has proven valuable in finding test coverage holes. Bugscope® generates verification properties by analyzing simulation, test results. For example it may note that in one test the condition “full==1′b0” is always true. If Bugscope® discovers that the same condition is false in a second test it treats the condition as a coverage property and generates the property “cover full==1′b1”. The coverage property condition is inverted with respect to the discovered property to direct a simulator to check for the inverted condition. We say that the second test covers the property. Bugscope® may note that the condition “overflow==1′b0” is always true in all tests. In this case it cannot tell if the property is a coverage property or an assertion. 
         [0012]    Using verification properties places an additional review burden on the verification engineer. In addition to reviewing code coverage data the verification team must also review generated properties and simulation property results. 
         [0013]    Verification teams would prefer EDA tools that simplify the task of reviewing code coverage and generated property results. Design and verification teams would like to speed up the overall development schedule by reducing time spent reviewing coverage items and get coverage information earlier in the development schedule. 
       SUMMARY DISCLOSURE 
       [0014]    A system and method are provided that use pass/fail test results to prioritize electronic design verification review issues. It may prioritize either generated properties or code coverage items or both. Thus issues, whether generated properties or code coverage items, that have never been violated in any passing or failing test may be given highest priority for review, while those that have been violated in a failing test but are always valid in passing tests may be given lower priority. Still further, where end-users have marked one or more properties or code coverage items as already-reviewed, the method will give these already-reviewed issues the lowest priority. 
         [0015]    As a result, both properties and code coverage items may be generated together in a progressive manner starting earlier in development. Properties for unchanged modules that have already been verified from a previous version of a chip can be removed or given lowest priority to avoid duplication of effort. Likewise, properties and code coverage items that are only violated at failing tests may be removed or given lower priority so that repetitive testing of such issues at every design regression can be minimized or avoided altogether. The number of issues to review is therefore significantly smaller than the old approach. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0016]      FIG. 1  shows a simulation test process. 
           [0017]      FIG. 2  shows an example table of property results on different tests. 
           [0018]      FIG. 3  shows a flowchart, in accord with the present invention, outlining the steps for prioritizing review properties. 
           [0019]      FIG. 4  shows a block diagram of a Verification Issue Rating System in accord with the present invention. 
       
    
    
     DETAILED DESCRIPTION 
       [0020]    A Verification Issue Rating System (VIRS) in accord with the present invention uses pass/fail test results to prioritize electronic design verification review issues. Properties that have never been violated in any passing or failing test are given highest priority. Properties that have been violated in a failing test are given lower priority. Similarly, code coverage items that have never been exercised in any passing or failing test are given highest priority. Code coverage items that have been exercised in a failing test are given lower priority. 
         [0021]    Verification teams typically use simulators to generate code coverage reports to discover which lines of RTL design code have not been exercised. For example, an RTL design may include a case statement specifying four conditions corresponding to the four combinations of values for a pair of binary signals. The code coverage report may report that cases 00, 01 and 10 are exercised but the case 11 is not exercised. The RTL code for case 11 is flagged for engineering review. 
         [0022]    EDA tools like Bugscope® generate properties by analyzing a design and its test simulation results. Verification engineers review the generated properties looking for test coverage holes and design errors. The generated properties may indicate a relationship that should always be true, called an assertion. The generated property may indicate a situation that hasn&#39;t been tested, called a coverage property. The generated property may also indicate a design error. Coverage properties that are true in all tests indicate a test coverage hole. Verification engineers are more interested in these types of coverage properties than in assertions. A coverage property may indicate that the verification team needs to generate a new test. 
         [0023]    During the early stages of development it is common to see test failures and assertion property violations. These assertion property violations are often the result of design errors. For example, a verification engineer may know that a FIFO should not overflow. The verification engineer creates functional tests trying to create a FIFO overflow and may manage to show conditions under which FIFO overflow can occur. After a design engineer corrects the design the test passes and the assertion properties pass. Tests that failed previously and now work give a strong indication that the failing properties were assertion properties and not coverage properties. To take advantage of this information the EDA tools need to maintain a database of test results over time. 
         [0024]      FIG. 1  shows a diagram  100  that illustrates the test process. The design engineer creates test stimuli  110  that define waveform signal inputs. The simulator  120  simulates the design using test stimuli  110  and creates results that are fed into a result checker  130 . The result checker  130  checks the results and decides if the test passes or fails. In some cases the result checker will have a file of expected results that it compares to the actual results. In other cases the checker will utilize an independent model of expected behavior, generate expected results and compare them to the simulation results. The engineering team may have made mistakes in the result checker  130 , in the design being simulated in simulator  120 , or in the test stimuli  110 . 
         [0025]    In the case of a result checker error, the design behavior observed in the failure test is legal but the result checker thinks it is illegal. Assume that there is a property P which gets violated in this failure test. Given that it is a checker failure, the verification engineer will fix the result checker while the stimulus will be kept the same. When the result checker is fixed, you will see that this property P still gets violated. The VIRS will ignore this property P because it cannot be an assertion and it isn&#39;t a coverage property that is true for all tests. 
         [0026]    In the case of a design error, assume that a property P is violated. When the design is fixed, testing P can have only two consequences: a) P holds true; b) P is still incorrect. For case a), P is very likely to be an assertion because it is violated in a failure test and once it is fixed, it holds. For case b), P must be a coverage property. This implies that the failure test exercises a previously uncovered corner case, and therefore finds a bug. When the design is fixed, the test still exercises this corner case, and that is why this P is still violated. In this way, we find with high probability that a property is an assertion. The VIRS uses this information to prioritize both generated properties as well as code coverage. 
         [0027]    In the case of a test stimulus error, the analysis is very similar to that of the design error. The only difference is that the verification engineer has to fix test stimuli instead of the design. Testing the violated property P will still have two consequences: a) P holds true; b) P is still incorrect. For case a), P is very likely to be an assertion. For case b), P must be a coverage property. 
         [0028]      FIG. 2  shows an example table  200  showing property failures in passing and failing tests. The column heading  210  shows the names of the properties, P 1 , P 2  . . . P 1000 . The row heading  220  shows the names of the tests and for each test for a passing row and a failing row. In this example property P 1  is violated in failing tests T 1   f  and T 2   f.  The VIRS only considers properties that are true in all passing tests. The VIRS gives high review priorities to those properties that are true in all passing and failing tests. The VIRS gives lower review priorities to those properties are true in all passing tests but have been violated in failing tests. In this example property P 1  has low review priority and the other properties have high review priority. 
         [0029]      FIG. 3  is an exemplary and non-limiting flowchart  300  for prioritizing electronic design verification property issues. The same approach applies to code coverage issues. In S 310  the VIRS reads the Test Results Database and creates a list of properties that are true in all passing tests. In S 320  the VIRS starts a loop that processes the properties identified in S 310 . The first time the VIRS executes S 320  the VIRS selects the first property. On subsequent iterations the VIRS processes the next property. In S 320  the VIRS decides if the selected property has high review priority. The VIRS assigns high priority if the property is true in all passing and failing tests. If the VIRS decides the property has high review priority the VIRS proceeds to S 330 . If the VIRS decides the property does not have high review priority the VIRS proceeds to S 340 . At S 330  the VIRS adds the selected property to a list of high priority review items and proceeds to S 350 . At S 340  the VIRS adds the selected property to a list of low priority review items and proceeds to S 350 . At S 350  the VIRS decides if it has finished processing all properties in the list identified in S 310 . If the VIRS has more properties to process the VIRS it proceeds to S 320 . If the VIRS has no more properties to process the VIRS it proceeds to S 360 . In S 360  the VIRS reports high and low priority review items using the lists it constructed at S 330  and S 340 . In one embodiment the VRS stores the high and low priority review items in a report file. 
         [0030]    The VIRS handles code coverage review items in a similar manner to the way it handles properties. In S 310  the VIRS would list code coverage items instead of properties. In S 320  the VIRS would give high priority to code coverage items that aren&#39;t covered in any passing or failing test. Subsequent steps would apply to code coverage items. 
         [0031]    In one embodiment the VIRS allows users to mark verification items as “already reviewed”. The VIRS takes account of the user&#39;s “already reviewed” designation when prioritizing review items. In one embodiment the VIRS creates 4 categories of review items: a) high review priority and not “already reviewed”; b) high review priority and “already reviewed”; c) low review priority and not “already reviewed”; and d) low review priority and “already reviewed”. 
         [0032]    The embodiments disclosed herein can be implemented as hardware, firmware, software, or any combination thereof. Moreover, the software is preferably implemented as an application program tangibly embodied on a program storage unit or computer readable medium. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units (“CPUs”), a memory, and input/output interfaces. The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, which may be executed by a CPU, whether or not such computer or processor is explicitly shown. In addition, various other peripheral units may be connected to the computer platform such as an additional data storage unit and a printing unit. Furthermore, a non-transitory computer readable medium is any computer readable medium except for a transitory propagating signal. 
         [0033]      FIG. 4  is an exemplary and non-limiting diagram  400  showing a Verification Issue Rating System (VIRS)  420  in accord with the present invention. The VIRS  420  runs as an application program on a central processing unit (CPU). The VIRS  420  interacts with a user through an input device,  440  and a display,  450 . Using the input device  440  the user starts the VIRS  420  execution and specifies VIRS  420  inputs. In one embodiment the VIRS  420  displays the prioritized issues in the form of a Rated Verification Issue Report  430  on the display  450 . The VIRS  420  reads verification issues from a Test Results Database  410 . The VIRS  420  uses the pass/fail history in the Test Results Database  410  to generate prioritized issues. In one embodiment the VIRS is encapsulated as an application within an EDA tool-chain. In another embodiment the VIRS is encapsulated as a software module within another EDA application program.