Analyzing behavior of a device under test

A method of analyzing behavior of a device under test includes obtaining event traces that include a current sequence trace and a reference sequence trace. The event traces include one or more transactions that include one or more properties. A list of relevant properties of one or more transactions is obtained. A first set of n-tuples including values of the relevant properties for the current sequence trace is extracted. A second set of n-tuples including values of the relevant properties for the reference sequence trace is extracted. The first set of n-tuples is compared with the second set of n-tuples to indicate one or more transaction indices corresponding to differences in transactions between the current sequence trace and the reference sequence trace. Transactions corresponding to the transaction indices are annotated to obtain annotated transactions. The current sequence trace and/or the reference sequence trace are displayed with the annotated transactions.

BACKGROUND

Technical Field

The embodiments herein generally relate to electronic systems, and, more particularly, a system and method to analyze behavior of a computing device under test based on difference between event traces.

Description of the Related Art

Electronic systems are groupings of electronic circuits and components which are designed to accomplish one or more complex functions. The electronic systems which includes telecommunication systems, computer systems, power distribution systems, and electronic music systems. In a process of designing, complex electronic systems can be tested by different techniques. Technical specialists include engineers frequently implement large test-benches, test-suits, assertion suits and emulation systems that are employed to verify that the complex electronic systems meet specific requirements that are mandated as per a functional specification. The test-benches, test-suits, assertion suits and emulation systems may report an error whenever output of a computing device under test is not as per behavior specified by functional specification. The technical specialists have traditionally implements manual process using visual inspection of traces which are time consuming approach and error prone.

In order to debug errors, the engineers need to analyze the behavior of the system to conclude on why the device under test misbehaved. The analysis and conclusion may assist the engineers in fixing errors in the device under test. However, with the complexity of electronic systems increasing with each successive generation, time taken to analyze and debug issues is increasing exponentially. Hence, leads to loss of productivity for the users and associated teams. Accordingly, there remains a need for a new tool to assist users in quickly analyzing failures, increasing productivity, and efficiency of the electronic systems.

SUMMARY

In view of the foregoing, an embodiment herein provides a method of analyzing behavior of a device under test. The method includes obtaining from a device under test simulation model event traces that include a current sequence trace and a reference sequence trace. The event traces include one or more transactions that include one or more properties. A list of relevant properties of one or more transactions is obtained from a user from the one or more properties. A first set of n-tuples including values of the relevant properties for the current sequence trace is extracted. The n is equal to a number of the relevant properties, and a number of the first set of n-tuples is equal to a number of transactions.

A second set of n-tuples including values of the relevant properties for the reference sequence trace is extracted. The n is equal to a number of the relevant properties, and a number of the second set of n-tuples is equal to a number of transactions. The first set of n-tuples is compared with the second set of n-tuples to indicate one or more transaction indices corresponding to differences in transactions between the current sequence trace and the reference sequence trace. Transactions corresponding to the transaction indices are annotated to indicate the differences in the transactions to obtain annotated transactions. The current sequence trace or the reference sequence trace, or both the current sequence trace and the reference sequence trace is displayed with the annotated transactions.

Each of the first or second set of n-tuples may be given a unique name and character combinations. The character combinations may include one or more characters. A type of difference between the transactions of the current sequence trace and the reference sequence trace may be determined. The type of difference between the transactions may be indicated. A start time for the at least one event traces, and an end time for the at least one event traces may be recorded. The current sequence trace and the reference sequence trace are timed may include at least one time stamp. The current sequence trace may be converted into the reference sequence trace by minimizing number of operations required to transform one sequence into other sequence based on a shortest edit script or longest common subsequence algorithm. The operations may include start, insertion, or deletion of transactions from sequences to transform one sequence to another sequence.

In another embodiment, a computer implemented system for analyzing behavior of a device under test is provided. The system includes (i) a memory unit that stores (a) a set of modules, (b) a database, and instructions, wherein the database includes (i) information associated with a current sequence trace and a reference sequence trace, and (ii) a processor when configured by the instructions executes the set of modules. The set of modules include an event traces obtaining module, a properties obtaining module, a first n-tuples extracting module, a second n-tuples extracting module, a n-tuples comparing module, and a display module. The event traces obtaining module, executed by the processor, obtains event traces including a current sequence trace and a reference sequence trace. The properties obtaining module, executed by the processor, obtains a list of relevant properties of the plurality of transactions from the plurality of properties from a user. The first n-tuples extracting module, executed by the processor, extracts a first set of n-tuples including values of the relevant properties for the current sequence trace. The n is equal to a number of the relevant properties, and a number of the first set of n-tuples is equal to a number of transactions. The second n-tuples extracting module, executed by the processor, extracts a second set of n-tuples including values of the relevant properties for the current sequence trace. The n is equal to a number of the relevant properties, and a number of the second set of n-tuples is equal to a number of transactions. The n-tuples comparing module, executed by the processor, compares the first set of n-tuples with the second set of n-tuples to indicate one or more transaction indices corresponding to differences in transactions between the current sequence trace and the reference sequence trace. The display module, executed by the processor, displays the current sequence trace or the reference sequence trace at each of the transaction indices to indicate the differences in the transactions.

The set of modules may further include a transaction converting module and an annotating module. The transaction converting module that converts transactions of the current sequence trace into character combinations based on comparison between transactions of the current sequence trace and the reference sequence trace. The annotating module may annotate transactions corresponding to the transaction indices to indicate the differences in the transactions to obtain annotated transactions. The database further includes information associated with an untimed or a timed difference of the current sequence trace and the reference sequence trace. The event traces obtaining module may compare the current sequence trace and the reference sequence trace based on the least one of (i) the start time for the at least one event traces, and (ii) the end time for the at least one event traces. The event traces may be obtained from an emulation or a hardware prototyping system. The information from the emulation or the hardware prototyping system may be collected and processed in order to construct a new event trace. The event traces may be obtained by listening to a network socket or a stream or a data communication.

In yet another embodiment, one or more non-transitory computer readable storage mediums storing one or more sequences of instructions is provided, which when executed by one or more processors, causes obtaining, from a device under test simulation model, event traces that include a current sequence trace and a reference sequence trace. The event traces include a plurality of transactions that include one or more properties. A list of relevant properties of one or more transactions is obtained from a user from the one or more properties. A first set of n-tuples including values of the relevant properties for the current sequence trace is extracted. The n is equal to a number of the relevant properties, and a number of the first set of n-tuples is equal to a number of transactions.

A second set of n-tuples including values of the relevant properties for the reference sequence trace is extracted. The n is equal to a number of the relevant properties, and a number of the second set of n-tuples is equal to a number of transactions. The first set of n-tuples is compared with the second set of n-tuples to indicate one or more transaction indices corresponding to differences in transactions between the current sequence trace and the reference sequence trace. A type of difference between the current sequence trace meta string and the reference sequence trace meta-string is determined for each of the differences in the character combinations between the current sequence trace meta-string and the reference sequence trace meta-string. The current sequence trace or the reference sequence trace at each of the transaction indices is annotated and displayed to indicate the differences in the transactions to indicate the type of difference.

Each of the first or second set of n-tuples may be given a unique name and character combinations. The character combinations may include one or more characters. The current sequence trace and the reference sequence trace may be selected from a group including of (i) signal level traces, (ii) symbol level traces, (iii) protocol level traces, or (iv) combination thereof.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

As mentioned, there remains a need for a new tool to assist users in quickly analyzing failures, increasing productivity, and efficiency of the electronic systems. The embodiments herein achieve this by providing a client application that analyzes behavior of a computing device that is under test based on difference between one or more event traces. The client application supports a user to represent the one or more event traces in a form for comparison of the traces. A user may compare one or more event traces that form an input or an output of the system/device under test or its components or logic circuits. The client application compares and analyzes one or more event traces to show multiple difference outputs. Referring now to the drawings, and more particularly toFIGS. 1 through 6, where similar reference characters denote corresponding features consistently throughout the figures, these are shown preferred embodiments.

FIG. 1illustrates a system view of a behavioral analysis of a device under test (DUT) simulation model102based on a client application108within one or more computing devices104A-B according to an embodiment herein. The system view100includes the device under test (DUT) simulation model102, the one or more computing devices104A-B, a network106, and the client application108. The client application108obtains one or more event traces (e.g., a current sequence trace, and a reference sequence trace) as an input from the device under test (DUT) simulation model102. In one embodiment, the current event trace and reference event trace might be generated from the same DUT simulation model or from two different simulation models. The client application108compares one or more event traces to identify differences that point to an issue/problem. In one embodiment, one or more event traces are signal level traces, symbol level traces, a protocol level traces, and combination thereof. In one embodiment, the device under test (DUT) simulation model and/or a device under test (DUT) emulation system interact with the one or more computing devices104A-B for a behavioral analysis. In one embodiment, the one or more computing devices104A-B acts as a host system for simulation, and the computing device104B acts as an analytics system and vice versa.

In one embodiment, the event traces are a list of transactions. The transaction is a communicative action or activity involving one or more components of a system. The transaction properties are special quality or characteristics of a transaction (for e.g., such as size, direction, origin, address etc). In one embodiment, one or more event traces are timed (each event trace step carries as timestamp and two events are considered as not different when one or more identified attributes are same and the two events occur at the same timestamp) or untimed (two events are considered as not different when one or more identified attributes are same). In one embodiment, the client application108is implemented in the computing device104A. In another embodiment, the client application108is implemented in the computing device104B. The client application108interacts with the computing device104B through the network106to analyze the behavior of one or more event traces. In one embodiment, the network106is an internet. In one embodiment, the one or more computing devices104-B is a personal computer, a mobile phone, a smart phone, a tablet PC, a laptop, and an ultra-book, component chip of a system, a sub-component/logic circuit of the chip. The client application108represents the one or more event traces in a form that may allows comparison of event traces. The one or more event traces may be represented in a graphical representation. The client application108compares and analyzes the one or more event traces to show multiple difference outputs. In one embodiment, the one or more event traces are compared using different algorithms (e.g., a shortest edit script/longest common subsequence algorithm). The one or more differences associated with one or more event traces may be represented in a graphical representation. The one or more differences associated with one or more event traces are displayed to a user in the display unit (as shown inFIG. 3C). In one embodiment, one or more traces is obtained from an emulation or hardware prototyping system. The information from such a system can be collected and processed in order to construct the event trace. In one embodiment, the input traces are obtained from a file stored on a storage device such as a hard disk. In another embodiment, the input traces are obtained by listening to a network socket or stream. The examples are not exhaustive and the input may be obtained by other similar means of data communication. In one embodiment, the reference trace may be a golden, error free trace while the current trace may be the output of the DUT and might show incorrect behavior. The comparison between the reference trace and current trace and subsequent annotation of the current trace with the differences would help a user in identifying the cause of the abnormal behavior. In one embodiment, each element of the current trace or reference trace can be annotated as “addition” or “deletion” or “same” by the client application108.

FIG. 2illustrates an exploded view of the client application108ofFIG. 1according to an embodiment herein. The exploded view108includes a data base202, an event traces obtaining module204, a relevant properties obtaining module206, a first n-tuples extracting module208, a second n-tuples extracting module210, a n-tuples comparing module212, an annotating module214, a display module216, and a transaction converting module218. In one embodiment, a tuple is a data structure that has a specific number and sequence of elements. The database202stores information associated with a one or more event traces, and information associated with untimed/timed difference of event traces. The event traces obtaining module204comprises a current sequence trace and a reference sequence trace. The event traces obtaining module204comprises one or more transactions that comprise one or more properties. The one or more transactions include one or more first transactions and one or more second transactions. The one or more event traces may be compared based on a shortest edit script/longest common subsequence algorithm

For example, given two traces, deleting and/or inserting trace elements such as transactions, packets, transfers, into specific part of a trace may convert one trace to another trace. The current sequence trace is converted into the reference sequence trace by minimizing the number of operations (insertion/deletion) required to transform one sequence into the other sequence based on computing the SES/LCS using a suitable algorithm. The relevant properties obtaining module206obtains a list of relevant properties of the one or more transactions from the one or more properties from a user. The user is presented with a list of properties that are relevant to a trace element. The user then selects which properties he considers as relevant for the computation of the difference. For example, a trace might consist of thousands of transactions, and each transaction might have 5 properties: source address, destination address, type, size and a sequence-number. The user might mark the source address, destination address, type and size properties as the relevant properties (leaving out sequence-number). The user might do this as every time the DUT is simulated, even though the traffic pattern between a source and destination address pair might be the same, but the starting value of the sequence-number might be different due to the functional requirements of the DUT. By marking the relevant properties, the user can make sure that any differences in the irrelevant properties (in this case, the sequence-number) does not show up as a difference in the output of the client application108. In one embodiment, the first n-tuples extracting module208extracts a first number of n-tuples comprising value of the relevant properties for the current sequence trace. In one embodiment, n is equal to a number of the relevant properties of the one or more first transactions, and a number of the first number of n-tuples is equal to a first number of the one or more first transactions. The second n-tuples extracting module210extracts a second number of n-tuples comprising value of the relevant properties for the current sequence trace. In one embodiment, n is equal to a number of the relevant properties of the one or more second transactions, and the second number of n-tuples is equal to a second number of the one or more second transactions.

The a n-tuples comparing module212compares the first one or more n-tuples with the second one or more n-tuples to indicate one or more transaction indices corresponding to differences in transactions between the current sequence trace and the reference sequence trace. In one embodiment, an algorithm such as SES/LCS can be used to compute the differences. The algorithm will output “same”, “addition” or “deletion” for every transaction index of the current or reference or both input traces. In one embodiment, the annotating module214annotates transactions corresponding to the transaction indices to indicate the differences in the transactions to obtain annotated transactions. The display module216displays the current sequence trace or the reference sequence trace or both alongside at each of the transaction indices to indicate the differences in the transactions.

With reference toFIG. 3AandFIG. 3B,FIG. 3Bis an exemplary view300B illustrates an untimed differences of two traces according to an embodiment herein. The exemplary view300A represents an example embodiment of untimed differences of two sequence traces (a current sequence trace, and a reference sequence trace). For example, considering two sequences to be S1, S2as shown inFIG. 3A. For the given traces, the user specifies that he considers the Device, Dir (direction) and Size properties of the trace transactions as relevant and does not consider the time property as relevant. This would mean that if, due to some reason, the time stamps in the current trace change, when compared with the reference trace, the client application108will not show any difference only if the timestamps are different. The client application108will show a difference if some other relevant property of the transaction is also different. A first sequence S1corresponds to the reference sequence trace. Similarly, a second sequence S2corresponds to the current sequence trace. For simplification and conceptual clarity, every unique n-tuple is given a label such as a1, a2, a3, a4, a5as shown inFIG. 3B.FIG. 3Cshows the application of the SES/LCS algorithm to compute the differences between the two traces. The labels on the top horizontal axis correspond to the labels for the n-tuples for every transaction in the current trace and the labels on the left vertical axis correspond to the labels for the n-tuples for every transaction in the reference trace.

FIG. 3Cillustrates the identification of differences300C of the two sequence traces according to an embodiment herein. The representation300B represents a shortest edit script (SES) graph. The SES algorithm will give a path from (0,0) to (6,5) such that it is composed of the least number of addition/deletion operations required to change one trace into the other trace. For example, a path is represented as (0, 0) to (6, 5) and an operation which includes insertion/deletion of transactions from traces which converts one trace to another trace. The edit graph which represents one or more operations (e.g., match insert, delete) performed at corresponding path (0, 0) to (6, 5) for two sequence traces (e.g., S1and S2). The one or more differences associated with the two sequence traces may be identified based on the edit script.

FIG. 3Dis a user interface view300D illustrates one or more differences associated with the two sequence traces according to an embodiment herein. The user interface view300C displays one or more differences of the two sequence traces. In one embodiment, the one or more differences associated with the two sequence traces are represented one or more indication (e.g., one or more color code). For example, (i) a green color code that represents insertion of a transaction, (ii) a red color code that represents a deletion of a transaction, and (iii) a grey color code that represents modification in the reference sequence.

FIG. 4is a flow diagram400illustrating a method of analyzing behavior of the computing device under test based on difference between one or more event traces according to an embodiment herein. In step402, event traces that include a current sequence trace and a reference sequence trace is obtained. In step404, a list of relevant properties of one or more transactions from one or more properties of the event traces is obtained. In step406, a first set of n-tuples including values of the relevant properties for the current sequence trace is extracted. In step508, a second set of n-tuples including values of the relevant properties for the reference sequence trace is extracted.

In step410, the first plurality of n-tuples with the second plurality of n-tuples to indicate one or more transaction indices corresponding to differences in transactions between the current sequence trace and the reference sequence trace is compared. In step412, transactions corresponding to the transaction indices to indicate the differences in the transactions to obtain annotated transactions is annotated. In step414, the current sequence trace or the reference sequence trace, or both the current sequence trace and the reference sequence trace with the annotated transactions is displayed.

FIG. 5illustrates an exploded view of the one or more computing devices102A-B having an a memory502having a set of computer instructions, a bus504, a display506, a speaker508, and a processor510capable of processing a set of instructions to perform any one or more of the methodologies herein, according to an embodiment herein. In one embodiment, the receiver may be the personal communication device. The processor510may also enable digital content to be consumed in the form of video for output via one or more displays506or audio for output via speaker and/or earphones608. The processor510may also carry out the methods described herein and in accordance with the embodiments herein.

Digital content may also be stored in the memory502for future processing or consumption. The memory502may also store program specific information and/or service information (PSI/SI), including information about digital content (e.g., the detected information bits) available in the future or stored from the past. A user of the one or more computing devices102A-B may view this stored information on display506and select an item of for viewing, listening, or other uses via input, which may take the form of keypad, scroll, or other input device(s) or combinations thereof. When digital content is selected, the processor610may pass information. The content and PSI/SI may be passed among functions within the one or more computing devices102A-B using the bus604.

The techniques provided by the embodiments herein may be implemented on an integrated circuit chip (not shown). The chip design is created in a graphical computer programming language, and stored in a computer storage medium (such as a disk, tape, physical hard drive, or virtual hard drive such as in a storage access network). If the designer does not fabricate chips or the photolithographic masks used to fabricate chips, the designer transmits the resulting design by physical means (e.g., by providing a copy of the storage medium storing the design) or electronically (e.g., through the Internet) to such entities, directly or indirectly.

The stored design is then converted into the appropriate format (e.g., GDSII) for the fabrication of photolithographic masks, which typically include multiple copies of the chip design in question that are to be formed on a wafer. The photolithographic masks are utilized to define areas of the wafer (and/or the layers thereon) to be etched or otherwise processed.

A representative hardware environment for practicing the embodiments herein is depicted inFIG. 6. This schematic drawing illustrates a hardware configuration of an information handling/computer system in accordance with the embodiments herein. The system comprises at least one processor or central processing unit (CPU)10. The CPUs10are interconnected via system bus12to various devices such as a random access memory (RAM)14, read-only memory (ROM)16, and an input/output (I/O) adapter18. The I/O adapter18can connect to peripheral devices, such as disk units11and tape drives13, or other program storage devices that are readable by the system. The system can read the inventive instructions on the program storage devices and follow these instructions to execute the methodology of the embodiments herein.

The system further includes a user interface adapter19that connects a keyboard15, mouse17, speaker24, microphone22, and/or other user interface devices such as a touch screen device (not shown) or a remote control to the bus12to gather user input. Additionally, a communication adapter20connects the bus12to a data processing network25, and a display adapter21connects the bus12to a display device23which may be embodied as an output device such as a monitor, printer, or transmitter, for example.

A representation of an event trace may be as a string of a meta-alphabet based on the SES/LCS algorithm to the trace comparison problem. The client application108supports at least one of (i) reduction in npoints data structure size, (ii) reduction of a timed sequence to an untimed sequence to allow comparison between traces (otherwise exact time match is a problem), (iii) representation of a signal trace as a string of a meta-alphabet, (iv) show all possible paths by considering all possible arbitrary decisions between reference and current, (v) ability to choose trace sections (start/n markers) that need to be compared, and (vi) ability to get a golden trace from a golden model and compare the result of a test vector with the golden trace. The client application108enables users (e.g., an engineer) to analyze the behavior of a device under test and develop insights by easily comparing two traces. The system can couple the understanding of the functional specification and larger system requirements, which helps an engineer in narrowing down the possible issues in his/her design. The engineer can then proceed to work on eliminating the differences or understanding them in more detail to fix the design.