Patent ID: 7146305

Claim:
A computerized method for identifying malicious code in a target program running in a virtual machine of a computer system, the method comprising: evaluating a file format of the target program; evaluating control fields within a header of a file containing the target program; automatically configuring a memory map of the virtual machine by assigning areas of the memory map to receive predetermined types of data from the target program based on the file format in order to execute the target program, the virtual machine being capable of executing the target program in one of three modes of operation based on the file format and the control fields within the header of the file, a first mode of operation comprising a real mode for executing programs comprising instructions based on DOS, a second mode of operation for executing target programs comprising a high level programming language, and a third mode of operation comprising a protected mode for executing target programs comprising thirty-two bit code; constructing the virtual machine from one or more layered operating system shells that correspond with the memory map so that the virtual machine is capable of executing DOS target programs; simulating values of the computer system with the one or more layered operating system shells of the virtual machine; setting and resetting behavior flags in a register in order to track behavior of the target program in response to the simulated values during execution of the target program by the virtual machine; storing a sequence in which the behavior flags are set and reset in the register by the target program during execution of the target program by the virtual machine; passing behavior flag data and sequence flag data from the virtual machine to the computer system for evaluation after execution of the target program by the virtual machine; terminating the virtual machine after execution of the target program, thereby removing from the computer system a copy of the target program that was contained within the virtual machine; and evaluating the behavior flag data and sequence flag data with the computer system to determine if the target program contains malicious code.