Patent Description:
Many sandboxes are implemented as virtualized systems that can easily be rolled back to a clean state once the analysis is complete. However, these virtualized systems may not realistically simulate a real user's machine in one or more respects. Given the use of sandboxing to identify a malware threat, attackers have employed multiple techniques to detect the presence of such a virtual environment and change behavior of the malware application's behavior based on this detection. Moreover, some of today's malware is designed to bypass detonation using many different evasion techniques. A large amount of time and effort is spent by malware researchers in identifying these evasive techniques and patching the detonation environment to avoid to such checks.

Reference is made to <CIT> which describes a malware detection system for evaluating sample programs which incorporates an evasion code detector. The evasion code detector includes semantic patterns for identifying conditional statements and other features employed by evasion code. The system inserts breakpoints and conditional statements, compares expected and actual evaluated values of conditional values of the conditional statements, and changes the execution path of the sample program based on the comparison. Changing the execution path of the sample program to an expected execution pathcounteracts the evasion code, allowing for the true nature of the sample program to be revealed during runtime.

The disclosed embodiments provide for improved methods and systems of malware detonation. As described above, many security researchers rely on sandbox environments to analyze malware applications and better understand their behaviors. This understanding is used for a variety of purposes, including developing more effective malware detection techniques, identifying security vulnerabilities in existing software, and providing signatures or behavior profiles to improve run-time detection of malware applications.

As malware has become more sophisticated, techniques have been introduced by malware authors to detect when the malware is running in a sandbox environment. Once this is detected, the malware inhibits one or more of its features, making it more difficult and/or impossible to gather the forensics. Some malware monitors, for example, a number of CPUs of a system executing the malware. Since many sandbox environments are virtualized, the number of CPUs reported by an operating system is typically a low number, such as one (<NUM>). In contrast, many common desktop computers include multiple CPU codes, and in general report a higher number. The malware uses, in some embodiments, a low number of CPUs as evidence that it may be running within a sandbox environment. Another technique is for the malware to analyze a "documents" folder. If a number of files included in the documents date, and/or modification dates of said documents do not appear typical for a real user computer, some malware considers this as evidence of a sandbox environment. Thus, a technical problem is presented in that it can be increasingly difficult to detonate malware in a sandbox testing environment, where information about the malware can be obtained via analysis, and mitigating actions taken based on the analysis.

To solve this technical problem, and to make a sandbox environment less visible to potential malware being analyzed on it, the disclosed embodiments intercept program execution of the malware in order to obscure certain characteristics of the sandbox environment and cause the malware to perform an analysis consistent with a real, non-sandbox environment. If the malware is unable to detect the sandbox, it generally will function normally, allowing full analysis of its methods of operation.

To prevent detection of the sandbox environment, some embodiments intercept or otherwise hook programmatic execution of the suspected malware application. In some embodiments, each individual instruction, such as an assembly or machine level instruction, bytecodes, precompiled code (p-code), other intermediate code instruction, or even a source level instructions in some embodiments, is intercepted and provided to a reinforcement learning model. The reinforcement learning model is trained to provide modification instructions based on the intercepted assembly instructions. The modification instructions indicate one or more modifications to perform on the program execution of the suspected malware application. For example, the modifications instructions indicate in various embodiments, one or more of modification of a register value, modification of a function return value, modification of a function input parameter, modification of a condition evaluation in the malware code, or modification of a branch instruction in the malware code.

Some embodiments do not intercept instructions executed by the suspected malware as described above, but instead intercept function calls to one or more external application programming interface(s) API(s). For example, some embodiments intercept JavaScript library API calls and/or operating system API calls. Upon intercepting an API call, information relating to the API call is provided to a reinforcement learning model. The information relating to the API call can include input parameters passed to the API call, a call stack of the suspect malware application, and in some cases, indications of code of the suspect application that is within a vicinity of the API call. The reinforcement learning model is, as above, configured to provide program execution modification instructions based on the provided information. The reinforcement learning model indicates, in various embodiments, to modify a return value of the API, an output parameter of the API, an input parameter of the API, a modification to a condition evaluation in the suspect malware application itself, a modification of a branch instruction in the suspect malware application itself, or other modification.

<FIG> is an overview diagram of an example system <NUM> implementing one or more of the disclosed embodiments. The system <NUM> includes a web crawler computer <NUM>, that accesses a network <NUM> (e.g. the Internet) via a firewall <NUM>. The web crawler computer <NUM> identifies data available via the network <NUM> (e.g., via web sites) and identifies applications that attempt to run on a computer that downloads the data. For example, a website identified by the web crawler attempts, in some examples, to download and install an executable file on the downloading machine, and/or run a scripting language application within a browser environment provided by the downloading computer. In some embodiments, these identified applications are provided by the web crawler computer <NUM> to a sandbox environment <NUM>. In some embodiments, the potential malware applications are identified manually. The sandbox environment <NUM> attempts to execute these applications. The sandbox environment <NUM> is at least partially isolated from other environments within the system <NUM> so as to mitigate any potential damage that may be done by execution of these identified applications.

One result of the analysis performed within the sandbox environment <NUM> is identification of malware applications, or those application with a malicious intent when they execute on a particular computer. Malware applications can have a variety of goals, including data destruction, data collection, or ongoing spying on a network in which they have been able to obtain access. Some malware applications utilize their host computers as proxies for other nefarious activities. For example, some denial of service attacks are orchestrated by armies of otherwise innocent computers which have been infected with malware bots, configured to carry out the denial of service attack upon receiving a command from a central controller, which is some circumstances, is located offshore in difficult to police jurisdictions.

After a malware application is identified, it is stored, in some embodiments, in a malware data store <NUM>. The malware data store <NUM> is then used, in various embodiments, to improve protection against malware applications. For example, malware stored in the malware data store <NUM> is studied, in some embodiments, to identify behavioral patterns which can be detected by dynamic threat assessment applications protecting most modem computing systems. The malware stored in the malware data store <NUM> is further used, in at least some embodiments, to identify vulnerabilities in the designs and/or architectures of widely used computer operating systems and/or applications.

<FIG> is an overview diagram of an example system <NUM> implementing one or more of the disclosed embodiments. The system <NUM> includes a browser application <NUM>, and a first potential malware application <NUM>. Each of the browser application <NUM> and first potential malware application are applications managed by an operating system <NUM>. The browser application <NUM> and first potential malware application <NUM> both interface with the operating system <NUM> via an operating system application programming interface (API) <NUM>. When the operating system <NUM> is a Microsoft Windows based operating system, the browser application <NUM> and first potential malware application <NUM> are ". com" files, at least in some embodiments. When the operating system <NUM> is a Linux operating system, the browser application <NUM> and first potential malware application <NUM> includes, in some embodiments, object code compatible with a hardware platform running the operating system <NUM>.

<FIG> also shows a second potential malware application <NUM>. The second potential malware application <NUM> differs from the first potential malware application <NUM> in that the second potential malware application <NUM> runs within an environment provided by the browser application <NUM>. In some embodiments, the second potential malware application is a JavaScript application. The browser application <NUM> provides a script API <NUM> for use by the second potential malware application <NUM> running within the environment provided by the browser application <NUM>. The second potential malware application <NUM> interfaces with the script API <NUM> to accomplish various functions. In some embodiments, the script API <NUM> calls out to the OS API <NUM> as necessary to perform at least a portion of those functions.

As described below, the disclosed embodiment intercept instructions and/or API calls executed by the first potential malware application <NUM> and/or the second potential malware application <NUM>. These intercepted instructions and/or API calls are provided to a machine learning model, and a modification to be made to the first potential malware application <NUM> and/or second potential malware application <NUM>.

<FIG> is a dataflow diagram of example dataflow within an example sandbox architecture <NUM> that is implemented in one or more of the disclosed embodiments. <FIG> shows a potential malware application <NUM>. We refer to the application as a potential malware application <NUM> because in some cases, the application is a malware application and in some other cases, it may be a perfectly benign application. The potential malware application <NUM> includes code <NUM>. The code <NUM> is executable code in some embodiments. For example, the code <NUM> is native assembly or machine language code compatible with a hardware platform and operating system upon which the potential malware application is executing in some embodiments. In other embodiments, the code <NUM> is intermediate code, such as p-code or even scripting source code that is at least partially interpreted before execution by a hardware processor.

<FIG> shows an API hook <NUM> implemented by some of the disclosed embodiments. The API hook <NUM> is configured to intercept function calls to one or more APIs (e.g., script API <NUM> or OS API <NUM>) that are executed by the potential malware application <NUM> and the code <NUM>. In some embodiments, the API hook <NUM> is a debug application. In these embodiments, the debug application is configured to intercept program execution of the potential malware application <NUM> when an address outside of the potential malware application <NUM> code <NUM> is accessed (e.g., to access an API library for example, such as the API library <NUM> discussed further below. In some embodiments, an operating system (e.g., operating system <NUM>) supporting the sandbox architecture <NUM> provides an API hook capability, and thus a debugger is not necessary. Some embodiments utilize a hardware emulator to implement the API hook <NUM>. For example, in some embodiments, a traditional hardware processor of a computer is replaced by a hardware emulator that is able to simulate operation of the OEM hardware circuitry, and also allow monitoring and/or intercepting of programmatic activity being executed by the simulated hardware processor.

Upon intercepting an API call <NUM> (or any function call), the API hook <NUM> determines information relating to the hooked API call. This information includes, in various embodiments, one or more of an indicator of the function name <NUM> of the API call <NUM>, values of input parameters <NUM> passed from the potential malware application <NUM> to the API call <NUM>, a call stack of the potential malware application <NUM> when the API call <NUM> is made, code of the potential malware application <NUM> in a vicinity or adjacent to the API call <NUM>, properties <NUM> of the potential malware application (e.g., a name of the potential malware application, size of the potential malware application, etc.) or other information to a machine learning model <NUM>. With respect to embodiments that provide code adjacent to the API call, some embodiments provide a predefined number of bytes of code prior to the API call in an image (file) of the potential malware application <NUM>, and a second predefined number of bytes of code after the API call in the image (file) of the potential malware application <NUM>. Some embodiments provide a predefined number of instructions of the potential malware application (assembly/machine instructions or intermediate code instructions, or source instructions) prior to and subsequent to the API call.

The model <NUM> includes data, defining relationships between previous suggested modifications and resulting success or unsuccessful execution of a potential malware application, as described below, and an algorithm to determine a modification based on inputs provided to the model <NUM>, and the data defining the relationships. The machine learning model <NUM> is trained to determine a suggested modification <NUM> to the potential malware application <NUM> based on the information provided by the API hook <NUM>. In some embodiments, the ML model implements a reinforcement learning algorithm. The machine learning model <NUM> is trained, in some embodiments, to generate suggested modifications that maximize a probability of the potential malware application <NUM> completing successfully. Successful completion has a variety of definitions depending on embodiment, but place one or more conditions on execution of the potential malware application <NUM>. Some embodiments define successful completion as establishment of a network connection by the potential malware application with a remote device, creating and/or writing to a file, writing to a system registry (e.g., the Microsoft Windows registry), spawning one or more new processes or threads (different from the malware application itself). Some embodiments evaluate whether the potential malware application <NUM> has created or written to any files as one criterion when evaluating whether the malware application has successfully operated. Some embodiments count a number of API calls made by the potential malware application. Some embodiments base a determination of whether the potential malware application has successfully executed on whether the count exceeds a predetermined threshold.

The suggested modification <NUM> is provided to an execution modifier component <NUM>, which implements the modification <NUM> on the execution of the potential malware application <NUM>. In some embodiments, the suggested modification <NUM> indicates a modification to an API return value of the API call <NUM>. In this case, the execution modifier invokes a real API in the API library <NUM> corresponding to the API call <NUM>, but substitutes any return value from the API of the API library <NUM> for a different return value. In some embodiments, the different return value is selected from a list of common (e.g., N most frequently returned) return values from the hooked API. The selected return value is then used as the different return value. In some embodiments, the suggested modification is no modification. In this case, the execution modifier <NUM> acts simply as a proxy, and invokes the hooked API of the real AP library, and passing through any input parameters, output parameters, and return values in an unmodified form.

In some embodiments, the suggested modification is a modification to the execution of a portion of code included in the potential malware application <NUM> itself. For example, some embodiments indicate a modification to a condition that is subsequent to the API call <NUM>, such as condition <NUM>. Modification of the condition <NUM> is accomplished using various means by various embodiments. Some embodiments modify instructions included in the potential malware application <NUM> that implement the condition <NUM> to perform an alternate operation. Other embodiments intercept operation of the condition <NUM> and modify register or memory values necessary to alter a result of the condition. As part of the modification process, the API call is allowed to return control <NUM> to the executing program by the execution modifier <NUM>.

<FIG> shows another embodiment of a sandbox architecture <NUM> implemented by one or more of the disclosed embodiments. The sandbox architecture <NUM> includes a potential malware application <NUM>. The potential malware application <NUM> executes a stream of instructions <NUM>, each of which is provided to a debugger <NUM>. In some embodiments, the potential malware application <NUM> is running "under" control of the debugger <NUM>, which is able to "single step" through each of the instructions executed by the potential malware application <NUM>. Some other embodiments do not use a debugger as illustrated in <FIG> to intercept the stream of instructions <NUM>. Instead, for example, some implementations utilize a hardware emulator, in a similar manner as that described above with respect to <FIG>, to intercept the stream of instructions <NUM>.

The stream of instructions is provided by the debugger <NUM> to an instruction stream processor <NUM>. The instruction stream processor <NUM> provides the stream of instructions <NUM> to a machine learning model <NUM>. The machine learning model <NUM> implements, in at least some embodiments, a reinforcement learning algorithm. The model <NUM> includes data, defining relationships between previously suggested modifications and the resulting successful or unsuccessful execution of a program to which the suggest modifications were applied, and an algorithm to determine a modification based on inputs provided to the model <NUM>, and the data defining the relationships. Via sequential application of the machine learning model <NUM>, the model <NUM> develops the data defining relationships between the suggested modifications and any result achieved from them, and is able to use this data to provide a suggested modification <NUM> that maximizes successful execution of the potential malware application <NUM>.

As discussed above, some embodiments define successful execution as that execution which results in the potential malware application <NUM> establishing a network connection with a remote device and/or spawning at least one additional process or thread (different from the malware process/thread itself). One or more of file I/O activity, creation and/or writing of a system registry entry or entries, and/or a number of API calls made by the potential malware application can also be used by various embodiments to determine if the malware application has successfully executed.

The suggested modification <NUM> indicates a variety of modifications in various embodiments. For example, the suggested modification <NUM> indicates, in some cases, modifications of register or memory values at particular places within the execution of the stream of instructions <NUM>. The suggested modification <NUM>, indicates, in some cases, modification of one or more of the instructions included in the stream of instructions <NUM>. For example, one or more instructions are modified to change operation of a branch condition, branch, or other conditional logic within the potential malware application <NUM>.

The suggested modification <NUM> is provided to an execution modifier <NUM>, which performs <NUM> the indicated modification.

<FIG> shows an example machine learning system 500according to some examples of the present disclosure. Machine learning system <NUM> utilizes a prediction module <NUM>.

In the prediction module <NUM>, current information <NUM> is input to the feature determination module 550b. The current information <NUM> represents characteristics of a potential malware application being analyzed by the disclosed embodiments (e.g., potential malware application <NUM> or potential malware application <NUM>). Feature determination module 550b determines, from the current information <NUM>, a set of features <NUM>. In some embodiments, the set of features includes an instruction stream of the potential malware application, an API call of the potential malware application, input parameters to the API call, a call stack of the potential malware application at the API call, properties of the potential malware application, or other characteristics of the potential malware application. The set of features <NUM> is provided to the machine learning model <NUM> to generate a suggested modification <NUM>. An indication of whether the potential malware application operated successfully is provided back to the model <NUM> as model input <NUM>.

<FIG> is a flowchart of a method for modifying execution of an application. In some embodiments, one or more of the functions discussed below with respect to <FIG> and method <NUM> are performed by hardware processing circuitry. In some embodiments, instructions (e.g. <NUM> discussed below) stored in a memory (e.g., memory <NUM> and/or <NUM> discussed below) configure a hardware processor (e.g., processor <NUM> discussed below) to perform one or more of the functions discussed below with respect to <FIG> and method <NUM>.

After start operation <NUM>, method <NUM> moves to operation <NUM>, where a sequence of instructions of an executing application are intercepted. As discussed above with respect to <FIG>, in some embodiments, a stream of instructions (e.g. stream of instructions <NUM>) are intercepted using a debugger <NUM>. Alternate embodiments utilize other technologies to intercept the instructions, such as a hardware emulator. In some embodiments, the executing application is a "native" application, such as first potential malware application <NUM>, which executes an instruction set native to the hardware upon which it is operating. In some embodiments, the executing application is an interpreted application or script-based application, such as the second potential malware application <NUM>, also discussed above with respect to <FIG>. In this case, the sequence of instructions are not "native" to the hardware, but are intermediate instructions such as p-code, or even source code instructions in some embodiments.

In some embodiments the sequence of instructions include a function call instruction or, in other words, an API function call.

In operation <NUM>, the sequence of instructions is provided to a machine learning model. As discussed above, some embodiments use a machine learning model configured to implement a reinforcement learning approach. The machine learning model is trained to maximize a likelihood of a successful execution of the executing application. Successful execution has various definitions in various embodiments. Some embodiments define successful execution as the executing application performing at least one of establishing a network connection with a remote computer, spawning a new process or thread, or creating or writing to a file.

In operation <NUM>, a suggested modification is received from the machine learning model. The suggested modification is in response to the provided sequence of instructions of operation <NUM>. As discussed above, several different modifications are suggested by the machine learning model in a variety of circumstances. In some cases, no modification is suggested. In some cases, the machine learning model indicates that a register value or data value of the executing application be modified at a particular execution point of the executing application. In some cases, the modification indicates that a return value or input parameter of an API function be modified. In some embodiments, the suggested modification is to change operation of a conditional statement in the executing application itself, for example, by modifying instructions themselves within the executing application, or by modifying registers and/or data values so as to change the conditional operation of the executing application.

In operation <NUM>, the indicated modification is performed. Some embodiments of method <NUM> are performed iteratively which a single application is being executed. As explained above, the disclosed embodiments provide for increased successful operation of potential malware in a sandbox environment. By enabling the potential malware to execute successfully, additional information regarding the malware is obtained. This information is used, in some embodiments, to configure run time malware detection software, to identify vulnerabilities in existing software, or to further improve a sandbox testing environment. After operation <NUM>, method <NUM> moves to end operation <NUM>.

Some embodiments of method <NUM> maintain a log or record of modifications to made to a potential malware application. A sequence of modifications are thus provided in this log in at least some embodiments. Some embodiments generate one or more reports providing or displaying at least a portion of data in the log after a particular potential malware application is executed in a sandbox environment. This report would thus identify modifications necessary to cause the potential malware application to successfully execute.

This sequence of modifications is used, in some embodiments, to make modifications to the sandbox environment such that the sandbox environment is less detectable to other potential malware applications. For example, if modification of an API call value to a particular value is frequently successful at facilitating successful execution of a potential malware application, a configuration of the sandbox is modified, in some cases, such that the API call value returns the particular value without any intervention by the disclosed embodiments. As one example, if an API call is modified to return a number of processors included in a sandbox computer as a value of eight (<NUM>) provides for successful execution of potential malware applications, a configuration file of an operating system is modified such that the API returns the value of eight (<NUM>).

Some embodiments provide an algorithm that analyzes the log of modifications and automatically (e.g. without human intervention) generates modifications to a sandbox environment to avoid the need for such modifications. For example, the algorithm, in some embodiments, identifies modified return values of one or more APIs and automatically changes a sandbox configuration such that the sandbox configuration is consistent with the modified return values. This will reduce the number of modifications required during dynamic execution of additional potential malware applications.

<FIG> is a flowchart of a method for modifying execution of an application. In some embodiments, one or more of the functions discussed below with respect to <FIG> and method <NUM> are performed by hardware processing circuitry. In some embodiments, instructions (e.g., <NUM> discussed below) stored in a memory (e.g., memory <NUM> and/or <NUM> discussed below) configure a hardware processor (e.g., processor <NUM> discussed below) to perform one or more of the functions discussed below with respect to <FIG> and method <NUM>. In some embodiments, method <NUM> is included in method <NUM>, discussed above with respect to <FIG>. For example, some embodiments of method <NUM> overlap with at least portions of embodiments of method <NUM>.

After start operation <NUM>, method <NUM> moves to operation <NUM>, which intercepts a function call of an executing application. In some embodiments, the executing application is executing within a sandbox environment. The sandbox environment is a virtualized computing environment in at least some embodiments. The application is suspected, in at least some embodiments, of being a malware application. Thus, the application has at least some potential for including malware (e.g. nefarious) features, but is not necessarily malware. In some cases, a potential malware application is a perfectly benign and harmless application. However, some other potential malware is actually malware and is configured to destroy computing resources, or perform other nefarious activities such as unauthorized copying or transfer of data, ransomware, or other functions understood to be included in the term malware.

As discussed above with respect to <FIG>, some embodiments intercept API calls made by an executing application. In some embodiments, the executing application is a "native" application, such as first potential malware application <NUM>, which executes an instruction set native the hardware upon which it is operating. In some embodiments, the executing application is an interpreted application or script-based application, such as the second potential malware application <NUM>, also discussed above with respect to <FIG>. The function called is included in an API library in some embodiments (e.g., API library <NUM>). Some embodiments of operation <NUM> capture or intercept a sequence of function calls. In those embodiments, each of the functions discussed below with respect to operation <NUM> and <NUM> are repeated for each captured function call.

In operation <NUM>, input parameter values to the function call are determined. In some embodiments, the function call includes no input parameters. Some embodiments also obtain a call stack of the executing application. For example, some embodiments send a signal <NUM> to a Java process, which causes a stack trace to be generated to a standard output (stdout) device. Some embodiments use a utility called jstack from a command line to obtain the call stack of the executing application. While these solutions apply to Java based solutions, other solutions exist for other technologies, such as Microsoft Windows. For example, Microsoft provides Debugging Tools for Windows. Some embodiments of operation <NUM> also capture a portion of the executing application. For example, a predetermined number of bytes or number of instructions are captured before and/or after the API call within the executing application.

In operation <NUM>, the information captured or collected in operation <NUM> is provided to a machine learning model. As discussed above, in some embodiments, the machine learning model is configured to use a reinforcement learning algorithm. The machine learning algorithm is trained, in some embodiments, to maximize a probability that the executing application executes successfully, with successful execution defined by, in some embodiments, one or more of the executing application opening a network connection to a remote device, spawning a new process or thread, or writing data or otherwise creating a file.

In operation <NUM>, a suggested modification is obtained from the machine learning model. The suggested modification is in response to, or otherwise based on, the information provided to the machine learning model in operation <NUM>, such as the provided call stack, and input parameter values of the API call. In some embodiments, the suggested modification is based on an indicated sequence of function calls, previously provided to the machine learning model by operation <NUM> as discussed above.

As discussed above, several different modifications are suggested by the machine learning model in a variety of circumstances. In some cases, no modification is suggested. In some cases, the machine learning model indicates that a register value or data value of the executing application be modified at a particular execution point of the executing application. In some cases, the modification indicates that a return value or input parameter of an API function be modified. In some embodiments, the suggest modification is to change operation of a conditional statement in the executing application itself, for example, by modifying instructions themselves within the executing application, or by modifying registers and/or data values so as to change the conditional operation of the executing application. For example, a conditional branch is modified, in some embodiments, to take a first path instead of a second path.

In operation <NUM>, the indicated modification is performed. Some embodiments of method <NUM> are performed iteratively while a single application is being executed. As explained above, the disclosed embodiments provide for increased successful operation of potential malware in a sandbox environment. By enabling the potential malware to execute successfully, additional information regarding the malware is obtained.

This information is used, in some embodiments, to configure run time malware detection software, to identify vulnerabilities in existing software, or to further improve a sandbox testing environment. For example, if, via the disclosed embodiments, a malware application is successfully executed and demonstrates one or more malware type effects (negative effects), a signature of the malware application (e.g. hash or checksum) is added to a security database in some embodiments. The security database is then compared against signatures of executing applications to detect instances of the malware application. In some embodiments, the security database is downloaded to client devices, such as client devices managed by an organization. A virus scanning filter executes on the client devices and detects an executing application. The virus scanning filter then computes a signature of the detected executing application, and compares it to signatures in the security database. The virus scanning filter then detects a match between the dynamically determined signature of the executing application and compares it to one or more signatures included in the security database. If a match is detected, the virus scanning filter causes execution of the application to be halted, aborted, or otherwise mitigated.

Thus, the disclosed embodiments create several technical effects, and represent several technical solutions. A first order technical effect is that a malware application that would otherwise detect a sandbox environment and inhibit performance of one or more of its features as a result, instead does not detect the sandbox environment and performs these one or more features. By performance of the features (such as establishing a network connection, deleting files, spawning additional process, etc), the application's behavior can be more accurately analyzed and profiled. This profile can then be used to identify other instances of the malware application in non-sandbox environments (e.g. operational environments).

An additional technical solution and/or effect provided by the disclosed embodiments is increased accuracy and/or completeness of signature information for known malware applications. By facilitating more complete analysis of malware applications in a sandbox environment, the disclosed embodiments provide enhanced insight into execution of malware applications than would otherwise be possible. This increased analysis ability leads to additional and more accurate signature information being included in security databases, resulting in an overall reduction in the instance of successful malware exploits.

After operation <NUM>, method <NUM> moves to end operation <NUM>.

<FIG> illustrates a block diagram of an example machine <NUM> upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform. In altemative embodiments, the machine <NUM> may operate as a standalone device or are connected (e.g., networked) to other machines. The machine <NUM> is a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile telephone, a smart phone, a web appliance, a network router, switch or bridge, a server computer, a database, conference room equipment, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. In various embodiments, machine <NUM> may perform one or more of the processes described above with respect to <FIG> above. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), and other computer cluster configurations.

Examples, as described herein, may include, or may operate on, logic or a number of components, modules, or mechanisms (all referred to hereinafter as "modules"). Modules are tangible entities (e.g., hardware) capable of performing specified operations and is configured or arranged in a certain manner. In an example, circuits are arranged (e.g., internally or with respect to external entities such as other circuits) in a specified manner as a module. In an example, the whole or part of one or more computer systems (e.g., a standalone, client or server computer system) or one or more hardware processors are configured by firmware or software (e.g., instructions, an application portion, or an application) as a module that operates to perform specified operations. In an example, the software may reside on a non-transitory computer readable storage medium or other machine readable medium.

For example, where the modules comprise a general-purpose hardware processor configured using software, the general-purpose hardware processor is configured as respective different modules at different times.

Machine (e.g., computer system) <NUM> may include a hardware processor <NUM> (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory <NUM> and a static memory <NUM>, some or all of which may communicate with each other via an interlink <NUM> (e.g., bus). In an example, the display unit <NUM>, input device <NUM> and UI navigation device <NUM> are a touch screen display.

The term "machine readable medium" may include any medium that is capable of storing, encoding, or carrying instructions for execution by the machine <NUM> and that cause the machine <NUM> to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. Non-limiting machine readable medium examples may include solid-state memories, and optical and magnetic media. Specific examples of machine readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; Random Access Memory (RAM); Solid State Drives (SSD); and CD-ROM and DVD-ROM disks. In some examples, machine readable media may include non-transitory machine readable media. In some examples, machine readable media may include machine readable media that is not a transitory propagating signal.

The instructions <NUM> may further be transmitted or received over a communications network <NUM> using a transmission medium via the network interface device <NUM>. The machine <NUM> may communicate with one or more other machines utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), mobile telephone networks (e.g., cellular networks), Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) <NUM> family of standards known as Wi-Fi®, IEEE <NUM> family of standards known as WiMax®), IEEE <NUM>. <NUM> family of standards, a Long Term Evolution (LTE) family of standards, a Universal Mobile Telecommunications System (UMTS) family of standards, peer-to-peer (P2P) networks, among others. In an example, the network interface device <NUM> may include one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas to connect to the communications network <NUM>. In an example, the network interface device <NUM> may include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. In some examples, the network interface device <NUM> may wirelessly communicate using Multiple User MIMO techniques.

Modules are tangible entities (e.g., hardware) capable of performing specified operations and are configured or arranged in a certain manner. In an example, circuits are arranged (e.g., internally or with respect to external entities such as other circuits) in a specified manner as a module. In an example, the whole or part of one or more computer systems (e.g., a standalone, client, or server computer system) or one or more hardware processors are configured by firmware or software (e.g., instructions, an application portion, or an application) as a module that operates to perform specified operations. In an example, the software may reside on a machine-readable medium.

Example <NUM> is a method, comprising: intercepting a sequence of instructions of an executing application; providing the sequence of instructions to a machine learning model; receiving, from the machine learning model based on the provided sequence of instructions, an indication of a modification to the executing application; and performing the indicated modification.

In Example <NUM>, the subject matter of Example <NUM> optionally includes wherein the sequence of instructions are machine instructions or bytecode instructions.

In Example <NUM>, the subject matter of any one or more of Examples <NUM>-<NUM> optionally include wherein the sequence of instructions include a function call instruction, the method further comprising: determining a call stack of the executing application at the function call instruction; determining parameters of the function call; and providing the call stack and the parameters to the machine learning model, wherein the indication of the modification is further based on the provided call stack and parameters.

In Example <NUM>, the subject matter of Example <NUM> optionally includes determining second parameters of a second function call, and providing the second parameters of the second function call to the machine learning model, wherein the indication of the modification is further based on the provided second parameters.

In Example <NUM>, the subject matter of any one or more of Examples <NUM>-<NUM> optionally include wherein the modification is a modification of an output of the function call.

In Example <NUM>, the subject matter of Example <NUM> optionally includes selecting, from a list of return values of the function call, a return value, and setting the output to the selected return value.

In Example <NUM>, the subject matter of any one or more of Examples <NUM>-<NUM> optionally include identifying a control-flow instruction of the executing application, wherein the modification is a modification of the control-flow instruction.

In Example <NUM>, the subject matter of any one or more of Examples <NUM>-<NUM> optionally include capturing a portion of executable code of the executing application within a predefined proximity of the function call, and providing the portion to the machine learning model, wherein the indication of the modification is based on the provided portion.

In Example <NUM>, the subject matter of any one or more of Examples <NUM>-<NUM> optionally include capturing a sequence of function calls by the executing application, and providing data indicating the sequence of function calls to the machine learning model, wherein the indication of the modification is based on the indicated sequence of function calls.

In Example <NUM>, the subject matter of any one or more of Examples <NUM>-<NUM> optionally include wherein the machine learning model is configured to apply a reinforcement learning algorithm, the machine learning model trained to generate a modification that results in successful execution of the executing application.

In Example <NUM>, the subject matter of Example <NUM> optionally includes wherein successful execution is detected when the executing application creates a new process, creates a new file, creates a new registry entry, establishes a network connection, or the executing application invokes a number of API calls that exceeds a predefined threshold.

In Example <NUM>, the subject matter of any one or more of Examples <NUM>-<NUM> optionally include storing a record in a data store, the record indicating the modification to the executing application.

In Example <NUM>, the subject matter of Example <NUM> optionally includes identifying based on a plurality of records in the data store, a modification common to the plurality of records; and modifying a sandbox environment configuration, such that the configuration is consistent with the identified modification.

Example <NUM> is a system, comprising: hardware processing circuitry; one or more hardware memories storing instructions that when executed configure the hardware processing circuitry to perform operations, comprising: intercepting a sequence of instructions of an executing application; providing the sequence of instructions to a machine learning model; receiving, from the machine learning model based on the provided sequence of instructions, an indication of a modification to the executing application; and performing the indicated modification.

In Example <NUM>, the subject matter of any one or more of Examples <NUM>-<NUM> optionally include wherein the sequence of instructions include a function call instruction, the operations further comprising: determining a call stack of the executing application at the function call instruction; determining parameters of the function call; and providing the call stack and the parameters to the machine learning model, wherein the indication of the modification is further based on the provided call stack and parameters.

Example <NUM> is a non-transitory computer readable storage medium comprising instructions that when executed configure hardware processing circuitry to perform operations comprising: intercepting a sequence of instructions of an executing application; providing the sequence of instructions to a machine learning model; receiving, from the machine learning model based on the provided sequence of instructions, an indication of a modification to the executing application; and performing the indicated modification.

Claim 1:
A system (<NUM>), comprising:
hardware processing circuitry (<NUM>); and
one or more hardware memories (<NUM>) storing instructions (<NUM>) that when executed configure the hardware processing circuitry (<NUM>) to perform operations, comprising:
intercepting (<NUM>) a sequence of instructions (<NUM>) of an executing application;
providing (<NUM>) the sequence of instructions (<NUM>) to a machine learning model (<NUM>);
receiving (<NUM>), from the machine learning model (<NUM>) based on the provided sequence of instructions (<NUM>), an indication of a modification (<NUM>) to the executing application; and
performing (<NUM>) the indicated modification (<NUM>), wherein the machine learning model (<NUM>) is configured to apply a reinforcement learning algorithm, the machine learning model (<NUM>) trained to generate a modification (<NUM>) that results in successful execution of the executing application.