Source: https://patents.google.com/patent/US9223963B2/en
Timestamp: 2019-10-21 05:45:49
Document Index: 518487097

Matched Legal Cases: ['Application No. 8154997', 'Application No. 8154997', 'Application No. 10842508', 'Application No. 10', 'Application No. 2010800628082', 'Application No. 2012', 'Application No. 10', 'Application No. 10', 'Application No. 8154997', 'Application No. 10842508']

US9223963B2 - Systems and methods for behavioral sandboxing - Google Patents
US9223963B2
US9223963B2 US13/898,284 US201313898284A US9223963B2 US 9223963 B2 US9223963 B2 US 9223963B2 US 201313898284 A US201313898284 A US 201313898284A US 9223963 B2 US9223963 B2 US 9223963B2
US13/898,284
US20130254884A1 (en
2013-05-20 Application filed by McAfee LLC filed Critical McAfee LLC
2013-05-20 Priority to US13/898,284 priority patent/US9223963B2/en
2013-09-26 Publication of US20130254884A1 publication Critical patent/US20130254884A1/en
2015-12-29 Publication of US9223963B2 publication Critical patent/US9223963B2/en
238000004458 analytical methods Methods 0 abstract claims description 157
230000003542 behavioural Effects 0 abstract claims description title 196
This application is a continuation application of U.S. patent application Ser. No. 12/638,660, filed on Dec. 15, 2009, entitled “Systems and Methods for Behavioral Sandboxing”, which is hereby incorporated by reference in its entirety.
Executable Application—For the purposes of the following specification, an executable application can include any complied binary application (e.g., .executable application or dynamical link libraries in a Microsoft Windows® environment), browser plug-in applications, browser or application scripts (e.g., Javascript™ or Visual Basic®. script), operating system control scripts (e.g., .bat files in a Microsoft Windows® environment or C-shell scripts in a Unix environment), and run-time interpreted code or applets (e.g., Java™ applets, Visual Basic® Controls, .Net™ code). Executable applications can encompass multiple individual processing units, such as processes, threads, and fibers. Additionally, some large executable applications may utilize dynamic link libraries (DLLs), scripts, or plug-in components to extend or enhance basic functionality. Within the following specification an executable application may be referred to variously as a process, an application, an executable, or simply as software.
The sandbox execution environment 220 can be configured to be totally segregated from the rest of system 200 in terms of resources and memory space. The sandbox execution environment 220 can provide the executable application a tightly controlled set of resources to run in, such as limited disk and memory space. The sandbox execution environment 220 can also control both local and remote network access, depending on configuration. For example, the sandbox execution environment 220 may allow certain local network access to read and write to a local database, but disallow any Internet access in order to limit potentially malicious behavior. In an example, the sandbox execution environment can be implemented through emulation. Using emulation, the sandboxed process runs within an interpreted environment rather than natively against the regular execution platform. In another example, the sandbox execution environment is implemented through the use of system virtualization. Virtualization allows for interception of resource operations and redirection of communications between processes. Alternatively, virtualization can be used to replicate the entire operating environment the process would execute in to mirror the non-sandboxed environment.
The behavioral analysis module 230 can be configured to analyze characteristics of an executable application as well as actual past or present operations performed by an executable application to predict the likelihood of malicious behavior. The behavioral analysis module 230 can analyze information about an executable application such as source, installation process, last modification, signature, and cryptographic certificate validity, among other things. The behavioral analysis module 230 can also analyze past logged operations as well as presently monitored operations performed by the executable application, such as opening of resources, loading of plug-in components, accessing the Internet, and attempting to read or write to configuration files or databases, among other things. Refer to discussion below related to FIG. 3 for additional details on an example behavioral analysis system.
The system monitoring engine 240 is used to monitor operations performed within the standard execution environment 210 as well as within the sandbox execution environment 220. The system monitoring engine 240 can provide valuable information to the behavioral analysis module 230 for use in determining whether an executable application is attempting (or likely to attempt) potentially malicious behavior. The system monitoring engine 240 can be configured to intercept attempts to access any of the system resources, including disk I/O, network device I/O, memory access, and even processor utilization. In certain examples, the system monitoring engine 240 assists in creating a sandbox environment by routing all resource requests to or from the sandbox environment. The system monitoring engine 240 can be configured to intercept resource access requests and alter the source or destination parameters before allowing the operation to continue. Examples of resources can include files, configuration databases (the Registry in Windows® as a prime example), memory (normally separated by process by the operating system, but could be manually redirected to provide finer granularity of separation), to name just a few.
For example, detection of high-risk activity such as the downloading of browser plug-ins from an unvetted website could be of interest to a browser script scanner. The script scanner may subject these plug-ins to a higher level of scanning. Additionally, a host-intrusion prevention product may also be interested in the download and subject the scripts to quarantining Finally, the determination that the web site the scripts were downloaded from was unvetted could have been announced via the publishing of a detection by a web site analysis tool.
The host environment may load security software including the behavioral detection framework 304 and provide at least basic capabilities. Some of this basic functionality may include file systems monitoring, network monitoring, and monitoring updating of a configuration database (e.g. monitoring registry operations, etc.).
The behavior detection framework 304 may include various contexts. For example, in one example, the context may include a local context where user mode operations are specific to a particular process. In this case, there may be one local context per process of interest.
Moving back to operation 408, if the behavioral analysis module 230 determines that the executable application can be executed within a standard execution environment 210, then the method 400 continues at 410 with the system 200 running the executable application within the standard execution environment 210. At 412, the method 400 continues with the behavioral analysis module 230 performing behavioral analysis on the executable application as it is running within the standard execution environment 210. At operation 412, the system 200 dynamically monitors the executable application for any signs to malicious (or suspicious) behaviors. The behavioral analysis module 230 watches for events that increase the exposure of the system 200 to malicious operations. Events watched for by the behavioral analysis module 230 can include: opening a resource files or databases, loading plug-ins (e.g., scripts and some executables such as libraries or DLLs), manipulation by other processes, manipulation of other processes, accessing the network, accessing the Internet, or accessing sites on the Internet outside a list of known good sites (e.g., blacklisted sites), to name just a few.
At 414, the method 400 continues with the behavioral analysis module 230 determining if the executable environment should be changed based on the monitored behavioral characteristics of the executable application. The dynamic behavioral analysis performed at operation 412 allows the system 200 to alter the executable environment of an executable application (or a portion of the executable application) while it is running. This enables the system 200 to protect itself from applications that appear to be safe but turn out to be malicious. For example, the behavioral analysis module 230 can determine that a certain application is from a known and trusted source, allowing the application to be launched within the standard execution environment 210. However, during operation the user may instruct the application to download a new plug-in from an unknown source. The behavioral analysis module 230 can be configured to intercept this event and instruct the system 200 to redirect the new plug-in into the sandbox execution environment 220 for execution and increased monitoring. In another example, the behavioral analysis module 230 may determine that an application can be executed within the standard execution environment 210 based on an analysis of historical behavior of the application. However, during execution the application may attempt to update the client system's 110 configuration database in violation of a security policy governing acceptable behaviors. In this example, the behavioral analysis module 230 can instruct the system 200 to move the executing application into the sandbox execution environment 220 to protect the system 200 from any further potential harm.
At 416, the method 400 checks with the behavior analysis module 230 to determine whether the executable application should be moved into the sandbox execution environment 220. If the executable application is to be moved, then the method 400 continues at 420 with the system 200 continuing execution of the executable application within the sandbox execution environment 220. If the executable application is not going to be moved into a sandbox environment, then the method 400 loops back to operation 408 and continues until the executable application terminates.
The method 500 includes operations 504 and 512 where the system monitoring engine 240 monitors a configurable range of operations performed by an executable application. At operation 504, the method 500 monitors the executable application executing within the standard execution environment 210. In an example, monitoring information gathered by the system monitoring engine 240 at 504 can be used by the behavioral analysis module 230 to determine whether the executable application should be moved into the sandbox environment 220. As discussed above, an executable application may need to be moved into a sandbox environment due to unexpected potentially malicious behavior detected after the application is initiated. At operation 512, the method 500 monitors the executable application executing within the sandbox execution environment 220. In an example, monitoring information gathered by the system monitoring engine 240 at 512 can be used by the behavioral analysis module 230 to determine whether the executable application can be moved out of the sandbox environment. Monitoring the execution of the executable application within the sandbox environment 220 can also be used to determine whether information generated by the application can be communicated outside the sandbox environment 220. Additionally, the monitored information can be used to dynamically increase or decrease the scope of restrictions applied to the sandbox environment 220.
The method 800 continues at 808 if the executable application is allowed to execution within a standard execution environment, such as standard execution environment 210. In an example at 808, the method 800 starts the executable application in the standard executable environment 210. At 810, the method 800 continues with the behavioral analysis module 230 performing behavioral analysis on the running executable application. In certain examples, the system monitoring engine 240 can be used to monitor operations performed by the executable application. The monitored operations can be analyzed for potentially malicious behavior by the behavioral analysis module 230. At 812, the behavioral analysis module 230 dynamically determines whether the running executable application should be moved into a sandbox environment. The method 800 continues at operation 814 if the executable application is not moved into a sandbox environment at 812. At 814, the method 800 continues with the system 200 determining if the executable application has terminated. The method 800 loops back to operation 808 if the executable application continues to run.
1. A method for dynamically determining an execution environment in a system having a plurality of execution environments including a first execution environment and a second execution environment, the second execution environment being a protected execution environment, the method comprising:
performing behavioral analysis on an executable application prior to each execution of the executable application, comprising:
requesting behavioral information regarding prior performance of suspicious or out of policy activities of the executable application from a remote security database prior to each execution of the executable application; and
evaluating the behavioral information obtained responsive to the request;
determining an initial execution environment based on the behavioral analysis prior to each execution of the executable application, comprising:
determining a first portion of a single execution of the executable application for execution in the first execution environment and a second portion of the single execution for execution in the second execution environment; and
determining the initial execution environment as one of the first execution environment and the second execution environment;
loading the executable application for execution in the initial execution environment;
collecting behavioral characteristics of the executable application as it is executed within the initial execution environment; and
determining from the collected behavioral characteristics whether execution of the executable application should be moved from the initial execution environment to a secondary execution environment while the executable application is still executing in the initial execution environment,
wherein only one of the initial execution environment and the secondary execution environment is a protected execution environment.
2. The method of claim 1, wherein performing behavioral analysis prior to execution further comprises evaluating one or more characteristics associated with the executable application that indicate an origin of the executable application.
trust level of a process that requested the executable application; and
signed certificate from a trusted source.
8. The method of claim 1, wherein determining from the collected behavioral characteristics comprises detecting potentially malicious activities including:
10. The method of claim 1, wherein the first portion a single execution of the executable application includes execution of:
a computer comprising one or more processors wherein the one or more processors are configured to:
perform behavioral analysis on an executable application prior to each execution of the executable application, comprising:
determine an initial execution environment based on the behavioral analysis prior to each execution of the executable application, comprising:
determining a first portion of a single execution of the executable application for execution in a first execution environment and a second portion of the single execution for execution in a second execution environment; and
load the executable application for execution in the initial execution environment;
collect behavioral characteristics of the executable application as it is executed in the initial execution environment; and
determine from the collected behavioral characteristics whether execution of the executable application should be moved from the initial execution environment to a secondary execution environment while the executable application is still executing in the initial execution environment,
12. The system of claim 11, wherein the one or more processors configured to perform behavioral analysis are further configured to evaluate a characteristic associated with the executable application, wherein the characteristic indicates whether the provider of the executable application can be trusted.
13. The system of claim 12, wherein the one or more processors configured to perform behavioral analysis are further configured to evaluate a plurality of characteristics including two or more of the following:
14. The system of claim 11, wherein the one or more processors configured to perform behavioral analysis are further configured to analyze, prior to execution of the executable application, a resource the executable application will utilize if executed.
15. The system of claim 11, wherein the one or more processors configured to perform behavioral analysis are further configured to determine a scope of restrictions a protected execution environment will use to control the execution of the executable application.
16. The system of claim 15, wherein the one or more processors configured to perform behavioral analysis are further configured to determine what types of activity should be contained within the protected execution environment.
17. The system of claim 15, wherein the one or more processors configured to perform behavioral analysis are further configured to apply a security policy when determining the scope of restrictions for use by the protected execution environment.
18. The system of claim 15, wherein the one or more processors configured to perform behavioral analysis are further configured to determine a disposition for operations performed within the protected execution environment by the executable application.
19. The system of claim 11, wherein the one or more processors configured to perform behavioral analysis are further configured to detect potentially malicious activities, including:
20. The system of claim 11, wherein the one or more processors configured to perform behavioral analysis are further configured to apply a security policy when evaluating a plurality of behavioral characteristics of the executable application to determine whether the initial execution environment should be a protected execution environment.
21. The system of claim 11, wherein the portion of the single execution of the executable application includes execution of:
22. A non-transitory computer-readable storage medium comprising instructions stored thereon that when executed cause one or more processors to:
determine an initial execution environment based on the behavioral analysis prior to each execution of the executable application, comprising instructions that when executed cause the one or more processors to:
determine a first portion of a single execution of the executable application for execution in a first execution environment and a second portion of the single execution for execution in a second execution environment; and
determine the initial execution environment as one of the first execution environment and the second execution environment;
23. The non-transitory computer-readable storage medium of claim 22, further comprising instructions stored thereon to cause one or more processors to:
monitor second behavioral characteristics of the executable application during execution within the secondary execution environment after the executable application is moved from the initial execution environment;
perform behavioral analysis on the monitored second behavioral characteristics to determine whether the executable application should be moved from the secondary execution environment back to the initial execution environment; and
move, based on the behavioral analysis of the monitored second behavioral characteristics, the executable application from the secondary execution environment to the initial execution environment prior to termination of the execution in the secondary execution environment.
US13/898,284 2009-12-15 2013-05-20 Systems and methods for behavioral sandboxing Active US9223963B2 (en)
US13/898,284 US9223963B2 (en) 2009-12-15 2013-05-20 Systems and methods for behavioral sandboxing
US12/638,660 Continuation US8479286B2 (en) 2009-12-15 2009-12-15 Systems and methods for behavioral sandboxing
US20130254884A1 US20130254884A1 (en) 2013-09-26
US9223963B2 true US9223963B2 (en) 2015-12-29
US12/638,660 Active 2030-12-19 US8479286B2 (en) 2009-12-15 2009-12-15 Systems and methods for behavioral sandboxing
US13/898,284 Active US9223963B2 (en) 2009-12-15 2013-05-20 Systems and methods for behavioral sandboxing
WO2014058409A1 (en) * 2012-10-08 2014-04-17 Hewlett-Packard Development Company, L.P. Robust hardware fault management system, method and framework for enterprise devices
FR2997529B1 (en) * 2012-10-29 2015-12-25 Pradeo Security Systems Method and system for verifying security of an application for use on a user apparatus
DE102014203164A1 (en) * 2014-02-21 2015-08-27 Siemens Aktiengesellschaft Analysis system and method for dynamically configuring an analysis system
JPWO2016121348A1 (en) * 2015-01-29 2017-11-24 日本電気株式会社 Anti-malware device, anti-malware system, anti-malware method, and anti-malware program
JP6333763B2 (en) * 2015-03-31 2018-05-30 株式会社日立アドバンストシステムズ Malware analysis apparatus and malware analysis method
WO2016203759A1 (en) * 2015-06-16 2016-12-22 日本電気株式会社 Analysis system, analysis method, analysis device, and recording medium in which computer program is stored
JP2018524720A (en) * 2015-06-27 2018-08-30 マカフィー， エルエルシー Malware mitigation
CN106055975A (en) * 2016-05-16 2016-10-26 杭州华三通信技术有限公司 Document detection method and sandbox
RU2665911C2 (en) 2017-02-08 2018-09-04 Акционерное общество "Лаборатория Касперского" System and method of file analysis for maliciousness in virtual machine
CA3054842A1 (en) * 2017-03-01 2018-09-07 Cujo LLC Detecting malicious behavior within local networks
KR102016226B1 (en) * 2017-11-22 2019-08-29 숭실대학교산학협력단 Method for adaptive dynamic analysing, adaptive dynamic analysing platform, device equipped with the same
EP3522057A1 (en) * 2018-02-06 2019-08-07 AO Kaspersky Lab System and method of detecting hidden behavior of a browser extension
EP1613010A2 (en) 2004-06-29 2006-01-04 Microsoft Corporation Incremental anti-spam lookup and update service
JP2007122745A (en) 2006-12-20 2007-05-17 Trend Micro Inc Assurance of security on transmission path to program provided to communication terminal via network
US20090216768A1 (en) 2008-02-22 2009-08-27 Microsoft Corporation Database sandbox
CN1675623A (en) 2002-06-07 2005-09-28 英特尔公司 System and method for protection against untrusted system management code by redirecting a system management interrupt and creating a virtual machine container
JP2008500653A (en) 2004-06-29 2008-01-10 インテル・コーポレーション Computer security improvement method according to the sand box method
WO2006099282A2 (en) 2005-03-14 2006-09-21 Webroot Software, Inc. Method and system for analyzing data for potential malware
Communication Pursuant to Article 93 (3) EPC Received for European Application No. 8154997.4, mailed on May 20, 2009, 1 page.
European Search Report received for European Patent Application No. 8154997.4, mailed on Aug. 27, 2008, 9 pages.
Extended Search Report received for European Patent Application No. 10842508.3, mailed on Nov. 21, 2013, 7 Pages.
Final Office Action Received for U.S. Appl. No. 11/738,703, mailed on Sep. 23, 2010, 17 pages.
International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2010/060340, mailed on Jun. 19, 2012, 5 pages.
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2010/060340, mailed on Aug. 25, 2011, 3 pages.
Notice of Allowance received for Korean Patent Application No. 10-2012-7018451, mailed on Sep. 1, 2014, 3 pages of Notice of Allowance Including 1 pages of English Translation.
Notice of Allowance received for U.S. Appl. No. 12/638,660, mailed on Feb. 27, 2013, 5 Pages.
Office Action received for Chinese Patent Application No. 2010800628082, mailed on Jul. 24, 2014, 7 pages of English Translation only.
Office Action received for Japanese Patent Application No. 2012-544720, mailed on Jan. 7, 2014, 9 pages of Office Action including 5 pages of English Translation.
Office Action Received for Korea Patent Application No. 10-2012-7018451, mailed on Mar. 13, 2014, 14 pages of Office action including 8 pages of English translation.
Office Action received for Korean Patent Application No. 10-2012-7018451, mailed on Sep. 23, 2013, 15 Pages of Office Action including 8 Pages of English Translation.
Office Action Received for U.S. Appl. No. 11/738,703, mailed on Apr. 15, 2010, 19 pages.
Office Action received for U.S. Appl. No. 12/638,660, mailed on Apr. 5, 2012, 9 Pages.
Office Action received for U.S. Appl. No. 12/638,660, mailed on Sep. 18, 2012, 13 Pages.
Summons to Attend Oral Proceedings Received for European Application No. 8154997.4, mailed on May 31, 2010, 4 pages.
Supplementary Search Report received for European Patent Application No. 10842508.3, mailed on Dec. 10, 2013, 1 Page.
EP2513805B1 (en) 2019-04-17