Patent ID: 12238124

DETAILED DESCRIPTION OF THE DISCLOSURE

Embodiments of the invention provide systems and methods for detecting suspicious malware by analyzing data of transfer protocols from a host within an enterprise. Malware as referenced herein includes malware downloads, file records, file transfers, emails, attachments etc. Embodiments of the invention also include devices such as user interfaces for alerting network analysts of suspicious incidents, based on the systems and methods employed herein. The embodiments herein extract information from internet transfer protocols including non-encrypted transfer protocols such as HTTP, FTP, SMB, SMTP, etc. The HTTP protocol is utilized herein as an exemplary internet transfer protocol. However, the systems and methods of the embodiments of the invention herein are not limited to analysis of only HTTP protocol data.

There are many benefits and advantages of the embodiments of the present invention. For example, the embodiments herein perform malware detection in a scalable and inexpensive way, which allows for implementations at a very large scale. Therefore, the embodiments herein are highly beneficial for networks and enterprises with high volumes of traffic, such as HTTP traffic. In addition, the embodiments of the methods of the present invention utilize an outlier detection model that is based on historical behaviors. As such, the embodiments herein “learn” common protocol behaviors that are specific to the network being analyzed. In doing so, the systems and methods of the present invention improve the ability to identify suspicious outliers or potential malware on an iterative basis over time. Additional benefits and advantages of the benefits herein are described below and illustrated with respect to the figures.

System Architecture for Detecting Malware

FIG.1illustrates a high-level architecture of an exemplary system10for monitoring enterprise network logs of HTTP network traffic so that suspicious malware by enterprise users can be detected and analyzed, according to an embodiment of the present invention. In this embodiment, malware downloads are being detected and analyzed. System10includes a plurality of layers100-103within a network or enterprise network. The internet103is connected to and in communication with the router/firewall layer100. The router/firewall layer100is connected to and in communication with the core switch layer101. The core switch layer101is connected to and in communication with the access switch layer102. The access switch layer102contains one or more enterprise segments110,111having a plurality of connected devices110a-n,111a-n(e.g., laptops, PCs, printers, tablets mobile phones etc.). These devices110a-n,111a-nare operated by network users, and represent potential targets for malware downloads from the internet103. A plurality of network traffic sensors104are included in each segment110,111within the access switch layer102.

The sensors104serve the purpose of mirroring network traffic or data sent to corresponding access switches105, parsing out the metadata (e.g., HTTP protocol data), and saving the data to a database106. The content of the parsed data can vary depending on application specifics and the desires of network analysts. In an embodiment, the data includes at least Path values from HTTP headers. The Path field of the HTTP header represents the string of characters that follows the Host in a URL. An example of an HTTP Path is “/document/d/1VWC/my_song.mp3”. The HTTP Path is utilized to determine which records represent file downloads. The data could also contain other fields from HTTP headers, like Host, Content-Length, User-Agent, etc. An example of an HTTP Host is “www.google.com.” For HTTP downloads, the Content-Length in the header of the HTTP response typically represents the size of the downloaded file. For HTTP downloads, the User-Agent in the header of the HTTP request typically contains some information about the platform requesting the download (e.g., “Mozilla/5.0”). The sensors104can consist of one or more hardware components. Sensors104may also include or be comprised entirely of software modules that run on the individual devices110a-n,111a-nwithin the access switch layer102.

InFIG.1, sensors104are located within each enterprise segment110,111of the access switch layer102. This enables each of the sensors104within the enterprise segments110,111to have complete coverage of its respective internal enterprise network traffic. An advantage of this configuration is that having dedicated sensors104provides coverage that results in increased detection capabilities. Sensors104write the data (e.g., transfer protocol metadata) to the database106. The database106may be either internal or external to the enterprise segment110,111. After the data is provided to the database106, it is loaded by the computation engine107. The computation engine107may be a distributed computation engine that has multiple processors (not shown), capable of running a cluster computing framework (e.g., Apache Spark) that allows data to be distributed to multiple processors and processed in parallel. The computation engine107contains a detection module108, that is adapted and configured to perform all or portions of malware detection processes, including detection of malware downloads within the network. In an embodiment, the detection module108generates malware detection alerts and sends them from the computation engine107to a display device109, which is in communication with the detection module108. The display device109may be remote from the system10and may also serve as an additional analysis module, where alerts are triaged by users such as network analysts. Feedback from users can also be sent back to the detection module108so that detection processes can be updated and improved over time.

The detection module108executes a series of steps at a regular frequency in order to identify suspicious downloads in the recently acquired data. The run frequency can be configured depending on the network data volume and the desires of network analysts. Each time the process, or process software, runs within the detection module108, it identifies suspicious downloads that occurred in the time interval since the previous run. This interval is referred to as the “test interval,” and the metadata written to database106during the test interval are referred to as the “test data.” As an example, the detection module108could be configured to run daily. In this case, the software would identify suspicious downloads in test data collected by sensors104during the 24-hour test interval prior to each run.

Methods for Malware Detection

FIG.2is a flow diagram of an embodiment of an exemplary method20of malware detection. All or some of the steps of the method20may be performed by the detection module108and/or computation engine107of the exemplary system10ofFIG.1. In the first step201, currently observed test data (i.e. current data) are loaded from the database106. In an embodiment, the test data contains HTTP records with HTTP metadata extracted by sensors104. The HTTP records may consist of information that includes file downloads and other information such as file uploads, HTTP POSTs, authorization requests, and other records produced by normal browsing of the internet103. In step202, the records are filtered based on one or more criteria. In an embodiment, the filtering step utilizes HTTP Path information, and the HTTP records are removed if they do not represent downloads (e.g., if the path is empty or does not contain a file extension). Other filters may also be employed at this step including, but not limited to file type, file size, file extension, file content, file download etc. For example, if network analysts are only interested in detecting suspicious downloads with one or more particular file extensions, then downloads of other extensions may be removed. File extensions of interest may be, for example: 7z, bat, bin, cmd, dll, dmg, doc, docm, dotm, exe, go, hta, iqy, jar, m68k, m86k, msi, pif, potm, ppam, ppsm, pptm, ps1, ps2, py, rar, scr, sh, sldm, tmp, uras, vb, vbe, vbs, ws, wsf, x86, xlam, xlsm, xltm, and zip. The output of step202is a table that contains just the data, e.g., file downloads of interest from the test data, which have been filtered. Herein, this group of files are referred to as “currently filtered test data”. In step203, the filtered test data is saved to the database106, and a filtered set of historical data (“historically filtered test data”) associated with previously observed test file records is loaded from the database106.

In step204, values of a plurality of suspiciousness features1-Nare computed or determined for the currently filtered test data and the historically filtered test data. Suspiciousness features are properties or characteristics related to file type, file typed, domain, outdated User-Agent, outdated Content-Type that are associated with potential malware downloads etc. Features1-Nmay be designed or pre-selected by a user or analyst, and may include, for example information about how many users have accessed a particular domain, the fraction of days within a time period with SLD, a Path, a Referer, a User-Agent, the fraction of the same-extension downloads in a given period of time with the same Content-Type or similar Content-length, and other file name-related suspiciousness. In some embodiments, features1-Nare determined based on historical network data. In sum, the features1-Nare defined to correlate with the suspicious behavior exhibited by malware.

As noted above, a plurality of features1-Nmay be used in the detection of malware according to the embodiments herein, and the invention is not limited to a specific feature set or number of features. In an exemplary embodiment, features 1-4 are described and calculated as follows. Feature 1 is a count of the number of times downloads are made from the observed HTTP Host on the network over an interval (e.g., 30 days) preceding the test interval, multiplied by a negative one (−1) so that unpopular hosts are represented by higher values. Feature 2 is a count of the number of times the observed HTTP Path was downloaded on the network over the 30-day interval preceding the test interval, multiplied by negative one so that unpopular Paths are represented by higher values. Feature 3 is a quantitative measure of outlierness (i.e., the amount by which the feature value is abnormal with respect to a group of other feature value data points) of the observed file size relative to other file downloads with the same extension that occurred on the network over the 30-day interval preceding the test interval. This could be quantified, for example, using the “Local Outlier Factor” algorithm. Feature 4 is a quantitative measure of how strongly the downloaded file name correlates with a list of known malware file names. This is measured, for example, using a supervised machine learning model like a “Long Short-Term Memory” network.

In step205, after the feature1-Nvalues of the currently filtered test data are acquired, they are saved to the database106. Then, a set of historical feature values, saved during previous runs, is loaded from the database106. In an embodiment, the amount of historical feature values that are loaded in step205is selected and configurable based on: 1) the resources available in the computation engine107; and/or 2) a predetermined amount selected by a user or network analyst. In an exemplary embodiment, the detection module108is configured with a daily test interval, and historical feature values that was saved over the seven days preceding the test interval is loaded from the database106. In an embodiment, in which feature 1 is used, a table or list containing the counts of each Host value in the currently filtered test data is written to the database106. Then in the second part, a selected length of time (e.g., 30 days) of the historical summary tables of the historical data from previous runs is loaded and used to compute features1-Nfor each download in the currently filtered test data. This aspect provides the advantage of computational efficiency in part because steps201and202only need to be executed once in each test interval.

In step206, an algorithm such as a Z-score, P-value or similar is used to calculate an outlier score of the values features1-N. In an embodiment, the historical feature values are used to calculate the outlier score for each feature value in the filtered test data. As illustrated in exemplaryFIGS.3and4, the outlier score is a Z-score. In this embodiment, the Z-score for a given observed feature value within a population of values, is a signed measure of the fractional number of standard deviations by which the observed value is above the population mean. An example Z-score calculation utilized in step206is illustrated inFIG.3. The plot contains a distribution of historical feature values300, and an arrow indicating a single observed value301. The mean and standard deviation of the historical distribution are calculated and indicated on the plot. The Z-score of the observed value with respect to the historical distribution is found to be 1.8σ, since the observation is 1.8 standard deviations above the mean of the distribution.

After outlier scores are computed for each features1-Nin step206, a table or list of filtered test data with features and Z-scores is created and passed to step207. In step207, the Z-scores for each row of the filtered test data are merged into a single “suspiciousness score” or output score. In an embodiment, the output score indicates how suspicious or likely a file download may be of qualifying as malware. The Z-score computation, and merging steps are illustrated inFIG.4. Example historical feature distributions and observed feature values for features1-Nare shown in graphs400a-x, respectively. The observed features1-Nvalues yield their respective Z-scores1-N. The individual Z-scores1-Nfor each feature are then passed to the merging step or algorithm. The merging step may include one more calculations such as a sum, weighted average, supervised machine learning model, etc. The merging step401combines the individual Z-scores1-Ninto an output score402. In this example, the output score402has a value of 3.8.

In step208, output scores402are compared to a threshold, and those below a predetermined/selected threshold are removed from the currently filtered test data. Alerts may be generated from the surviving records, i.e., file records associated with output scores402above the threshold, and these file records are considered or further analyzed as (e.g., suspicious downloads of) potential malware. Each alert may contain illustrations and/or data describing the suspicious indicators, behaviors, and underlying metadata for a single suspicious download. In an embodiment, illustrated in step209, the alerts are sent from the computation engine107to a display device109where they can be reviewed and analyzed by network analysts. In an embodiment, the display device109has additional analytical capabilities such that a user may triage the alerts and provide feedback and/or additional algorithms via the display device109and then send them back to the computation engine107so that details of the algorithm may be adjusted accordingly.

The subject matter described herein can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structural means disclosed in this specification and structural equivalents thereof, or in combinations of them. The subject matter described herein can be implemented as one or more computer program products, such as one or more computer programs tangibly embodied in an information carrier (e.g., in a machine readable storage device), or embodied in a propagated signal, for execution by, or to control the operation of, data processing apparatus (e.g., a programmable processor, a computer, or multiple computers). A computer program (also known as a program, software, software application, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file. A program can be stored in a portion of a file that holds other programs or data, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification, including the method steps of the subject matter described herein, can be performed by one or more programmable processors executing one or more computer programs to perform functions of the subject matter described herein by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus of the subject matter described herein can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processor of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. Information carriers suitable for embodying computer program instructions and data include all forms of nonvolatile memory, including by way of example semiconductor memory devices, (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks, (e.g., internal hard disks or removable disks); magneto optical disks; and optical disks (e.g., CD and DVD disks). The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

The subject matter described herein can be implemented in a computing system that includes a back end component (e.g., a data server), a middleware component (e.g., an application server), or a front end component (e.g., a client computer mobile device, wearable device, having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described herein), or any combination of such back end, middleware, and front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.

It is to be understood that the disclosed subject matter is not limited in its application to the details of construction and to the arrangements of the components set forth in the following description or illustrated in the drawings. The disclosed subject matter is capable of other embodiments and of being practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting. As such, those skilled in the art will appreciate that the conception, upon which this disclosure is based, may readily be utilized as a basis for the designing of other structures, methods, and systems for carrying out the several purposes of the disclosed subject matter. It is important, therefore, that the claims be regarded as including such equivalent constructions insofar as they do not depart from the spirit and scope of the disclosed subject matter.

Although the disclosed subject matter has been described and illustrated in the foregoing exemplary embodiments, it is understood that the present disclosure has been made only by way of example, and that numerous changes in the details of implementation of the disclosed subject matter may be made without departing from the spirit and scope of the disclosed subject matter, which is limited only by the claims which follow.