Automatically detecting a malicious file using name mangling strings

Automatically detecting a malicious file using name mangling strings. In one embodiment, a method may include (a) identifying a file, (b) identifying name mangling strings in the file, (c) concatenating the name mangling strings together, (d) hashing the concatenated name mangling strings to generate a signature for the file, (e) clustering the file with other files with matching signatures into a cluster, (f) determining that any of the files in the cluster is malicious, (g) adding the signature to a list of signatures of files known to be malicious, (f) identifying a network device file stored on a network device, (g) repeating (b)-(d) on the network device file, (h) determining that the signature for the network device file matches any signature in the list of signatures of files known to be malicious, and (i) performing a security action on the malicious file on the network device.

BACKGROUND

Some network security applications function to detect malicious files stored on network devices before the malicious files can be executed or otherwise employed in damaging the network or network devices. Examples of malicious files include files that contain viruses or malware.

For example, a network security application may attempt to identify a malicious file by comparing a suspicious file that is suspected of being malicious to a list of files known to be malicious. If the suspicious file matches any of the files in the list of files known to be malicious, the network security application may identify the suspicious file as a malicious file. However, purveyors of malicious files have attempted to avoid detection of their malicious files by causing variations of their malicious files to have minor differences which cause the malicious files to not match up exactly with the files in a list of malicious files. The proliferation of these variations of malicious files has rendered many network security applications incapable of accurately detecting many malicious files, which leaves network devices vulnerable to undetected malicious files.

SUMMARY

In one embodiment, a computer-implemented method for automatically detecting a malicious file using name mangling strings may be performed, at least in part, by a computing device including at least one processor. The method may include (a) identifying a file, (b) identifying name mangling strings in the file, (c) concatenating the name mangling strings together, (d) hashing the concatenated name mangling strings to generate a signature for the file, (e) clustering the file with other files with matching signatures into a cluster, (f) determining that one of the files in the cluster is malicious, (g) adding the signature to a list of signatures of files known to be malicious, (f) identifying a network device file stored on a network device, (g) repeating (b)-(d) on the network device file, (h) determining that the signature for the network device file matches any signature in the list of signatures of files known to be malicious and that the network device file is therefore a malicious file, and (i) performing a security action on the malicious file on the network device to protect the network device.

In some embodiments, the file and the network device file may be Portable Executable (PE) files. Also, in some embodiments, (b) may include identifying name mangling strings in a Data section of the PE file. Further, in some embodiments, (c) may include concatenating the name mangling strings together with one or more delimiter characters between each of the name mangling strings. Also, in some embodiments, (c) may include concatenating the name mangling strings together in a sequence that matches a sequence that the name mangling strings are listed in a Data section of a PE file. Further, in some embodiments, (d) may include performing a Secure Hash Algorithm-256 (SHA-256) algorithm on the concatenated name mangling strings to generate the signature for the file. Also, in some embodiments, (i) may include quarantining the malicious file on, or removing the malicious file from, the network device to prevent the malicious file from executing on the network device.

Further, in some embodiments, one or more non-transitory computer-readable media may include one or more computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform a method for automatically detecting a malicious file using name mangling strings.

It is to be understood that both the foregoing summary and the following detailed description are explanatory and are not restrictive of the invention as claimed.

DETAILED DESCRIPTION

Some embodiments in this disclosure relate to automatically detecting a malicious file using name mangling strings.

Some network security applications function to detect malicious files stored on network devices, such as files that include viruses or malware, before the malicious files can be executed or otherwise employed in damaging the network devices. For example, a network security application may attempt to identify malicious files by comparing a file signature of a suspicious file that is suspected of being malicious to a list of file signatures of files known to be malicious. If the suspicious file's signature matches any of the file signatures in the list of malicious files, the network security application may identify the suspicious file as a malicious file. However, purveyors of malicious files have attempted to avoid detection of their malicious files by causing variations of their malicious files to have minor differences which cause the file signatures of the malicious files to not match up exactly with any of the file signatures in lists of malicious files. The proliferation of these variations of malicious files has rendered many network security applications incapable of detecting many malicious files, which leaves networks and network devices vulnerable to undetected malicious files.

The embodiments disclosed herein may enable automatically detecting a malicious file using name mangling strings. In some embodiments, automatically detecting a malicious file using name mangling strings may include generating a file signature for a file by identifying name mangling strings in the file, concatenating the name mangling strings together, and hashing the concatenated name mangling strings to generate a file signature for the file. These file signatures generated using name mangling strings can be employed in the generation of a list of file signatures of files known to be malicious and in generating a suspicious file's signature. Even when purveyors of malicious files cause variations of their malicious files to have minor differences, a network security application that detects malicious files using file signatures generated using name mangling strings may be capable of accurately clustering variations of a malicious file together as matching malicious files with few or no false positives in the cluster. This may render the network security application capable of more accurately detecting malicious files, which leaves networks and network devices less vulnerable to undetected malicious files.

Turning to the figures,FIG. 1illustrates an example system100configured for automatically detecting a malicious file using name mangling strings. The system100may include a network102, network devices104a-104n, and a security server106.

In some embodiments, the network102may be configured to communicatively couple the network devices104a-104nto one another as well as to the security server106. In some embodiments, the network102may be any wired or wireless network, or combination of multiple networks, configured to send and receive communications between systems and devices. In some embodiments, the network102may include a Personal Area Network (PAN), a Local Area Network (LAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), a Storage Area Network (SAN), or some combination thereof. In some embodiments, the network102may also be coupled to, or may include, portions of a telecommunications network, including telephone lines, for sending data in a variety of different communication protocols, such as a cellular network or a Voice over IP (VoIP) network.

In some embodiments, each of the network devices104a-104nmay be any computer system capable of communicating over the network102, examples of which are disclosed herein in connection with the computer system400ofFIG. 4. Each of the network devices104a,104b, and104nmay store files103a-103n,105a-105n, and107a-107n, respectively. Any one of these files may be a malicious file. A malicious file may be, but is not limited to, a file that contains one or more of spyware, a virus, a worm, a logic bomb, a trapdoor, a Trojan horse, a Remote Admin Trojan (RAT), malware, mobile malicious code, a malicious font, and a rootkit.

In some embodiments, the security server106may be any computer system capable of communicating over the network102and capable of monitoring the network devices104a-104n, examples of which are disclosed herein in connection with the computer system400ofFIG. 4. In some embodiments, the server may include a security module108that may be configured to detect, over the network102, whether any of the files103a-103n,105a-105n, and107a-107non the network devices104a,104b, and104nis a malicious file, as disclosed in greater detail in connection withFIGS. 2 and 3A-3Cherein. In some embodiments, the security module108may include, or have access to, files109a-109n, file clusters110including clusters112a-112n, a list of malicious files114, and a list of clean files116. In some embodiments, the security module108may include, or be part of, an Antivirus (AV) application (which may actually protect against forms of malicious files beyond files that only include a virus) or a Security Information and Event Management (SIEM) application.

Modifications, additions, or omissions may be made to the system100without departing from the scope of the present disclosure. For example, in some embodiments, the system100may include additional components similar to the components illustrated inFIG. 1that each may be configured similarly to the components illustrated inFIG. 1.

FIG. 2illustrates a user interface200of an application that displays a file202including name mangling strings. In particular, the file202illustrated in the user interface200ofFIG. 2may be a file stored on a security server, such as the file109astored on the security server106. Alternatively, the file202may be a filed stored on a network device, such as the file103astored on the network device104aofFIG. 1. As disclosed inFIG. 2, the file202illustrated in the user interface200is named setup.exe and is found in a folder C:\Tools of a network device. In some embodiments, the file202may be a Portable Executable (PE) file that contains name mangling strings.

The name mangling strings in the data204of the file202may function to encode function and variable names of an executable file into unique names so that linkers can separate common names in the language of the executable file. The name mangling strings in the data204of the file202may be found in the Data section208of the file202. The name mangling strings may be identified by one or more particular delimiter character(s) at the beginning and the end of each name mangling string. For example, the file202may include name mangling strings that include a ‘?’ delimiter character at the beginning of the name mangling string and two ‘@@’ characters at the end of the name mangling string, such as the name mangling strings208,210, and212.

The name mangling strings of the file202may be employed in the generation of a file signature for the file202. This file signature may then be compared to a list of file signatures that were similarly generated for files known to be malicious files. If there is a match, the file202may be determined to be a malicious file, and the file may be added to a list of malicious files, such as the list of malicious files114ofFIG. 1, or an appropriate security action may be taken against the file202, such as removing or quarantining the file202to protect a network device on which the file202is stored, such as the network device104aofFIG. 1.

Modifications, additions, or omissions may be made to the example file202illustrated in the user interface200ofFIG. 2without departing from the scope of the present disclosure. For example, although the file202is a Portable Executable (PE) file, it is understood that the file202may be any file that includes name mangling strings. Further, the name mangling strings may have a format that is different from the name mangling strings illustrated inFIG. 2.

FIGS. 3A-3Care a flowchart of an example method300for automatically detecting a malicious file using name mangling strings. The method300may be performed, in some embodiments, by a device or system, such as by the security module108executing on the security server106ofFIG. 1. In these and other embodiments, the method300may be performed by one or more processors based on one or more computer-readable instructions stored on one or more non-transitory computer-readable media. The method300will now be described in connection withFIGS. 1, 2, and 3A-3C.

The method300may include, at block302, identifying a file. In some embodiments, the file identified at block302(and the network device file identified at block324, as discussed below) may be a Portable Executable (PE) file. For example, the security module108may identify, at block302, the file202, which may be a PE file and which may be the file109astored on the security server106. In this example, the files109a-109nmay be files that are targeted to have file signatures generated and that are targeted to be added to the file clusters110.

The method300may include, at block304, identifying name mangling strings in the file. In some embodiments, the name mangling strings identified at block304may include all name mangling strings in the file or a subset of all the name mangling strings in the file, such as all name mangling strings in a section of the file. For example, the security module108may identify, at block304, name mangling strings208,210, and212in the Data section206of the file202. In this example, the name mangling strings208may also include all other name mangling strings in the Data section206of the file202.

The method300may include, at block306, concatenating the name mangling strings together. In some embodiments, the concatenating at block306may include concatenating the name mangling strings together with one or more delimiter characters between each of the name mangling strings. In some embodiments, the concatenating at block306may include concatenating the name mangling strings together in a sequence that matches a sequence that the name mangling strings are listed in a Data section of a PE file. For example, the security module108may concatenate, at block306, the name mangling strings208,210, and212together, as well as all other name mangling strings in the Data section206of the PE file202, with one or more delimiter characters between each of the name mangling strings, and in a sequence that matches a sequence that the name mangling strings are listed in the Data section206of the PE file202.

The method300may include, at block308, hashing the concatenated name mangling strings to generate a signature for the file. In some embodiments, the hashing at block308may include performing a Secure Hash Algorithm-256 (SHA-256) algorithm on the concatenated name mangling strings to generate the signature for the file. For example, the security module108may perform, at block308, a SHA-256 algorithm on the concatenated name mangling strings to generate a signature for the file202.

The method300may include, at block310, clustering the file with other files with matching signatures into a cluster. For example, the security module108may cluster, at block310, the file202with other files with matching signatures into a cluster112ain the file clusters110of the security server106.

The method300may include, at decision block312, determining whether there are more files targeted for having a file signature generated and targeted for being added to a cluster. If so (Yes at decision block312), the method300may return to block302to process the next file. If not (No at decision block312), the method300may continue to block314. For example, the security module108may determine, at decision block312, that the files109b-109nstored on the security server106are still waiting to have a file signature generated and waiting to be added to one of the file clusters110(Yes at decision block312). Therefore, the method300may return to block302to process the next file109b. Alternatively, the security module108may determine, at decision block312, that all of the files109a-109nhave already had file signatures generated and have been added to one of the file clusters110(No at decision block312). Therefore, the method300may continue to block314.

The method300may include, at block314, identifying a cluster. For example, the security module108may identify, at block314, the cluster112ain the file clusters110.

The method300may include, at decision block316, determining whether one of the files in the cluster is malicious. If so (Yes at decision block316), the method300may include, at block318, adding the signature to a list of signatures of files known to be malicious. If not (No at decision block316), the method300may include, at block320, adding the signature to a list of signatures of files known to be clean. For example, the security module108may determine, at decision block316, that the cluster112aincludes a file that is malicious and may, at block318, add the file signature of the file (which is the same as all the other file signatures in the cluster112a) to the list of signatures in the list of malicious files114. Alternatively, the security module108may determine, at decision block316, that the cluster112aincludes a file that is not malicious and may, at block320, add the file signature of the file (which is the same as all the other file signatures in the cluster112a) to the list of signatures in the list of clean files116.

The method300may include, at decision block322, determining whether there are more clusters targeted for being categorized as either malicious or clean. If so (Yes at decision block322), the method300may return to block314to categorize the next cluster. If not (No at decision block322), the method300may continue to block324. For example, the security module108may determine, at decision block322, that the clusters112b-112nin the file clusters110are still waiting to be categorized as either malicious or clean (Yes at decision block322). Therefore, the method300may return to block314to categorize the next cluster112b. Alternatively, the security module108may determine, at decision block322, that all of the clusters112a-112nin the file clusters110have already been categorized as either malicious or clean (No at decision block322). Therefore, the method300may continue to block324.

The method300may include, at block324, identifying a network device file stored on a network device. For example, the security module108may identify, at block324, the file103astored on the network device104a.

The method300may include, at block326, repeating blocks304-308on the network device file. For example, the security module108may repeat, at block326, blocks304-308on the file103ain order to identify, at block304, name mangling strings in the file103a, concatenate, at block306, the name mangling strings together, and hash, at block308, the concatenated name mangling strings to generate a signature for the file103a.

The method300may include, at decision block328, determining whether the signature matches any signature in the list of signatures of files known to be malicious. If so (Yes at decision block328), the method300may include, at block330, identifying the network device file as a malicious file and, at block332, performing a security action on the malicious file on the network device to protect the network device. If not (No at decision block328), the method300may proceed to decision block334. In some embodiments, the performing of the security action at block332may include quarantining the malicious file on, or removing the malicious file from, the network device to prevent the malicious file from executing on the network device. For example, the security module108may determine, at decision block328, that the signature of the file103amatches any signature in the list of malicious files114and may therefore identify, at block330, the file103astored on the network device104aas a malicious file and may also perform, at block332, a security action on the malicious file103aon the network device104ato protect the network device104afrom the malicious file103a, such as quarantining the malicious file103aon, or removing the malicious file103afrom, the network device104ato prevent the malicious file103afrom executing on the network device104a. Alternatively, the security module108may determine, at decision block328, that the signature of the file103adoes not match any signature in the list of malicious files114. Therefore, the method300may continue to decision block334.

The method300may include, at decision block334, determining whether the signature matches any signature in the list of signatures of files known to be clean. If so (Yes at decision block334), the method300may include, at block336, identifying the network device file as a clean file. If not (No at decision block334), the method300may include, at block338, identifying the network device file as an unknown file. For example, the security module108may determine, at decision block334, that the signature of the file103amatches any signature in the list of clean files116and may therefore identify, at block336, the file103astored on the network device104aas a clean file. Alternatively, the security module108may determine, at decision block334, that the signature of the file103adoes not match any signature in the list of clean files116and may therefore identify, at block338, the file103aas an unknown file.

In some embodiments, even when purveyors of malicious files cause variations of their malicious files to have minor differences, the method300may be capable of more accurately clustering variations of a malicious file together as matching malicious files with few or no false positives in the cluster. This renders the method300capable of more accurately detecting malicious files, which leaves networks and network devices less vulnerable to undetected malicious files.

Although the blocks of the method300are illustrated inFIGS. 3A-3Cas discrete blocks, various blocks may be divided into additional blocks, combined into fewer blocks, reordered, expanded, or eliminated, depending on the desired implementation. For example, in some embodiments, blocks302-312may be performed without performing blocks314-338, or blocks314-322may be performed without performing blocks302-312and324-338, or blocks324-338may be performed without performing blocks302-322. Also, in some embodiments, block332may be eliminated or may be performed by a network administrator or other entity that is different from the entity performing the other blocks of the method300. Further, in some embodiments, blocks302-312,314-322, and324-338may be performed in parallel with each other.

Further, it is understood that the method300may improve the functioning of a network environment. For example, the functioning of the security server106or any of the network devices104a-104nofFIG. 1may itself be improved by the method300. For example, any of these computer systems may be improved by automatically detecting a malicious file stored on the computer systems using name mangling strings so that the malicious file may be removed or quarantined to prevent the malicious file from damaging the computer system. The method300may thus result in more accurate detection of malicious files than conventional detection methods that use conventional file signatures, which leaves networks and network devices less vulnerable to undetected malicious files.

Also, the method300may improve the technical field of detecting malicious files and securing network devices against malicious files. Employing name mangling strings in the generation of more accurate file signatures, and then employing the more accurate file signatures for detecting malicious files, is an improvement over conventional attempts at detecting malicious files using conventional file signatures.

FIG. 4illustrates an example computer system400that may be employed in automatically detecting a malicious file using name mangling strings. In some embodiments, the computer system400may be part of any of the systems or devices described in this disclosure. For example, the computer system400may be part of any of the network devices or the security server devices ofFIGS. 1 and 2.

The computer system400may include a processor402, a memory404, a file system406, a communication unit408, an operating system410, a user interface412, and a security module414, which all may be communicatively coupled. In some embodiments, the computer system may be, for example, a desktop computer, a client computer, a server computer, a mobile phone, a laptop computer, a smartphone, a smartwatch, a tablet computer, a portable music player, or any other computer system.

Generally, the processor402may include any suitable special-purpose or general-purpose computer, computing entity, or processing device including various computer hardware or software modules and may be configured to execute instructions stored on any applicable computer-readable storage media. For example, the processor402may include a microprocessor, a microcontroller, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a Field-Programmable Gate Array (FPGA), or any other digital or analog circuitry configured to interpret and/or to execute program instructions and/or to process data, or any combination thereof. In some embodiments, the processor402may interpret and/or execute program instructions and/or process data stored in the memory404and/or the file system406. In some embodiments, the processor402may fetch program instructions from the file system406and load the program instructions into the memory404. After the program instructions are loaded into the memory404, the processor402may execute the program instructions. In some embodiments, the instructions may include the processor402performing one or more blocks of the method300ofFIGS. 3A-3C.

The memory404and the file system406may include computer-readable storage media for carrying or having stored thereon computer-executable instructions or data structures. Such computer-readable storage media may be any available non-transitory media that may be accessed by a general-purpose or special-purpose computer, such as the processor402. By way of example, and not limitation, such computer-readable storage media may include non-transitory computer-readable storage media including Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory devices (e.g., solid state memory devices), or any other storage media which may be used to carry or store desired program code in the form of computer-executable instructions or data structures and which may be accessed by a general-purpose or special-purpose computer. Combinations of the above may also be included within the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause the processor402to perform a certain operation or group of operations, such as one or more blocks of the method300ofFIGS. 3A-3C. These computer-executable instructions may be included, for example, in the operating system410, in one or more applications, such as the security module414, or in some combination thereof.

The communication unit408may include any component, device, system, or combination thereof configured to transmit or receive information over a network, such as the network102ofFIG. 1. In some embodiments, the communication unit408may communicate with other devices at other locations, the same location, or even other components within the same system. For example, the communication unit408may include a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device (such as an antenna), and/or chipset (such as a Bluetooth device, an 802.6 device (e.g., Metropolitan Area Network (MAN)), a WiFi device, a WiMax device, a cellular communication device, etc.), and/or the like. The communication unit408may permit data to be exchanged with a network and/or any other devices or systems, such as those described in the present disclosure.

The operating system410may be configured to manage hardware and software resources of the computer system400and configured to provide common services for the computer system400.

The user interface412may include any device configured to allow a user to interface with the computer system400. For example, the user interface412may include a display, such as an LCD, LED, or other display, that is configured to present video, text, application user interfaces, and other data as directed by the processor402. The user interface412may further include a mouse, a track pad, a keyboard, a touchscreen, volume controls, other buttons, a speaker, a microphone, a camera, any peripheral device, or other input or output device. The user interface412may receive input from a user and provide the input to the processor402. Similarly, the user interface412may present output to a user.

The security module414may be one or more computer-readable instructions stored on one or more non-transitory computer-readable media, such as the memory404or the file system406, that, when executed by the processor402, is configured to perform one or more blocks of the method300ofFIGS. 3A-3C. In some embodiments, the security module414may be part of the operating system410or may be part of an application of the computer system400, or may be some combination thereof. In some embodiments, the security module414may function as the security module108ofFIGS. 1 and 2.

Modifications, additions, or omissions may be made to the computer system400without departing from the scope of the present disclosure. For example, although each is illustrated as a single component inFIG. 4, any of the components402-414of the computer system400may include multiple similar components that function collectively and are communicatively coupled. Further, although illustrated as a single computer system, it is understood that the computer system400may include multiple physical or virtual computer systems that are networked together, such as in a cloud computing environment, a multitenancy environment, or a virtualization environment.

As indicated above, the embodiments described herein may include the use of a special purpose or general purpose computer (e.g., the processor402ofFIG. 4) including various computer hardware or software modules, as discussed in greater detail below. Further, as indicated above, embodiments described herein may be implemented using computer-readable media (e.g., the memory404or file system406ofFIG. 4) for carrying or having computer-executable instructions or data structures stored thereon.

In some embodiments, the different components and modules described herein may be implemented as objects or processes that execute on a computing system (e.g., as separate threads). While some of the methods described herein are generally described as being implemented in software (stored on and/or executed by general purpose hardware), specific hardware implementations or a combination of software and specific hardware implementations are also possible and contemplated.

The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention as claimed to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described to explain practical applications, to thereby enable others skilled in the art to utilize the invention as claimed and various embodiments with various modifications as may be suited to the particular use contemplated.