Patent Description:
Malicious software developers attempt to install software (e.g., applications) on user devices in order to cause the user devices to execute operations harmful to users. For instance, malicious software, when installed on a user device, may steal sensitive data from a user, track the user device, and/or perform other fraudulent operations without a user's consent. Malicious software is generally referred to as malware, and once installed on a user device, the malware can continue to grow on the user device and/or spread to other devices in communication with the user device.

Identifying malware is generally a cat and mouse game, where one side continually pursues malware authors that are always updating their methods for installing malware on user devices to evade detection. Generally, malware authors spend significant effort in developing install patterns in order to install malware on user devices without being detected. While humans can add known applications associated with previously identified malware and/or known domain names accessed by malware to whitelists, malicious authors will periodically change the names of the applications and/or domain names in order to evade detection by these whitelists.

PEVNY AND SOMOL describe in <CIT> techniques for classifying devices as being infected with malware based on learned indicators of compromise. One method described therein includes receiving at a security analysis device, traffic flows from a plurality of entities destined for a plurality of users, aggregating the traffic flows into discrete bags of traffic, wherein the bags of traffic comprise a plurality of flows of traffic for a given user over a predetermined period of time, extracting features from the bags of traffic and aggregating the features into per-flow feature vector, aggregating the per-flow feature vectors into per-destination domain aggregated vectors, combining the per-destination-domain aggregated vectors into a per-user aggregated vector, and classifying a computing device used by a given user as infected with malware when indicators of compromise detected in the bags of traffic indicate that the per-user aggregated vector for the given user includes suspicious features among the extracted features.

<CIT> an on-device security vulnerability detection method that performs dynamic analysis of application programs on a mobile device. In one aspect, an operating system of a mobile device is described that is configured to include instrumentations and an analysis application program package is configured for installation on the mobile device to interact with the instrumentations. It is described that when an application program executes on the mobile device, the instrumentations enables recording of information related to execution of the application program. The described analysis application interfaces with the instrumented operating system to analyze the behaviors of the application program using the recorded information. The described application program is categorized (e.g., as benign or malicious) based on its behaviors, for example by using machine learning models.

<CIT> systems and methods that allow a computer security system to automatically classify target objects using a cascade of trained classifiers, for applications including mal-ware, spam, and/or fraud detection. The cascade described therein comprises several levels, each level including a set of classifiers. Described classifiers are trained in the predetermined order of their respective levels. Each described classifier is trained to divide a corpus of records into a plurality of record groups so that a substantial proportion (e.g., at least <NUM>%, or all) of the records in one such group are members of the same class. Between described training classifiers of consecutive levels of the cascade, a set of training records of the respective group is discarded from the training corpus. When used to classify an unknown target object, some embodiments employ the classifiers in the order of their respective levels.

One aspect of the disclosure provides a method for identifying malicious software. The method includes receiving, at data processing hardware, a software application, and executing, by the data processing hardware, the software application.

The method also includes identifying, by the data processing hardware, a plurality of uniform resource identifiers the software application interacts with during execution of the software application, and generating, by the data processing hardware, a vector representation for the software application using a feed-forward neural network configured to receive the plurality of uniform resource identifiers as feature inputs. The
method also includes, determining, by the data processing hardware, similarity scores for a pool of training applications stored in memory hardware in communication with the data processing hardware, each similarity score associated with a corresponding training application and indicating a level of similarity between the vector representation for the software application and a respective vector representation for the corresponding training application. The method also includes, flagging, by the data processing hardware, the software application as belonging to a potentially harmful application category when one or more of the training applications have similarity scores that satisfy a similarity threshold and include a potentially harmful application label.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, identifying the plurality of resource identifiers includes identifying a plurality of domain names the software application visits during the execution of the software application. The feed-forward neural network may include a vector space model configured to determine an n-dimensional numerical vector representation for each of the identified uniform resource identifiers, and calculate the vector representation for the software application by averaging the n-dimensional numerical vector representations for the identified uniform resource identifiers.

In some examples, determining the similarity scores for the pool of training applications includes calculating a respective cosine similarity between the vector representation for the software application and the respective vector representation for each corresponding training application. The vector representation for the software application may include an n-dimensional vector of numerical values. The method may also include retrieving, by the data processing hardware, the training applications associated with the top-k highest similarity scores in the pool of training applications from the memory hardware.

In some implementations, the method also includes identifying, by the data processing hardware, a potentially harmful category associated with a majority of the training applications in the pool of training applications each having the corresponding similarity score that satisfies the similarity threshold and comprising the potentially harmful application label, and assigning, by the data processing hardware, the software application to the identified potentially harmful category. The potentially harmful category assigned to the software application may include one of a hostile downloader application, a phishing application, a rooting Trojan application, a spyware application, a ransomware application, a malware application, or an escalating privileges application. This list of potentially harmful categories is non-exhaustive and may include any other application that is malicious or otherwise potentially harmful. The method may also include, after flagging the software application as belonging to the potentially harmful application category: receiving, by the data processing hardware, a download request to download the software application from a user device in communication with the data processing hardware; and in response to receiving the download request, transmitting a warning notification to the user device, the warning notification indicating that the software application is flagged as belonging to the potentially harmful application category.

Another aspect of the disclosure provides a method for identifying malicious software. The method includes receiving, at data processing hardware, an application install pattern from a user device, the application install pattern indicating a sequence of n-applications installed on the user device. For each application in the sequence of n-applications, the method also includes, generating, by the data processing hardware, a numerical vector representation for the corresponding application using a feed-forward neural network configured to receive each application and the order of each application in the sequence of n-applications as feature inputs, and clustering, by the data processing hardware, the corresponding application in a free vector space based on the numerical vector representation for the corresponding application. The method also includes determining, by the data processing hardware, whether any of the applications in the sequence of n-applications are clustered with training applications identified as malware, and for each application clustered with training applications identified as malware, identifying, by the data processing hardware, the corresponding application in the sequence of n-applications as malware.

This aspect may include one or more of the following optional features. The method may optionally include, for each application in the sequence of n-applications identified as malware: labeling, by the data processing hardware, the application as belonging to a potentially harmful application category; and storing, by the data processing hardware, the application and the corresponding numerical vector representation for the application in the memory hardware in communication with the data processing hardware. The numerical vector representation generated for each corresponding application may include a cryptographic hash.

In some implementations, the method further includes: determining, by the data processing hardware, whether any of the applications in the sequence of n-applications were previously identified as malware; and when at least one of the applications in the sequence of n-applications was previously identified as malware, identifying, by the data processing hardware, one or more of the remaining applications in the sequence of n-applications as malware. The feed-forward neural network model may include a vector space model configured to determine an n-dimensional numerical vector representation for each application in the sequence of n-applications installed on the user device. Additionally, the vector space model may optionally be configured to cluster each application in the sequence of n-applications in the free vector space near training applications having similar n-dimensional numeric vector representations. The method may also include, after identifying a corresponding application in the sequence of n-applications as malware, transmitting a warning notification to the user device, the warning notification indicating that the corresponding application installed on the user device comprises malware.

Another aspect of the disclosure provides a system for identifying malicious software. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed by the data processing hardware cause the data processing hardware to perform operations that include receiving a software application, executing the software application, and identifying a plurality of uniform resource identifiers the software application interacts with during execution of the software application. The operations also include generating a vector representation for the software application using a feed-forward neural network configured to receive the plurality of uniform resource identifiers as feature inputs and determining similarity scores for a pool of training applications stored in the memory hardware. Each similarity score is associated with a corresponding training application and indicates a level of similarity between the vector representation for the software application and a respective vector representation for the corresponding training application. The operations also include flagging the software application as belonging to a potentially harmful application category when one or more of the training applications have similarity scores that satisfy a similarity threshold and include a potentially harmful application label.

This aspect may include one or more of the following optional features. In some implementations, identifying the plurality of resource identifiers includes identifying a plurality of domain names the software application visits during the execution of the software application. The feed-forward neural network may include a vector space model configured to determine an n-dimensional numerical vector representation for each of the identified uniform resource identifiers, and calculate the vector representation for the software application by averaging the n-dimensional numerical vector representations for the identified uniform resource identifiers.

In some examples, determining the similarity scores for the pool of training applications includes calculating a respective cosine similarity between the vector representation for the software application and the respective vector representation for each corresponding training application. The vector representation for the software application may include an n-dimensional vector of numerical values. The operations may also include retrieving the training applications associated with the top-k highest similarity scores in the pool of training applications from the memory hardware.

In some implementations, the operations also include identifying a potentially harmful category associated with a majority of the training applications in the pool of training applications each having the corresponding similarity score that satisfies the similarity threshold and comprising the potentially harmful application label, and assigning the software application to the identified potentially harmful category. The potentially harmful category assigned to the software application may include one of a hostile downloader application, a phishing application, a rooting Trojan application, a spyware application, a ransomware application, a malware application, or an escalating privileges application. This list of potentially harmful categories is non-exhaustive and may include any other application that is malicious or otherwise potentially harmful. The operations may also include, after flagging the software application as belonging to the potentially harmful application category: receiving a download request to download the software application from a user device in communication with the data processing hardware; and in response to receiving the download request, transmitting a warning notification to the user device, the warning notification indicating that the software application is flagged as belonging to the potentially harmful application category.

Another aspect of the disclosure provides a system for identifying malicious software. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed by the data processing hardware cause the data processing hardware to perform operations that include receiving an application install pattern from a user device, the application install pattern indicating a sequence of n-applications installed on the user device. For each application in the sequence of n-applications, the operations also include generating a numerical vector representation for the corresponding application using a feed-forward neural network configured to receive each application and the order of each application in the sequence of n-applications as feature inputs, and clustering the corresponding application in a free vector space based on the numerical vector representation for the corresponding application. The operations also include determining whether any of the applications in the sequence of n-applications are clustered with training applications identified as malware, and for each application clustered with training applications identified as malware, identifying the corresponding application in the sequence of n-applications as malware.

This aspect may include one or more of the following optional features. The operations may optionally include, for each application in the sequence of n-applications identified as malware: labeling the application as belonging to a potentially harmful application category; and storing the application and the corresponding numerical vector representation for the application in the memory hardware in communication with the data processing hardware. The numerical vector representation generated for each corresponding application may include a cryptographic hash.

In some implementations, the operations further include: determining whether any of the applications in the sequence of n-applications were previously identified as malware; and when at least one of the applications in the sequence of n-applications was previously identified as malware, identifying one or more of the remaining applications in the sequence of n-applications as malware. The feed-forward neural network model may include a vector space model configured to determine an n-dimensional numerical vector representation for each application in the sequence of n-applications installed on the user device. Additionally, the vector space model may optionally be configured to cluster each application in the sequence of n-applications in the free vector space near training applications having similar n-dimensional numeric vector representations. The operations may also include, after identifying a corresponding application in the sequence of n-applications as malware, transmitting a warning notification to the user device, the warning notification indicating that the corresponding application installed on the user device comprises malware.

This disclosure is directed toward identifying malicious software based upon an application install pattern from a user device and/or uniform resource identifiers (URIs) that a software application interacts with during execution in a controlled environment. The URIs may include uniform resource locators (URLs) or domain names. An application install pattern indicates a sequence of applications installed on a corresponding user device. As used herein, the term 'sequence of applications' refers to which applications are installed on the user device and the order in which the applications are installed on the user device. Generally, a user device infected with malware will alter the application install pattern on the device as the malware builds by controlling the user device to continue installing malicious software applications on the user device. On the other hand, malicious software executing on a user device may also access/interact with specific URIs in order to connect the user device to malicious servers for infecting the user device with malware and/or extracting user sensitive data from the user device.

The techniques described herein for identifying malicious software help to ensure/maintain the integrity and security of a user device, to avoid or reduce the likelihood of the device, or data therein, becoming corrupted or accessed, used, modified, and/or distributed in an unauthorized manner. The proposed techniques may allow for this to take place, without unduly affecting/interfering with user engagement with a user device and, in the case of the techniques that identify URIs including URLs or domain names, without the application necessarily even being first downloaded to a user device.

In some implementations, malicious software is identified using a neural network model trained upon application install patterns from different user devices. The neural network model may include a vector space model that determines an n-dimensional numerical vector representation in a vector space for each application in a corresponding application install pattern of a corpus of application installed patterns. The n-dimensional numerical vector representation may be referred to as a corresponding hash or embedding in the vector space. During training, the vector space model automatically clusters the applications based on the n-dimensional numerical vector representations in the vector space such that each cluster of applications may indicate/identify a corresponding application category. For instance, news applications may be clustered together, social media applications may be clustered together, and different types of malware may be clustered together. Advantageously, when the vector space model clusters a new or unknown application into a category associated with previously identified malware, there is a strong assumption that the new or unknown application is also malware. Moreover, when an application in a given install pattern has been previously identified as malware, the vector space model may identify other applications in that same install pattern as new variations of malware that may not have been identified.

In other implementations, the neural network model is trained upon URIs that different software applications interact with to identify malicious software. The URIs may include URLs associated with domain names the different software applications access while executing in a controlled execution environment. For instance, when a software application is released by a developer, an emulator may execute the software application in a controlled execution environment and identify the URIs (e.g., domain names) the software application interacts with during execution of the software application. In some examples, the neural network model includes a vector space model configured to determine an n-dimensional numerical vector representation in a vector space for each of the identified URIs (e.g., domain names) and calculate a vector representation for the software application by averaging the n-dimensional numerical vector representations for the identified URIs. During training, the vector space model determines vector representations for a pool of training applications and automatically clusters the training applications based on the vector representations such that each cluster of applications may indicate/identify a corresponding application category. Advantageously, by representing a new or unknown application based on corresponding n-dimensional numerical vector representations for URIs associated with the application, the new/unknown application may be assigned to a category associated with other applications represented by similar n-dimensional numerical representations for URIs associated with those applications. Hence, new or unknown applications may be flagged as potentially harmful when they fall into a category that includes other applications previously labeled as being potentially harmful.

Referring to <FIG>, in some implementations, an example system <NUM> includes a developer device <NUM> and a customer device <NUM> in communication with a remote system <NUM> via network <NUM>. The developer device <NUM> may be associated with a developer <NUM> that releases software applications <NUM> onto the remote system <NUM> and the customer device <NUM> may be associated with a customer <NUM> that may download released software applications <NUM> from the remote system <NUM>. The remote system <NUM> may be a distributed system (e.g., a cloud environment) having scalable / elastic computing resources <NUM> (e.g., data processing hardware) and/or storage resources <NUM>. In some implementations, computing resources <NUM> of the remote system <NUM> execute a malware identifier <NUM> that receives a software application <NUM> released by the developer device <NUM> and/or receives an application install pattern <NUM> indicating a sequence of applications <NUM> installed on the customer device <NUM>. In some examples, the applications <NUM> correspond to mobile applications. The remote system <NUM> may communicate with data storage (e.g., memory hardware) <NUM>.

In some implementations, the malware identifier <NUM> of the data processing hardware <NUM> implements an emulator <NUM> configured to execute the software application <NUM> received from the developer device <NUM> in a secure execution environment and identify one or more uniform resource identifiers (URIs) <NUM> the software application <NUM> interacts with (i.e., accesses) during execution of the software application <NUM>. As used herein, the URIs may refer to domain names <NUM> or uniform resource locators (URLs) <NUM> associated with respective domain names. The emulator <NUM> may include one or more test devices capable of executing the software application <NUM> in a secure execution environment. The test devices may include virtual devices and/or physical devices.

Additionally, the malware identifier <NUM> of the data processing hardware <NUM> employs a neural network model <NUM> that may be trained using URIs <NUM> as feature inputs. The training URIs <NUM> may be associated with a pool of training applications <NUM> that interact with the training URIs <NUM> such that the neural network model <NUM> calculates a numerical vector representation <NUM> (e.g., domain embedding) in a free vector space for each URI <NUM> and then generates a corresponding vector representation <NUM> (e.g., application embedding) for each training application <NUM> by averaging the numerical vector representations for the corresponding URIs <NUM>. The neural network model <NUM> may store the numerical vector representations <NUM> for the URIs <NUM> and the vector representations <NUM> for the training applications <NUM> in the data storage hardware <NUM>.

In some examples, the neural network model <NUM> is trained using application install patterns <NUM> from different customer devices <NUM> as feature inputs. Each training application install pattern <NUM> indicates a corresponding sequence of n-applications <NUM> installed on the corresponding customer device <NUM>. In these examples, the neural network model <NUM> is configured to calculate a numerical vector representation <NUM> in a free vector space for each application <NUM> of a corresponding training application install pattern <NUM> and cluster each application <NUM> in the free vector space based on the corresponding numerical vector representation <NUM>. The numerical vector representation <NUM> may also be referred to as an application embedding <NUM>. The neural network model <NUM> may store the numerical vector representations <NUM> for the applications <NUM> of the training application install patterns <NUM> received from the customer devices <NUM> in the data storage <NUM>. In some implementations, each numerical vector representation <NUM> (domain embedding) is represented as a cryptographic hash value that cannot be decrypted back. For instance, each application embedding <NUM> may include a corresponding cryptographic hash obtained from a secure hash algorithm (SHA) such as the SHA-<NUM> algorithm generating a <NUM>-bit (<NUM>-byte) hash. In some examples, the neural network model <NUM> determines a corresponding vector representation <NUM> for each application install pattern <NUM> that is composed of multiple hashes each associated with a corresponding application embedding <NUM> representing a corresponding application <NUM> installed on the customer device <NUM>. The hashes in vector representation <NUM> are ordered by application install date such that the first hash in the vector representation <NUM> identifies the first application <NUM> in the sequence of n-applications <NUM> installed on the customer device <NUM>.

Referring to <FIG>, in some implementations, the malware identifier <NUM> of the data processing hardware <NUM> executes a domain embedding process <NUM> for identifying malware. <FIG> shows the process <NUM> receiving a corpus <NUM> of training applications <NUM>, 210T, 210a-n. The emulator <NUM> executes each application <NUM> in a secure execution environment and identifies a set of URIs <NUM> that each application <NUM> interacts with during execution of the corresponding application <NUM>. In the example shown, the URIs <NUM> include domain names. In other examples, the URIs <NUM> may include uniform resource locators (URLs) that may translate to respective domain names. The neural network model 170a may include a domain vector space model 170a that receives the identified set of domain names <NUM> (e.g., www. com and www. com) associated with a given application <NUM> as feature inputs and determines an n-dimensional numerical vector representation <NUM> (e.g., domain embedding) for each domain name <NUM>. In some examples, each domain embedding <NUM> is associated with a <NUM>-dimensional numerical vector representation in a free vector space. In the example shown, the domain vector space model 170a outputs a mapping (e.g., hash map) <NUM> between each domain name <NUM> and the corresponding domain embedding <NUM>.

In some examples, each domain embedding <NUM> is represented as a cryptographic hash value that cannot be decrypted back. For instance, each domain embedding <NUM> may include a corresponding cryptographic hash obtained from a secure hash algorithm (SHA) such as the SHA-<NUM> algorithm generating a <NUM>-bit (<NUM>-byte) hash. The domain vector space model 170a may further determine a vector representation <NUM> for each training application 210T by averaging the domain embedding <NUM> for the corresponding domain names <NUM>. The mapping (e.g. hash map) <NUM> between each domain name <NUM> and domain embedding <NUM> and the vector representation 260T for each application 210T may be stored in the data storage <NUM>.

Referring to <FIG>, the domain embedding process <NUM> shows the emulator <NUM> executing a software application (App <NUM>) 210a in a secure execution environment and identifying a plurality of domain names 220a the application 210a interacts with during execution of the application 210a. In the example shown, the emulator <NUM> provides the identified domain names 220a to the domain vector space model 170a and sends an inquiry <NUM> to a results manager <NUM> inquiring whether or not the application 210a belongs to a potentially harmful category <NUM>, 240b and/or includes malware based on the identified domain domains <NUM>. In some examples, the emulator <NUM> provides URLs 220a and the domain vector space model 170a and/or results manager <NUM> translates the received URLs 220a into respective domain names (e.g., www. com and www.

The domain vector space model 170a is configured to receive the identified domain names 220a as feature inputs and determine an n-dimensional numerical vector representation <NUM>, 250a (e.g., domain embedding) for each domain name 220a. The n-dimensional numerical vector representation <NUM> may include a <NUM>-dimensional numerical vector representation within a free vector space. The domain vector space model 170a may store n-dimensional numerical vector representation 250a for each identified domain name 220a in domain embedding storage 180a of the data storage <NUM>. For instance, the domain vector space model 170a may store a respective mapping (e.g., hash map) 204a between each identified domain name 220a and the corresponding n-dimensional numerical vector representation 250a. In some implementations, the domain vector space model 170a generates a corresponding vector representation 260a (e.g., aggregate application embedding) for the application (App <NUM>) 210a by averaging the n-dimensional vector representations for the corresponding identified domain names 220a. In the example shown, the domain vector space model 170a stores the corresponding vector representation 260a for the application 210a in application embedding storage 180b of the data storage <NUM>. Accordingly, the domain vector space model 170a continuously receives applications <NUM> as new training data for use in clustering applications <NUM> for identifying malware. Training applications 210a and corresponding vector representations 260a may be used to update the domain vector space model 170a on a daily basis.

In the example shown, in response to receiving the inquiry <NUM> including the identified domain names 220a (or receiving URIs/URLs for conversion into respective domain names), the results manager <NUM> retrieves the corresponding n-dimensional numerical vector representation 250a for each identified domain name 220a (i.e., using the corresponding mapping 204a) from the domain embedding storage 180a and calculates the corresponding vector representation 260a for the application 210a by averaging all of the n-dimensional numerical vector representations 250a. Optionally, the results manager <NUM> may instead retrieve the corresponding vector representation 260a generated by the domain vector space model 170a from the application embedding storage 180b. At the same time, the results manager <NUM> may retrieve training vector representations 260T for a pool of training applications 210T from the application embedding storage 180b.

In some implementations, the results manager <NUM> determines similarity scores <NUM> for the pool of training applications 210T based on the corresponding vector representation 260a for the application 210a and the training vector representations 260T. Each similarity score <NUM> is associated with a corresponding training application 210T and indicates a level of similarity (e.g., semantically related) between the corresponding vector representation 260a for the application (App <NUM>) 210a and a respective training vector representation 260T for the corresponding training application 210T. In some examples, determining the similarity scores <NUM> includes calculating a respective cosine similarity between the corresponding vector representation 260a for the application (App <NUM>) 210a and the respective training vector representation 260T for each corresponding training application 210T.

In some examples, the inquiry <NUM> associated with the set of identified domain names 220a representing the application (App <NUM>) 210a requests the results manager <NUM> to suggest a category <NUM> associated with the application App <NUM>) 210a. Here, the suggested category <NUM> may indicate that the application 210a belongs to a legitimate application category 240a or a potentially harmful application category 240b based on the similarity scores <NUM> for the pool of training applications 210T. Additionally or alternatively, the suggested category <NUM> may further specify a specific category type associated with the application 210a such as a news application, social media application, or a specific type of malware. Additionally or alternatively, the inquiry <NUM> may further request the results manager <NUM> to return a similarity list <NUM> that includes the training applications 210T associated with the top-k highest similarity scores <NUM>. Here, the top-k highest similarity scores <NUM> may indicate a specified number k of nearest training vector representations 260T to the corresponding vector representation 260a for the software application. In other words, the similarity list <NUM> may include training applications 210T clustered near the application (App <NUM>) 210a.

Referring to <FIG>, the results manager <NUM> generates a response <NUM> that includes similarity list <NUM> indicating all of the training applications 210T associated with the top-k highest similarity scores <NUM>. In the example shown, after determining the similarity scores <NUM>, the results manager <NUM> retrieves the top-k similar training applications 210Ta-Tk from the application embedding storage 180b. Here, the top-k similar training applications 210Ta-210Tk correspond to the training applications 210T associated with the top-k highest similarity scores <NUM>. Additionally, the results manager <NUM> retrieves a list of domain names 220Ta-Tk that each of the top-k similar training applications 210Ta-Tk interact with. Thereafter, the results manager <NUM> generates the similarity list <NUM> to include the retrieved top-k similar training applications 210Ta-Tk and returns the response <NUM> including the similarity list <NUM> to the emulator <NUM>. The inquiry <NUM> and response <NUM> between the results manager <NUM> and the emulator <NUM> may include remote procedure calls (RPCs). In some examples, each top-k similar training application 210Ta-Tk in the similarity list <NUM> includes the corresponding similarity score <NUM> and the corresponding list of domain names 220Ta-Tk. The similarity list <NUM> may additionally or alternatively include a category <NUM> (e.g., a legitimate application category 240a or a potentially harmful application category 240b). In the example shown, the first top-k similar training application (App a) 210Ta includes a corresponding similarity score 262Ta, a corresponding list of domain names 220Ta, and a category <NUM> assigned to the corresponding application 210Ta.

In some examples, training applications 210T within the application embedding storage 180b identified as belonging to potentially harmful application (PHA) categories 240b are labeled with a PHA label. In some configurations, the application embedding storage 180b is partitioned to include a separate embedding storage for training applications 210T including the PHA label. Partitioning of this separate embedding storage may reduce computational load when retrieving training vector representations 260TPHA for all training applications 210TPHA including the PHA label. Additionally or alternatively, one or more of the domain embedding storage 180a, the application embedding storage 180b, or the separate embedding storage including training applications 210T with PHA labels may use threads, cache techniques, and/or hash tables to reduce storage requirements and enhance performance for accessing stored data.

Referring to <FIG>, the results manager <NUM> generates a response <NUM> including a suggested PHA category 240b for the application 210a represented by the set of corresponding domain names 220a included in the inquiry <NUM> (<FIG>). In the example shown, the results manager <NUM> retrieves training vector representations 260TPHA for all training applications 210TPHA including the PHA label from the application embedding storage 180b and identifies which retrieved training applications 210TPHA include corresponding similarity scores <NUM> that satisfy a similarity threshold. In some examples, a similarity score <NUM> satisfies the similarity threshold when the similarity score <NUM> exceeds the similarity threshold. Each similarity score <NUM> may include a respective cosine similarity between the corresponding training vector representation 260TPHA for each training application 210TPHA and the corresponding vector representation 260a for the application (App <NUM>) 210a. For each training application 210TPHA identified as satisfying the similarity threshold, the results manager <NUM> may determine a corresponding PHA category 240b associated with the identified application 210TPHA. In some examples, the results manager <NUM> determines the suggested PHA category 240b for the application 210a based on the identified PHA category 240b associated with a majority of the training applications 210TPHA identified as satisfying the similarity threshold. For instance, the results manager <NUM> may use majority voting to assign the suggested PHA category 240b for inclusion in the response <NUM> returned to the emulator <NUM>. The suggested PHA category 240b may include one of a hostile downloader application, a phishing application, a rooting Trojan application, a spyware application, a ransomware application, a malware application, an escalating privileges application. This list of potentially harmful categories 240b is non-exhaustive and may include any other application that is malicious or otherwise potentially harmful.

Implementations further include the process <NUM> flagging an application <NUM> as belonging to the PHA category 240b when one or more of the training applications 210TPHA have similarity scores <NUM> that satisfy the similarity threshold and include the PHA label. Accordingly, and with reference to <FIG>, when a customer device <NUM> sends a download request <NUM> to download an application <NUM> flagged as belonging to the PHA category 240d from the remote system <NUM>, the malware identifier <NUM> may transmit a warning notification <NUM> to the user device indicating that the requested application <NUM> is flagged as belonging to the PHA category 240b.

<FIG> schematically illustrates an example application install pattern <NUM>. In some implementations, the application install pattern <NUM> includes device information <NUM> indicating a device type for the customer device <NUM> associated with the application install pattern <NUM>. For instance, the device type may specify that the customer device <NUM> is one of a smart phone, tablet, laptop, desktop, or other type of computing device. The device type may also include a make and model of the customer device <NUM>. The device information <NUM> may further include an operating system of the customer device <NUM>. The application install pattern <NUM> further indicates the corresponding sequence of n-applications <NUM>, 210a-n (e.g., Application A, Application B,. , Application N) installed on the customer device <NUM>. Here, the sequence orders the applications <NUM> by install date on the customer device <NUM>. For instance, Application A may correspond to the first application 210a installed on the user device, Application B may correspond to the second application 210b installed on the customer device <NUM>, and Application N may correspond to the last or most recent application 210n installed on the customer device <NUM>. Accordingly, the application install pattern <NUM> may also include the application install date for each of the n-applications <NUM> installed on the customer device <NUM>.

Referring to <FIG>, in some implementations, the malware identifier <NUM> of the data processing hardware <NUM> executes an application embedding process <NUM> for identifying malware. Referring to FIG. 4A, the process <NUM> receives a corpus <NUM> of application install patterns <NUM>, 300A-N from corresponding customer devices <NUM>, 104a-n with each application install pattern <NUM> including a corresponding sequence of n-applications <NUM> installed on the corresponding customer device <NUM>. For instance, application install pattern 300A includes the corresponding sequence of n-applications <NUM>, 210Aa-An. In some examples, the neural network model <NUM> includes an application vector space model 170b that receives each application <NUM> in a sequence of n-applications <NUM> associated with a corresponding application install pattern <NUM> as feature inputs and generates a corresponding numerical vector representation <NUM> (e.g., application embedding <NUM>) for each application <NUM>. While receiving a sequence of n-applications <NUM> belonging to a corresponding application install pattern <NUM>, the neural network model <NUM> may predict the next application <NUM> installed on the customer device <NUM>.

Thereafter, the neural network model 170b clusters each corresponding application <NUM> in a free vector space <NUM> based on the numerical vector representation <NUM> for the corresponding application <NUM>. An application cluster plot <NUM> may map each numerical vector representation <NUM> (e.g., application embedding) to the free vector space <NUM> to semantically embed related applications <NUM> near one another. As the training data of application install patterns <NUM> input to the application vector space model 170b increases, the application vector space model 170b becomes more robust and forms many different groups of clusters where related applications are grouped/clustered together in the free vector space <NUM>. As such, when multiple applications <NUM> in a given cluster are known/learned/identified to include malware or belong to a PHA category (e.g., include a PHA label), new or previously unidentified applications <NUM> grouped into the same given cluster may be identified as also including malware.

Similarly, since the neural network model <NUM> may predict the next application <NUM> installed on the customer device <NUM>, when an actual next application <NUM> in a corresponding application install pattern <NUM> is different that the predicted next application <NUM>, the actual next application <NUM> may correspond to a hostile downloader changing the application install pattern <NUM>. In other words, if an application is installed which does not fit the prediction, a flag may raise to indicate the identification of a possible hostile downloader.

<FIG> shows an example plot <NUM> mapping applications <NUM> in the free vector space <NUM> based on the corresponding numerical vector representations <NUM> output from the application vector space model 170b. In the example shown, the numerical vector representations <NUM> include a SHA-<NUM> digest representation embedded in the free/continuous vector space where semantically similar digests (e.g., embeddings <NUM>) are embedded nearby each other to form clusters <NUM>, 510a-n. Each cluster <NUM> may be used to learn/identify a corresponding application category <NUM>, 240a-b associated with the applications <NUM> of the cluster <NUM>. For instance, cluster 510n may belong to a PHA category 240b (e.g., Block Hostile Downloader) while cluster 510a may belong to a legitimate application category 240a.

The process <NUM> of <FIG> may similarly map applications <NUM> in a similar free vector space <NUM> based on the corresponding vector representations <NUM> in order to form clusters of related applications <NUM> based on the URIs <NUM> (domain names) each application <NUM> interacts with while executing on the emulator <NUM>. Optionally, the domain vector space model 170a may map domain names in a free vector space <NUM> based on the corresponding domain embeddings <NUM> (cryptographic hashes) in order to cluster similar domain names close together. In this manner, the domain vector space model 170a may not only identify malware in applications <NUM>, but may learn to identify domain names that are associated with malware.

Referring back to <FIG>, in some implementations, each application embedding <NUM> is associated with a <NUM>-dimensional numerical vector representation in the free vector space <NUM>. In the example shown, the application vector space model 170b outputs a mapping (e.g., hash map) <NUM> between each application <NUM> and the corresponding application embedding <NUM>. The hash map <NUM> associated with each application install pattern <NUM> may be stored in the data storage <NUM>. In some examples, the application embedding <NUM> is represented as a cryptographic hash value, such as a SHA-<NUM> hash. The application vector space model 170b may further determine a corresponding vector representation <NUM> for each application install pattern <NUM> that is composed of multiple hashes. Each hash in the vector representation <NUM> is associated with a corresponding application embedding <NUM> that represents a corresponding application <NUM> installed on the customer device <NUM>. The hashes <NUM> in the vector representation <NUM> are ordered by application install date such that the first hash in the vector representation <NUM> identifies the first application <NUM> in the sequence of n-applications <NUM> installed on the customer device <NUM>.

While the applications <NUM> of each install pattern <NUM> are ordered in sequence by the application install date, an actual date (e.g., January <NUM>, <NUM>) of when the applications <NUM> were installed on the customer device <NUM> is not included in the install pattern <NUM> provided to the application vector space model 170b. In fact, the applications <NUM> in the sequence of n-applications <NUM> correspond to application digests that do not include package names, install times, or any other information. Since the application digests do not preserve any structure, the application vector space model 170b will generate different application embeddings <NUM> (e.g., hashes) for different versions of the same application <NUM>. However, it is with high probability that different version of a same application will be clustered in close proximity to one another in the free vector space.

In the example shown, the hash map <NUM> associated with a corresponding application install pattern <NUM> includes each of the n-applications <NUM> ordered by install date and the corresponding application embedding <NUM> for each application <NUM>. Each application embedding <NUM> may be a numerical vector representation (e.g., n-dimensional vector representation) or cryptographic hash. A first application 210a includes a Social App, a second application 210b includes a Messaging App, a third application 210c includes a Fake App 210c, the fourth application 210d is an Unknown App, and the last application 210n in the sequence of n-applications <NUM> is a rooting Trojan. The Social App 210a, Messaging App 210b, Fake App 210c, and Rooting Trojan 210d may be previously seen/identified applications <NUM> having assigned categories <NUM> (e.g., social, messaging, fake, rooting). Moreover, the Fake App 210c and the Rooting Trojan 210n may include PHA category labels indicating that the applications 210c, 210n have been previously identified as belonging to the PHA category 240b. On the other hand, the Social App 210a and the messaging App 210b are known applications that have been previously identified as belonging to the legitimate application category 240a. The Unknown App 210d has not been previously seen and the application category <NUM> associated with the Unknown App 210d is therefore unknown.

In some implementations, the process <NUM> determines whether the Unknown App 210d belongs to the legitimate application category 240a or the PHA category 240b based on where the cluster plot <NUM> maps the application embedding <NUM> (cryptographic hash/n-dimensional vector representation) representing the Unknown App 210d in the free vector space <NUM>. For instance, if the process <NUM> groups the application embedding <NUM> for the Unknown App 210d into cluster 510n, the Unknown App 210d may be identified as potentially harmful and labeled as belonging to the PHA category 240b. On the other hand, if the process <NUM> groups the application embedding <NUM> into cluster 510a, the process <NUM> may identify the Unknown App 210d as a legitimate application and label the Unknown App 210d as belonging to the legitimate application category 240a.

In some examples, the application vector space model 170d identifies the Unknown App 210d as malware based on determining that the Fake App 210c and Rooting Trojan 210n were previously identified as malware. In these examples, since the Unknown App 210d is ordered close to the Fake App 210c in the sequence of n-applications <NUM>, there is a very high probability that the Unknown App 210d is also malware. Upon identifying the Unknown App 210d as malware, the malware identifier <NUM> (<FIG>) may label the App 210d as belonging to the PHA category 240b and then store the App 210d with both the PHA category label and the corresponding numerical vector representation <NUM> in the data storage <NUM>. Accordingly, and with reference to <FIG>, the malware identifier <NUM> may transmit a warning notification <NUM> to the user device indicating that the Unknown App 210d is identified as malware (e.g., flagged as belonging to the PHA category 240b. The warning notification <NUM> may further indicate the presence of the Fake App 210c, the Rooting Trojan 210n, and any other applications <NUM> in the sequence of n-applications identified as malware (e.g., belong to the PHA category 240b).

<FIG> is a flow chart of an example arrangement of operations for a method <NUM> of identifying malware. The data processing hardware <NUM> may execute the operations for the method <NUM> by executing instructions stored on the memory hardware <NUM>. At operation <NUM>, the method <NUM> includes receiving a software application <NUM> and executing the software application <NUM>. An emulator <NUM> may execute the software application <NUM> in a secure execution environment. At operation <NUM>, the method includes identifying a plurality of uniform resource identifiers (URIs) <NUM> the software application <NUM> interacts with during execution of the software application <NUM>. The URIs may include uniform resource locators (URLs) such as domain names.

At operation <NUM>, the method <NUM> includes generating a vector representation <NUM> for the software application using a feed-forward neural network (i.e., domain vector space model 170a) configured to receive the plurality of URIs <NUM> as feature inputs. Here, the model 170a determines an n-dimensional numerical vector representation <NUM> for each of the identified URIs <NUM> and calculates the vector representation <NUM> for the software application <NUM> by averaging the n-dimensional numerical vector representations <NUM> for the identified URIs <NUM>.

At operation <NUM>, the method <NUM> includes determining similarity scores <NUM> for a pool of training applications 210T stored in the memory hardware <NUM> (e.g., data storage <NUM>). Each similarity score <NUM> is associated with a corresponding training application 210T and indicates a level of similarity between the vector representation <NUM> for the software application <NUM> and a respective vector representation 260T for the corresponding training application 210T.

At operation <NUM>, the method <NUM> includes flagging the software application <NUM> as belonging to a potentially harmful application (PHA) category 240b when one or more of the training applications 210T have similarity scores <NUM> that satisfy a similarity threshold and include PHA label.

<FIG> is a flow chart of an example arrangement of operations for a method <NUM> of identifying malware. The data processing hardware <NUM> may execute the operations for the method <NUM> by executing instructions stored on the memory hardware <NUM>. At operation <NUM>, the method <NUM> includes receiving an application install pattern <NUM> from a user device <NUM>, the application install pattern <NUM> indicating a sequence of n-applications <NUM> installed on the user device <NUM>. At operation <NUM>, for each application <NUM> in the sequence of n-applications, the method <NUM> includes generating a numerical vector representation <NUM> for the corresponding application <NUM> using a feed-forward neural network (i.e., application vector space model 170b) configured to receive each application <NUM> and order of each application <NUM> as feature inputs. The numerical vector representation <NUM> may include a cryptographic hash. At operation <NUM>, for each application <NUM> in the sequence of n-applications, the method <NUM> includes clustering the corresponding application <NUM> in a free vector space <NUM> based on the numerical vector representation <NUM> for the corresponding application <NUM>.

At operation <NUM>, the method <NUM> includes determining whether any of the applications <NUM> in the sequence of n-applications <NUM> are clustered with training applications identified as malware. At operation <NUM>, the method <NUM> includes, for each application <NUM> clustered with training applications <NUM> identified as malware, identifying the corresponding application <NUM> in the sequence of n-applications as malware.

In some examples, the method <NUM> further includes, for each application in the sequence of n-applications identified as malware: labeling the application <NUM> as belonging to a potentially harmful application category 240b and storing the application <NUM> and the corresponding numerical vector representation <NUM> for the application <NUM> in memory hardware <NUM> (e.g., data storage <NUM>) in communication with the data processing hardware <NUM>.

For example, it may be implemented as a standard server 800a or multiple times in a group of such servers 800a, as a laptop computer 800b, or as part of a rack server system 800c.

Claim 1:
A method (<NUM>) for identifying malicious software, the method (<NUM>) comprising:
receiving, at data processing hardware (<NUM>), a software application (<NUM>);
executing, by the data processing hardware (<NUM>), the software application (<NUM>);
identifying, by the data processing hardware (<NUM>), a plurality of uniform resource identifiers (<NUM>) the software application (<NUM>) interacts with during execution of the software application (<NUM>);
generating, by the data processing hardware (<NUM>), an embedding vector representation (<NUM>) for the software application (<NUM>) using a feed-forward neural network configured to receive the plurality of uniform resource identifiers (<NUM>) as feature inputs;
determining, by the data processing hardware (<NUM>), similarity scores (<NUM>) for a pool of training applications (<NUM>) stored in memory hardware (<NUM>) in communication with the data processing hardware (<NUM>), each similarity score (<NUM>) associated with a corresponding training application (<NUM>) and indicating a level of similarity between the embedding vector representation (<NUM>) for the software application (<NUM>) and a respective embedding vector representation (<NUM>, <NUM>) for the corresponding training application (<NUM>); and
flagging, by the data processing hardware (<NUM>), the software application (<NUM>) as belonging to a potentially harmful application category (<NUM>) when one or more of the training applications (<NUM>) have similarity scores (<NUM>) that satisfy a similarity threshold and comprise a potentially harmful application label.