Learning Molecule Graphs Embedding Using Encoder-Decoder Architecture

An Encoder-Decoder architecture uses two neural networks that work together to learn molecule embedding without any labeled data by transform the molecule graph to an embedding, and then mapping that embedding to a character-based representation of that molecule. An encoder operates as a molecule embedding model to produce a vector of length “n” that reperesents the molecule as a point in an n-dimentional cartesian space. The generated vector is used by a decoder to predict the molecule's character-based representation such as a SMILES, only based on the molecule structure. A loss function is applied to the decoded character-based representation compared to the actual character-based representation of that molecyle, to generate a gradient of the error determined by the loss function which is used to modify weights in the encoder-decoder model during training.

BACKGROUND

Measuring molecule properties and detecting similar molecules play a major role in drug discovery and development. Properties of a first molecule may be known. It may be desirable to identify other molecules that have properties similar to the properties the first molecule. But using a lab to identify molecules similar to known molecules based on some specific criteria is very expensive and time consuming And selecting which properties to measure may also be time consuming and expensive. Depending on the instrument and measurement procedure, there may be inconsistencies in measured data, which may affect the usability of the measured data. Furthermore, because of budgetary and time limitations, it may not be possible to measure selected properties on all eligible molecules.

SUMMARY

An Encoder-Decoder architecture uses two neural networks that work together to transform a molecule graph to a character-based sequence. Mapping molecule graphs to an embedding space is performed using a Graph Neural Networks (GNNs) model as an encoder. The encoder operates as a molecule embedding model to produce a vector that is an n-dimensional representation of a point in the embedding space. The generated vector is used by a decoder to generate the molecule's character-based representation such as a simplified molecular-input line-entry system (SMILES) token or sequence, only based on the molecule 2D structure. A loss function is applied to the character-based representation compared to the generated squence by the decoder to calculate a gradient of the error which is porpageted through the whole network and modifies the molecule embedding model and decoder during training.

DETAILED DESCRIPTION

The functionality can be configured to perform an operation using, for instance, software, hardware, firmware, or the like. For example, the phrase “configured to” can refer to a logic circuit structure of a hardware element that is to implement the associated functionality. The phrase “configured to” can also refer to a logic circuit structure of a hardware element that is to implement the coding design of associated functionality of firmware or software. The term “module” refers to a structural element that can be implemented using any suitable hardware (e.g., a processor, among others), software (e.g., an application, among others), firmware, or any combination of hardware, software, and firmware. The term, “logic” encompasses any functionality for performing a task. For instance, each operation illustrated in the flowcharts corresponds to logic for performing that operation. An operation can be performed using, software, hardware, firmware, or the like. The terms, “component,” “system,” and the like may refer to computer-related entities, hardware, and software in execution, firmware, or combination thereof. A component may be a process running on a processor, an object, an executable, a program, a function, a subroutine, a computer, or a combination of software and hardware. The term, “processor,” may refer to a hardware component, such as a processing unit of a computer system.

Furthermore, the claimed subject matter may be implemented as a method, apparatus, or article of manufacture using standard programming and engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computing device to implement the disclosed subject matter. The term, “article of manufacture,” as used herein is intended to encompass a computer program accessible from any computer-readable storage device or media. Computer-readable storage media can include, but are not limited to, magnetic storage devices, e.g., hard disk, floppy disk, magnetic strips, optical disk, compact disk (CD), digital versatile disk (DVD), smart cards, flash memory devices, among others. In contrast, computer-readable media, i.e., not storage media, may additionally include communication media such as transmission media for wireless signals and the like.

Measuring molecule properties and detecting similar molecules may be important to drug discovery and development. Certain properties of a first molecule may be known. It may be desirable to identify other molecules that have properties similar to the certain properties of the first molecule. For example, a first molecule may be known to be effective for treating HIV, and it may be desirable to identify other molecules that have properties similar to the first molecule because such other molecules may also be effective for treating HIV. But identifying other molecules that have properties similar to the certain properties of the first molecule may be challenging. Identifying similar molecules may involve expensive and time-consuming laboratory work.

Selecting which properties of eligible molecules to measure may also be time consuming and expensive. Depending on the instrument and measurement procedure, there may be inconsistencies in measured data, which may affect the usability of the measured data. Furthermore, because of budgetary and time limitations, it may not be possible to measure selected properties on all the eligible molecules.

An Encoder-Decoder architecture uses two neural networks that work together to transform the molecule graph to a character-based sequence. Mapping molecule graphs to an embedding space is performed using Graph Neural Networks (GNNs). The GNNs operate as a molecule embedding model to produce a vector of size n that is the reperesentation of the molecule in an n-dimensional cartesian space. The generated vector is used by a decoder to generate the molecule's character-based representation such as a simplified molecular-input line-entry system (SMILES) sequence, only based on the molecule 2D structure. A loss function is applied to the character-based representation compared to the generated squence by the decoder to calculate the gradient of the error which is porpageted through the whole network and modifies the molecule embedding model during training.

Once the molecule embedding model is trained (encoder), the resulting embedding space can be used to find similar molecules. Also, the embedding model can be merged with another model to predict different properties of a molecule, the same way a pretrained ResNet, DenseNet, etc is used in computer vision models.

Mapping the molecule to the embedding space may allow efficient comparison of the molecule with another molecule (which may have certain known properties) that has been mapped to the embedding space. Mapping the molecule to the embedding space may also allow efficient predictions regarding whether the molecule will be effective for a particular task or will possess a particular property.

One way the embedding model may facilitate finding molecules with similar properties is through mapping molecules to the embedding space. Once the molecules are mapped to the embedding space, distances between the molecules can be calculated using co-sign or cartesian distance between two points in an n-dimensional cartesian space. When a first molecule is close to a second molecule in the embedding space (which may be referred to as neighboring molecules), the first molecule and the second molecule may have similar properties. Thus, if the first molecule has known properties, subsequent lab testing may focus on molecules that neighbor the first molecule to determine whether those neighboring molecules also have the known properties. Using the approach of identifying close molecules in the embedding space can reduce the search space considerably and consequently reduce the required time and expenses incurred in lab testing.

Another way the embedding model may facilitate identifying molecules with certain properties is through merging the embedding model with another model (such as a task-specific model) to learn how to predict different properties of a molecule (such as predicting whether a given molecule has toxic properties) even with a few labeled data during the training phase. This use of the embedding model may be similar to how pretrained ResNet and DenseNet models are used in connection with computer vision models.

By applying the encoder decoder approach to retrain the molecule embedding model (encoder), learned weights may be used to hot-start the training phase of different downstream prediction tasks, such as predicting binding results for a set of inhibitors of human β-secretase 1, or predicting octanol/water distribution coefficient (logD at pH 7.4). On several public datasets, an improvement in ROC-AUC (receiver operating characteristic area under the curve) classification metric, in classification tasks, and in RMSE (root mean square error) in regression tasks has been observed.

The generic embedding space can be used as a core of other models to improve the accuracy and training time. Also, it can be very helpful when there is insufficient access to labeled training data.

The graph representation of the molecule (which may be referred to as a molecule graph) may include a node (which may be referred to as a vertex) for each atom in the molecule and an edge (which may be referred to as a link) for each bond connecting atoms in the molecule. Each node in the graph and each edge in the graph may have features. The features may convey information regarding the node or the edge. Features of each node in the graph may be based on attributes and characteristics of each corresponding atom, such as atomic number, chirality, charge, etc. Features of each edge in the graph may be based on attributes and characteristics of each corresponding bond, such as bond type, bond direction, etc. The graph representation of the molecule may be based on a 2D structure of the molecule which can be deducted from molecule's SMILES. Other character-based representations may be used in further embodiments, but for convenience, description of various embodiments will reference SMILES representations. A SMILES may be a chemical notation that represents a chemical structure of a molecule in text or character-based way. Other representations may be used in further embodiments. An RDKit library may translate SMILES to molecule structure. The molecule structure generated by RDKit may be converted to a graph data structure that may be consumed by molecule embedding model as an input.

The embedding model may include a node to vector model (an atom embedding model), which may use graph neural networks to map each atom of a molecule to a feature space based on a molecule structure of the molecule. The embedding model may include an aggregation model that generates molecule features based on learned features of atoms in the molecule. The node to vector model may, using graph neural networks, generate embedded atom features (learned features) for each atom in the molecule. The aggregation model may generate embedded molecule features (learned features) for the molecule, based on the learned features of it's atoms. The learned features for the molecule may define a location of the molecule in an embedding space.

The atom embedding model may include an embedding layer and one or more graph neural network (GNN) layers. A GNN may be a type of neural network that operates directly on a graph structure. A GNN may follow a recursive neighborhood aggregation scheme.

The embedding layer may map an atomic number of each atom (which may be represented as a node in an input graph) to a denser feature space, which may help the embedding model learn a more accurate feature space for atoms. The embedding layer may map an atomic number of each node to a vector of a defined size using linear mapping and/or a lookup table. The embedding layer may be a standard way of moving from a discrete set of entities (such as atoms) to a denser space (such as a vector of size n). The vector associated with each atomic number plus other features of the atom may define updated features of the node. The atomic number and the other features of the atom may be input features of the node that represents the atom. The updated features of the node may be based on the input features of the node. The updated features of the node may be a singular representation that has all the information of the input features of the node embedded into it. The input features of the node may be based on attributes and characteristics of the node.

Each GNN layer in the one or more GNN layers may receive a molecule graph and determine embedded atom features for each atom in the molecule graph. The embedded atom features of an atom may convey specific information regarding the atom, its associated bonds, and a neighborhood of the atom. A first GNN layer in the one or more GNN layers may receive the input graph or the updated graph and determine first layer embedded atom features for each atom in the molecule. Each subsequent GNN layer may receive an output graph from a previous GNN layer and determine next layer embedded atom features for each atom based on the output graph. The one or more GNN layers may be customized Graph Isomorphism Network (GIN) layers.

The one or more GNN layers included in the embedding model may use a message-passing framework. At each of the one or more GNN layers, each node in a graph (which may be a molecule graph) may receive a message from each neighboring node. Two nodes may be neighboring nodes if the two nodes are connected by an edge in the graph. A message may be based on node features of a sending node and edge features of an edge connecting the sending node to a receiving node. For example, the one or more GNN layers may construct the message by concatenating the node features of the sending node with the edge features of the edge connecting the sending node to the receiving node.

The one or more GNN layers may use an attention mechanism to prioritize (i.e., weight) messages from neighboring nodes. An attention layer may determine a weight (which may be referred to as an edge weight) to apply to each message. The edge weight for each message may be based on node features of a node sending the message (a sending node) and node features of a node receiving the message (a receiving node). The one or more GNN layers may learn to determine the edge weight for each message based on a correlation between the node features of the sending node and the node features of the receiving node. For example, the one or more GNN layers may determine the edge weight by concatenating the node features of the sending node and the node features of the receiving node, applying a linear layer, and applying a sigmoid activation to the output. By using an attention mechanism, the embedding model may learn how to prioritize different messages sent to a receiving node based on a relationship between features of a sending node and features of the receiving node. Using the attention mechanism and edge weights that are based on features of a sending node and features of a receiving node may improve accuracy of the embedding model when used in connection with performing downstream tasks.

The following expression illustrates one example of how the one or more GNN layers may determine features xi′for a node i in a graph:

where xi′is an output of a GNN layer for node i, (xj+ej,i) (which may be referred to as mj,i) is the message from node j to node i, xjis the features of node j, ej, iis the features of the edge connecting node j to node i, ewj,iis the edge weight for the message from node j to node i (N(i), i, and hθdenotes a neural network.

The following expression illustrates one example of how ewj,imay be determined:

where ewj,iis the edge weight and the attention mechanism, xjis the features of the sending node, xiis the features of the receiving node, Wfis a learned weighting coefficient, bfis a learned bias coefficient, and δ is a non-linearity. Wfmay be learned based on features of two ends of the edge.

As noted above, each of the one or more GNN layers may output embedded atom features for each atom in a molecule graph. The outputted embedded atom features may be referred to as a hidden state for the atom. An attention layer may use the hidden states (or, in a case of an attention layer associated with a first GNN layer, atom features of an input graph or updated graph) to generate edge weights for a GNN layer (which may be referred to as a next GNN layer) subsequent to a GNN layer (which may be referred to as a previous GNN layer) that generated the hidden states. The next GNN layer may receive the hidden states from the previous GNN layer as atom features and may receive the edge weights from the attention layer. The next GNN layer may output new hidden states based on the hidden states and the edge weights. The atom embedding model may include multiple attention layers and GNN layers stacked on top of each other. Each additional layer may provide visibility to further neighbors from any given node.

After generating embedded atom features using a stack of GNN layers, the atom aggregation model may generate a molecule embedding (which may also be referred to as molecule features). The atom aggregation model may generate the molecule embedding based on the embedded atom features. The atom aggregation model may first aggregate embedded atom features generated by each of the one or more GNN layers to generate aggregated atom features for each atom in the molecule graph. The atom aggregation model may then aggregate the aggregated atom features to generate the molecule features. One aggregation strategy may be based on concatenating the embedded atom features generated by each of the one or more GNN layers to generated aggregated atom features and then using an attention pooling layer to prioritize aggregated atom features of different atoms. The attention pooling layer may learn how to prioritize aggregated atom features of different atoms to calculate molecule features such that the embedding model achieves the highest accuracy in all downstream tasks.

Training using the output of a decoder to generate a back propagation gradient of the error may be used to train the embedding model. The training may result in the embedding model being sufficiently generic such that the embedding model may be used as a core in different regression, classification, or clustering models. To achieve this result the encoder-decoder model may be trained on a wide range of molecule graphs and their SMILES. By training the encoder-decoder model on a wide range of molecule graphs, the embedding model (encoder) may generate high quality molecule features and a denser embedding space that captures a wide range of important features of the molecule. Therefore, there is a higher chance that the molecule embedding contains the required information to be used in a variety of tasks. For example, such a generic embedding model may be used as a core of other models to improve accuracy and training time for the other models. Using a generic embedding model trained using the encoder-decoder architecture, as a core, may also be helpful when there are insufficient labeled data for the downstream prediction task. The embedding space itself may also be used to find similar molecules or find molecule clusters that share interesting properties (such as solubility).

FIG. 1illustrates an encoder-decoder system100. The system100may include a graph102, an embedding model108, and a decoder114.

The graph102may be a data structure. The graph102may contain information regarding real-world entities and relationships between the real-world entities. As one example, the graph102may represent a molecule and contain information regarding atoms that form the molecule and regarding bonds between and among the atoms of the molecule. In the case of a molecule, the graph102may be based in part on a SMILES of the molecule. As another example, the graph102may represent a social network, a biological system, or a financial system.

The graph102may include nodes104(which may also be referred to as vertices) and edges106(which may also be referred to as links).

The nodes104may represent component entities that make up the graph102. The nodes104may have features. The features may contain information regarding properties of the nodes104. For example, consider that the graph102represents a molecule and the nodes104represent atoms within the molecule. The atoms within the molecule may have certain properties such as atomic numbers and chirality. The features of the nodes104may include the properties of the atoms. The features of the nodes104may be based on the properties of the atoms. For example, the features of the nodes104may be determined using one-hot encoding and/or linear mapping based on the properties of the atoms. The features of the nodes104may be represented in a vector.

The edges106may represent relationships between pairs of nodes. The edges106may be directional or non-directional. The edges106may have features that contain information regarding the relationships between the pairs of nodes. For example, in the situation in which the graph102represents a molecule, the edges106may represent bonds between atoms within the molecule. The bonds between the atoms within the molecule may have certain properties, such as bond type and bond direction. The features of the edges106may include the properties of the bonds. The features of the edges106may be based on the properties of the edges106. For example, the features of the edges106may be generated based on the properties of the bonds. The features of the edges106may be represented in a vector.

The embedding model108may include a machine learning model that receives a graph (such as the graph102) and outputs a representation of the graph in an embedding space. The embedding space may be a Euclidean space. The embedding space may be any space in which a point in an n-dimensional embedding space can be defined using a vector of size n. The embedding space may have a defined number of dimensions. Each point in the embedding space may be defined by certain values for each dimension. The representation of the graph in the embedding space may be a vector having a same number of dimensions as the embedding space. The embedding space may be denser than a space in which the graph exists. For example, the graph may represent a molecule. The molecule may exist in a space of all molecules. The embedding model108may output a representation of the molecule in an embedding space. The representation of the molecule in the embedding space may be molecule features of the molecule. The embedding space may be denser than the space of all molecules.

The embedding model108may include a node embedding model110and a node aggregation model112.

The node embedding model110may include one or more GNN layers. Each of the one or more GNN layers may receive an input graph and output an embedded graph (which may be a hidden state). At each of the one or more GNN layers, each node in the input graph may have a corresponding node in the embedded graph. Each node in the input graph may have input features. Each corresponding node in the embedded graph may have embedded features. Embedded features of an output node in an embedded graph (which may correspond to an input node in an input graph) may contain more information about the output node than is contained in input features of the input node. Each of the one or more GNN layers may learn to take the input features (which may have no correlation or an unknown correlation) and neighborhood information and map the input features and the neighborhood information to a singular representation (embedded features) that has all that information embedded into it. The one or more GNN layers may learn to determine the embedded features to achieve a highest accuracy on all downstream tasks. Each of the one or more GNN layers may access structure information contained in the input graph in determining the embedding features.

At least one of the one or more GNN layers may use a message-passing framework and an attention mechanism to determine, based on an input graph, embedded features for an embedded graph. Each node in the input graph may receive a message from each neighboring node in the input graph. A neighboring node of a node may be any node connected to the node by an edge. A message from a neighboring node to a receiving node may be based on features of the neighboring node and features of an edge connecting the neighboring node to the receiving node. A GNN layer may use messages received by a receiving node from neighboring nodes to determine embedded features of the receiving node.

A GNN layer may use the attention mechanism to weight each of the messages received by the receiving node in determining the embedded features. The GNN layer may receive weights for each of the messages from an attention layer. The attention layer may, for each message, determine a weight based on features of a node in the input graph that is sending the message and features of a node in the input graph that is receiving the message. The weights may communicate to the GNN layer which neighboring node's information is most important. The attention layer may learn how to put weights on the messages. The attention layer may learn how to put weights on the messages based on a correlation of features of a receiving node and features of a sending node. Utilizing weights determined based on features of a receiving node and features of a sending node, may increase in order to determine embedded features may increase an accuracy of the embedding model108in connection with performing downstream tasks. These weights may also be used to investigate and identify portions of a molecule structure that were more important during the inference.

The node aggregation model112may determine molecule features for an input graph (such as the graph102) based on embedded graphs generated by the one or more GNN layers. The molecule features may define a point in an embedding space of the input graph. The node aggregation model112may determine aggregated node features for each node in the input graph. The aggregated node features for a node may be based on embedded features of the node in the embedded graphs. For example, the node aggregation model112may determine the aggregated node features by determining an average of the embedded features of the node in the embedded graphs.

The node aggregation model112may determine the molecule features based on the aggregated node features of the nodes. The node aggregation model112may prioritize aggregated node features of some nodes of the input graph over other nodes of the input graph. The node aggregation model112may determine a weight to apply to aggregated node features of each node in the input graph in determining the molecule features. The node aggregation model112may learn to determine weights to apply to aggregated node features to achieve a highest accuracy on downstream tasks.

A decoder114may receive an output of the embedding model108. The output of the embedding model108may be a vector representative of the molecule features. The generated vector from the molecule embedding model is the encoder output that is used in the decoder to predict the SMILES sequence. The decoder in one embodiment is an attention based RNN (recurrent neural network), where in each iteration the next token in the smiles sequence is predicted based on the next value in the encoder output, the attention output, and the previous hidden state of the RNN. When the RNN predicts the end token, that will be the end of that SMILES sequence. A loss function is used during training to calculate the gradient that should be propagated through the network which contains the embedding model108.

Attention allows the decoder network to focus on a different part of the encoder's outputs for every step of the decoder's outputs. First, we calculate a set of attention weights. These will be multiplied by the encoder output vectors to create a weighted combination. The result should contain information about that specific part of the input sequence, and thus help the decoder choose the right output SMILES token.

The loss function in this model is based on the cross entropy of the decoder output and the tokenized SMILES at any given point of the SMILES sequence.

The output of the embedding model108may be used to map the input graph to a point in the embedding space. The embedding space may allow for determining a distance between the input graph and other molecules mapped to the embedding space. Molecules that are within a threshold distance in the embedding space may have similar properties.

FIG. 2illustrates an example graph202. The graph202may represent a molecule. The graph202may be an input to an embedding model (such as the embedding model108), an input to an embedding layer, an output of an embedding layer, a hidden state within an embedding model, or an output of a node embedding model (such as the node embedding model110).

The graph202may include nodes204a,204b,204c,204d,204e,204f,204g,204h,204i,204j,204k,204l,204m,204n,204o, and204p, referred to as204a-p. In other designs, the graph202may include fewer or more nodes. Each of the nodes204a-pmay represent an atom in a molecule. The nodes204a-pmay include features216a,216b,216c,216d,216e,216f,216g,216h,216i,216j,216k,216l,216m,216n,216o, and216p, referred to as216a-p. The features216a-pmay be based on properties of atoms represented by the nodes204a-p. For example, the node204amay represent a first atom in a molecule. The first atom may have an atomic number, a chirality, and a charge. The features216amay be based on the atomic number, the chirality, and the charge of the first atom. The features216a-pmay be represented in vectors. The features216a-pmay be embedded features.

The graph202may include edges206ab,206bc,206be,206cd,206eg,206af,206fg,206fh,206ai,206ij,206jk,206jl,206jm,206jn,206mn,206ao,206op(which may be referred to as edges206ab-op). The edges206ab-opmay represent bonds in the molecule. Each of the edges206ab-opmay include edge features. The edge features may be based on properties of the bonds represented by the edges206ab-op. For example, the edge206abmay represent a first bond in a molecule. The first bond may have a bond type and a bond direction. Edge features of the edge206abmay be based on the bond type and the bond direction. The edge features may be represented in vectors.

In situations in which the graph202is a hidden state within an embedding model, the features216a-pmay be based on more than properties of the atoms that the nodes204a-prepresent. Consider an example in which the graph202is a hidden state (an output) of a first graph neural network layer in an embedding model. Assume that the first graph neural network layer receives an input graph. The features216aof the node204amay be based not only on properties of an atom that the node204arepresents but may also be based on features of neighboring nodes (which, if temporarily viewing the graph202as the input graph, would be the features216bof the node204b, the features216fof the node204f, the features216iof the node204i, and the features216oof the node204o). The features216aof the node204amay further be based on edge properties of edges that connect the node204ato its neighboring nodes (which, if temporarily viewing the graph202as the input graph, would be the edge206ab, the edge206af, the edge206ai, and the edge206ao). In a situation in which the first graph neural network layer utilizes an attention mechanism, the features216amay be based on edge weights. The edge weights may be based on features of the neighboring nodes of the node204ain the input graph and the features216ain the input graph.

Consider another example in which the graph202is a hidden state (an output) of a second graph neural network layer that is subsequent to the first graph neural network layer of the example above. In such an example, the features216aof the node204amay be further based not only on features of neighboring nodes of the node204abut also on features of nodes that neighbor the neighboring nodes of the node204a(which, if temporarily viewing the graph202as an output from the first graph neural network layer, would be the features216cof the node204c, the features216eof the node204e, the features216gof the node204g, the features216hof the node204h, the features216jof the node204j, and the features216pof the node204p). The features216aof the node204amay further be based on edge features (which, if temporarily viewing the graph202as the output from the first graph neural network layer, would be the edge206bc, the edge206be, the edge206fg, the edge206fh, the edge206ij, and the edge206op). In a situation in which the second graph neural network layer utilizes an attention mechanism, the features216amay be based on edge weights. The edge weights may be based on features of the neighboring nodes of the node204ain the output from the first graph neural network layer and the features216ain the output from the first graph neural network layer.

FIG. 3Aillustrates a node embedding model310. The node embedding model310may receive a graph302. The graph302may represent a molecule. The graph302may be the graph102or the graph202.

The node embedding model310may include attention layers318a-dand GNN layers320a-d. The GNN layers320a-dmay determine hidden states324a-d, and the attention layers318a-dmay determine weights322a-d. Although the node embedding model310includes four GNN layers, in other designs, a node embedding model may include fewer GNN layers (such as a single GNN layer) or more GNN layers. Although the node embedding model310includes an attention layer for each GNN layer, in other designs, one or more GNN layers may not have an associated attention layer. For example, a node embedding model may include a first GNN layer and a second GNN layer. The first GNN layer may not have an associated attention layer while the second GNN layer may have an associated attention layer.

The GNN layer320amay receive an input graph. The input graph may be the graph302or a modified version of the graph302. For example, the node embedding model310may use a mapping layer to map atomic numbers to a dense feature space and replace the atomic number in each node with generated features. Each node in the input graph may receive a message from each neighboring node. A node that receives a message may be referred to as a receiving node and a node that sends the message may be referred to as a sending node. The message may include features of the sending node and features of an edge connecting the sending node and the receiving node. The features of the edge connecting the sending node and the receiving node may be different from features of an edge connecting the receiving node to the sending node. In other words, edges of the input graph may be directional.

The attention layer318amay receive the graph302or a modified version of the graph (or a subset of the foregoing). The attention layer318amay output the weights322ato the GNN layer320a. The weights322amay include a weight for each message sent by a sending node to a receiving node. The attention layer318amay determine the weights322abased on features of the sending node and features of the receiving node. For example, the attention layer318amay determine the weights322abased in part on concatenating the features of the sending node and the features of the receiving node. The attention layer318amay learn how to determine the weights322abased on a relationship between features of a sending node and features of a receiving node. For example, the attention layer318amay learn a weighting coefficient and a bias coefficient for determining the weights322a. The attention layer318amay apply the weighting coefficient to a concatenation of the features of the sending node and the features of the receiving node. The attention layer318amay concatenate the bias coefficient to a result of the foregoing calculation. The attention layer318amay then apply a sigmoid.

The GNN layer320amay determine the hidden state324afor the input graph. The hidden state324amay be a graph identical to the input graph in terms of its structure, except that nodes of the hidden state324amay have features different from input features of nodes in the input graph. The features of a node of the hidden state324amay be referred to as embedded features of the node or a hidden state of the node. The GNN layer320amay determine embedded features for each node in the hidden state324a. The embedded features for each node in the hidden state324amay be based on messages received by the node, weights associated with the messages received by the node (which may be contained in the weights322a), and input features of the node in the input graph. The GNN layer320amay learn how to determine the embedded features for each node in the hidden state324asuch that one or more downstream tasks may be performed with a lowest error. Edges of the hidden state324amay have edge features identical to edges of the input graph.

The GNN layer320bmay receive the hidden state324a. Each node in the hidden state324amay receive a message from each neighboring node. The message may include features of the sending node and features of an edge connecting the sending node and the receiving node. The features of the sending node may give the receiving node visibility to features of nodes that neighbor the sending node.

The attention layer318bmay receive the hidden state324aor a subset of the hidden state324a. The attention layer318bmay output the weights322bto the GNN layer320b. The weights322bmay include a weight for each message sent by a sending node to a receiving node. The attention layer318bmay determine the weights322bbased on features of the sending node and features of the receiving node. For example, the attention layer318bmay determine the weights322bbased in part on concatenating the features of the sending node and the features of the receiving node. The attention layer318bmay learn how to determine the weights322bbased on a relationship between features of a sending node and features of a receiving node.

The GNN layer320bmay determine the hidden state324bfor the hidden state324a. The hidden state324bmay be a graph identical to the hidden state324astructurally, except that nodes of the hidden state324bmay have features different from features of nodes of the hidden state324a. The features of a node of the hidden state324bmay be referred to as embedded features of the node or a hidden state of the node. The GNN layer320bmay determine the embedded features for each node in the hidden state324b. The embedded features for each node in the hidden state324bmay be based on messages received by the node, weights associated with the messages received by the node (which may be contained in the weights322b), and features of the node in the hidden state324a. The GNN layer320bmay learn how to determine the embedded features for each node in the hidden state32basuch that one or more downstream tasks may be predicted with the lowest error. Edges of the hidden state324bmay have edge features identical to edges of the hidden state324a.

The GNN layer320cmay receive the hidden state324b. Each node in the hidden state324bmay receive a message from each neighboring node. The message may include features of the sending node and features of an edge connecting the sending node and the receiving node. The features of the sending node may give the receiving node visibility to features of nodes that neighbor neighbors of the sending node (two hops information).

The attention layer318cmay receive the hidden state324bor a subset of the hidden state324b. The attention layer318cmay output the weights322cto the GNN layer320c. The weights322cmay include a weight for each message sent by a sending node to a receiving node. The attention layer318cmay determine the weights322cbased on features of the sending node and the receiving node. For example, the attention layer318cmay determine the weights322cbased on concatenating the features of the sending node and the features of the receiving node. The attention layer318cmay learn how to determine the weights322cbased on a relationship between features of a sending node and features of a receiving node. The attention layer318cmay learn how to determine the weights322cin a same way as the attention layer318amay learn to determine the weights322a.

The GNN layer320cmay determine the hidden state324cfor the hidden state324b. The hidden state324cmay be a graph identical to the hidden state324bstructurally, except that nodes of the hidden state324cmay have features different from features of nodes of the hidden state324b. The features of a node of the hidden state324cmay be referred to as embedded features of the node or a hidden state of the node. The GNN layer320cmay determine the embedded features for each node in the hidden state324c. The embedded features for each node in the hidden state324cmay be based on messages received by the node, weights associated with the messages received by the node (which may be contained in the weights322c), and features of the node in the hidden state324b. The GNN layer320cmay learn how to determine the embedded features for each node in the hidden state324csuch that one or more downstream tasks may be predicted with the lowest error. Edges of the hidden state324cmay have edge features identical to edges of the hidden state324b.

The GNN layer320dmay receive the hidden state324c. Each node in the hidden state324cmay receive a message from each neighboring node. The message may include features of the sending node and features of an edge connecting the sending node and the receiving node. The features of the sending node may give the receiving node visibility to features of nodes that neighbor neighbors of neighbors of the sending node (three hops visibility).

The attention layer318dmay receive the hidden state324cor a subset of the hidden state324c. The attention layer318dmay output the weights322dto the GNN layer320d. The weights322dmay include a weight for each message sent by a sending node to a receiving node. The attention layer318dmay determine the weights322dbased on features of the sending node and features of the receiving node. For example, the attention layer318dmay determine the weights322dbased on concatenating the features of the sending node and the features of the receiving node. The attention layer318dmay learn how to determine the weights322dbased on a relationship between features of a sending node and features of a receiving node.

The embedded features for nodes included in the hidden states324a-dmay have a same size or different sizes.

FIG. 3Billustrates a receiving node and four sending nodes that may exist in the graph302, a graph input into the GNN layer320a, or the hidden states304a-c.

A node304amay include features316a.

The node304amay receive a message334bafrom node304b. The node304bmay include features316b. Edge306bamay include features332-1. The message334bamay be based on the features316band the features332-1.

The node304amay receive a message334cafrom node304c. The node304cmay include features316c. Edge306camay include features332-2. The message334camay be based on the features316cand the features332-2.

The node304amay receive a message334dafrom node304d. The node304dmay include features316d. Edge306damay include features332-3. The message334damay be based on the features316dand the features332-3.

The node304amay receive a message334eafrom node304e. The node304emay include features316e. Edge306eamay include features332-4. The message334eamay be based on the features316eand the features332-4.

Assume the node304areceives the messages334ba,334ca,334da,334eawithin the GNN layer320bshown inFIG. 3A. The node304amay apply a weight to each of the messages334ba,334ca,334da,334ea. The node304amay apply a weight to each of the messages334ba,334ca,334da,334eabased on the weights322b. The weights322bmay include a weight for each of the messages334ba,334ca,334da,334ea. For example, the weights322bmay include a first weight for the message334ba, a second weight for the message334ca, a third weight for the message334da, and a fourth weight for the message334ea.

The attention layer318bmay determine the weights322b. The attention layer318bmay determine the first weight for the message334babased on the features316band the features316a. The attention layer318bmay determine the second weight for the message334cabased on the features316cand the features316a. The attention layer318bmay determine the third weight for the message334dabased on the features316dand the features316a. The attention layer318bmay determine the fourth weight for the message334eabased on the features316eand the features316a. The first weight, the second weight, the third weight, and the fourth weight may be further based on a weighting coefficient and a bias coefficient. The attention layer318bmay learn the weighting coefficient and the bias coefficient.

Continuing with this example, the GNN layer320bmay determine embedded features for the node304abased on the messages334ba,334ca,334da,334ea, the first weight, the second weight, the third weight, the fourth weight, and the features316a. For example, the message334bamay be a concatenation of the features332-1and the features316b. The message334camay be a concatenation of the features332-2and the features316c. The message334damay be a concatenation of the features332-3and the features316d. The message334eamay be a concatenation of the features332-4and the features316e. The GNN layer320bmay apply the first weight to the message334bato generate a weighted first message. The GNN layer320bmay apply the second weight to the message334cato generate a weighted second message. The GNN layer320bmay apply the third weight to the message334dato generate a weighted third message. The GNN layer320bmay apply the fourth weight to the message334eato generate a weighted fourth message. The GNN layer320bmay sum the weighted first message, the weighted second message, the weighted third message, and the weighted fourth message to generate a message sum. The GNN layer320bmay concatenate the message sum and the features316ato generate intermediate features. The GNN layer320bmay determine the hidden state for the node304abased on the intermediate features. The GNN layer320bmay learn to determine the hidden state for the node304abased on the intermediate features in order to achieve the lowest error on one or more downstream tasks. Utilizing the first weight, the second weight, the third weight, and the fourth weight may improve the ability of the GNN layer320bto capture different properties of a molecule (and an embedding model that includes the GNN layer320b). These weights may also make the node embedding model310more transparent and explainable because the weights may make it possible to see which part of a molecule structure played a more important role during the inference.

FIG. 4illustrates a node aggregation model412. The node aggregation model412may include node aggregation428, graph aggregation430, and an attention pooling layer426.

Node aggregation428may aggregate embedded features of each node in a graph to generate aggregated node features for each node in the graph. The aggregated node features for each node in the graph may represent aggregated atom features when the graph represents a molecule. Consider the node embedding model310. The node aggregation428may, for each node in the graph302, aggregate embedded features for the node contained in the hidden states324a-dto generate aggregated node features for the graph302. The node aggregation428may apply any of a variety of aggregation policies possible for set-to-one mapping in order to determine the aggregated node features.

Consider a first node in the graph has first embedded features in the hidden state324a, second embedded features in the hidden state324b, third embedded features in the hidden state324c, and fourth embedded features in the hidden state324d. One aggregation policy may involve the node aggregation428concatenating the first embedded features, the second embedded features, the third embedded features, and the fourth embedded features to determine aggregated node features (which may also be referred to as final node features) for the node. As another example, the node aggregation428may select embedded features contained in one of the hidden states324a-d(such as the fourth embedded features for the node in the hidden state324d) as the final node features for the node. As another example, the node aggregation428may calculate a mean or a sum of the first embedded features, the second embedded features, the third embedded features, and the fourth embedded features.

As another example, the node aggregation428may determine a maximum value of each axis in the first embedded features, the second embedded features, the third embedded features, and the fourth embedded features. Assume that the first embedded features, the second embedded features, the third embedded features, and the fourth embedded features are each vector having n dimensions. For each dimension in the first embedded features, the second embedded features, the third embedded features, and the fourth embedded features, the node aggregation428may choose a maximum value among the first embedded features, the second embedded features, the third embedded features, and the fourth embedded features. The maximum value for each dimension is used to form the aggregated node features of the node.

Graph aggregation430may aggregate the aggregated node features determined by the node aggregation428to determine graph features for a graph. The graph features may be molecule features when the graph represents a molecule. The graph features may define a location of the graph in an embedding space. The graph aggregation430may apply any of a variety of aggregation policies to determine the graph features. For example, the graph aggregation430may apply any of the policies described above with respect to aggregating embedded features for a node.

The graph aggregation430may utilize an attention pooling layer426to determine the graph features. The attention pooling layer426may learn how to weight aggregated node features of nodes in a graph such that the graph aggregation430determines graph features that allow an embedding model to achieve the lowest error in downstream tasks. For example, consider a graph that includes a first node and a second node. Assume the first node has first aggregated node features and the second node has second aggregated node features. The attention pooling layer426may determine a first weight to apply to the first aggregated node features and a second weight to apply to the second aggregated node features. The first weight may be different from the second weight.

FIG. 5illustrates a decoder500that is used to produce a text based representation of a molecule from the output vector of the encoder. The vector encodes contextual and relational information from the molecule graph. The output vector from the encoder is used as an input context to the decoder.

At every step of decoding, the decoder500is given an input token505and a hidden state510and encoder output512as the context. The initial input token505may be a start-of-string <SOS> (which represents a start of the sentence), and the first hidden state510is initialized to a zero tensor. For the subsequent steps, the previous decoder hidden state will be used as hidden state510and the next value in the encoder output will he used as input505.

An attention515layer allows the decoder network to “focus” on a different part of the encoder's outputs for every step of the decoder's own outputs. A set of attention weights520is first calculated. The attention weights520will be multiplied at525by the encoder output vectors530to create a weighted combination535. The weighted combination535contains information about that specific part of the input, and thus help the decoder500choose the right output words540on every step of decoding (each decoded token).

Calculating the attention weights520may done with a feed-forward layer545, using the decoder's input505and hidden state510as inputs. Because there are molecule graphs of all sizes in the training data, to train the feed-forward layer, all the tokenized SMILES (character based representation of the molecule) are padded to a specific length. The output of the decoder should be the same length as well.

The decoder500also identifies a gradient of the error between the decoded output and a label for the molecule via a loss function for backpropagation during training of the encoder-decoder model. The decoder500is an attention based RNN (recurrent neural network) in one embodiment, where each iteration predicts a next token540in a SMILES sequence based on the next value in the encoder output, the attention output, and the previous hidden state of the RNN. When the RNN predict the end token, that will be the end of the predicted SMILES sequence.

The loss function in one embodiment is based on the cross entropy of the decoder500output and the tokenized SMILES at any given point of the SMILES sequence. In further embodiments a Connectionist temporal classification function (CTC Loss) may be used as the loss function. The encoder-decoder architecture100provides a way to train the molecule encoder without labeled data (unsupervised training).

FIG. 6illustrates an example computer implemented method600of determining embedded features of the encoder model. Method600may include receiving at operation602an edge weight for a message sent from a second node of a graph to a first node of the graph, wherein an edge connects the second node to the first node, the first node comprises first features, the second node comprises second features, the edge comprises edge features, the message includes the edge features, and the edge weight is based on the first features and the second features. The edge weight may be further based on a learned weighting coefficient. The graph may represent a molecule. The graph may be based on a SMILES of the molecule. A graph neural network may receive the edge weight. The graph neural network may be a graph isomorphism network.

The method600may include receiving at operation604a second edge weight for a second message sent from a third node of the graph to the first node of the graph, wherein a second edge connects the third node to the first node, the third node comprises third features, the second edge comprises second edge features, the second message includes the second edge features, and the second edge weight is based on the first features and the third features. The graph neural network may receive the second edge weight. The second edge weight may be further based on the learned weighting coefficient.

The method600may include determining at operation606embedded features of the first node, wherein the embedded features of the first node are based on the message, the edge weight, the second message, and the second edge weight. The graph neural network may determine the embedded features of the first node.

FIG. 7illustrates an example method700of receiving a graph of a molecule and identifying one or more graphs within a threshold difference in an molecule embedding space.

Method700may include receiving at operation702a graph, wherein the graph comprises nodes and edges, each of the nodes comprises node features, and each of the edges comprises edge features. The graph may represent a molecule. The graph may be based on a simplified molecular-input line-entry system (SMILES) of the molecule.

The method700may include determining at operation704two or more embedded features for the nodes as described with respect toFIG. 6, wherein embedded features for a node are based on messages received by the node from one or more neighboring nodes and edge weights associated with the messages, wherein each message comprises edge features of an edge connecting a neighboring node to the node and node features of the neighboring node, and wherein each edge weight is based on the node features of the neighboring node and node features of the node. Two or more graph neural network layers may determine the two or more embedded features for the nodes.

The method may include mapping at operation712the graph features to an embedding space.

The method may include identifying at operation714one or more graphs within a threshold distance of the graph in the embedding space.

FIG. 8illustrates an example computer implemented method800.

The method800may include a receiving operation802to receive examples from a training data batch, wherein the examples from the training data batch comprises molecule graphs that represent various molecules. An embedding model may receive the examples. The embedding model may include one or more graph neural network layers and one or more attention layers. The graph may include nodes and edges. The one or more graph neural network layers may use a message-passing framework. The one or more attention layers may determine edge weights to be applied to messages received by a receiving node in the graph from one or more sending nodes in the graph based on how the message-passing framework propagates information in the graph. The edge weights may be based on features of the receiving node and the one or more sending nodes and on a weighting coefficient.

The method800may include an outputting operation804molecule features for each example received from the training data batch, wherein the molecule features map to an embedding space. The embedding model may output the molecule features. The molecule features may be based in part on the edge weights and the messages.

The method800may include a decoding operation806the molecule features to predict a character-based representation. An attention based recurrent neural network decoder may be used in one embodiment. At operation808, a loss function may be applied to the output of the decoder and a character-based representation of the input molecule graph to calculate a loss. At operation810, the loss is backpropagated through the encoder-decoder model. Learnable weights of the embedding model (encoder) are changed based on the back propagation at operation812. The one or more attention layers may learn how to prioritize different messages based on the back propagation.

Reference is now made toFIG. 9. One or more computing devices900can be used to implement at least some aspects of the techniques disclosed herein.FIG. 9illustrates certain components that can be included within a computing device900.

The computing device900includes a processor901and memory903in electronic communication with the processor901. Instructions905and data907can be stored in the memory903. The instructions905can be executable by the processor901to implement some or all of the methods, steps, operations, actions, or other functionality that is disclosed herein. Executing the instructions905can involve the use of the data907that is stored in the memory903. Unless otherwise specified, any of the various examples of modules and components described herein can be implemented, partially or wholly, as instructions905stored in memory903and executed by the processor901. Any of the various examples of data described herein can be among the data907that is stored in memory903and used during execution of the instructions905by the processor901.

Although just a single processor901is shown in the computing device900ofFIG. 9, in an alternative configuration, a combination of processors (e.g., an Advanced RISC (Reduced Instruction Set Computer) Machine (ARM) and a digital signal processor (DSP)) could be used.

The computing device900can also include one or more communication interfaces909for communicating with other electronic devices. The communication interface(s)909can be based on wired communication technology, wireless communication technology, or both. Some examples of communication interfaces909include a Universal Serial Bus (USB), an Ethernet adapter, a wireless adapter that operates in accordance with an Institute of Electrical and Electronics Engineers (IEEE) 802.11 wireless communication protocol, a Bluetooth® wireless communication adapter, and an infrared (IR) communication port.

The computing device900can also include one or more input devices911and one or more output devices913. Some examples of input devices911include a keyboard, mouse, microphone, remote control device, button, joystick, trackball, touchpad, and lightpen. One specific type of output device913that is typically included in a computing device900is a display device915. Display devices915used with embodiments disclosed herein can utilize any suitable image projection technology, such as liquid crystal display (LCD), light-emitting diode (LED), gas plasma, electroluminescence, wearable display, or the like. A display controller917can also be provided, for converting data907stored in the memory903into text, graphics, and/or moving images (as appropriate) shown on the display device915. The computing device900can also include other types of output devices913, such as a speaker, a printer, etc.

The various components of the computing device900can be coupled together by one or more buses, which can include a power bus, a control signal bus, a status signal bus, a data bus, etc. For the sake of clarity, the various buses are illustrated inFIG. 9as a bus system919.

The techniques disclosed herein can be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a specific manner. Any features described as modules, components, or the like can also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques can be realized at least in part by a non-transitory computer-readable medium having computer-executable instructions stored thereon that, when executed by at least one processor, perform some or all of the steps, operations, actions, or other functionality disclosed herein. The instructions can be organized into routines, programs, objects, components, data structures, etc., which can perform particular tasks and/or implement particular data types, and which can be combined or distributed as desired in various embodiments.

The term “processor” can refer to a general purpose single- or multi-chip microprocessor (e.g., an Advanced RISC (Reduced Instruction Set Computer) Machine (ARM)), a special purpose microprocessor (e.g., a digital signal processor (DSP)), a microcontroller, a programmable gate array, or the like. A processor can be a central processing unit (CPU). In some embodiments, a combination of processors (e.g., an ARM and DSP) could be used to implement some or all of the techniques disclosed herein.

The term “memory” can refer to any electronic component capable of storing electronic information. For example, memory may be embodied as random access memory (RAM), read-only memory (ROM), magnetic disk storage media, optical storage media, flash memory devices in RAM, various types of storage class memory, on-board memory included with a processor, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) memory, registers, and so forth, including combinations thereof.

The terms computer-readable medium, machine readable medium, and storage device do not include carrier waves or signals to the extent carrier waves and signals are deemed too transitory.

The steps, operations, and/or actions of the methods described herein may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps, operations, and/or actions is required for proper functioning of the method that is being described, the order and/or use of specific steps, operations, and/or actions may be modified without departing from the scope of the claims.

The term “determining” (and grammatical variants thereof) can encompass a wide variety of actions. For example, “determining” can include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining and the like. Also, “determining” can include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like. Also, “determining” can include resolving, selecting, choosing, establishing and the like.

Examples

1. A computer implemented method includes receiving training data comprising molecule graphs and for each molecule graph, mapping nodes of the graph via a molecule embedding model encoder to an embedding space to generate a node embedding, aggregating the node embeddings to generate molecule graph embedding vector, decoding the molecule embedding vector, via an attention based recurrent neural network decoder to generate an output including a character-based representation of the molecule, and using backpropagation of a gradient of an error from a loss function applied to the output of the decoder and a character-based representation of the input molecule graph, to modify weights of the molecule embedding model encoder and decoder for mapping of the molecule graph to the embedding space.

2. The method of example 1 wherein the loss function comprises a cross entropy of the output character-based representation and a tokenized representation of the molecule.

3. The method of example 1 wherein the loss function comprises a Connectionist temporal classification function of the output character-based representation and a tokenized representation of the molecule.

4. The method of any of examples 1-3 and further comprising initializing attention weights of the of the attention based recurrent neural network decoder.

5. The method of example 4 wherein the attention weights are initialized using a feed forward layer based on an input token and the molecule embedding vector.

6. The method of example 5 and further comprising setting a maximum decoder input length for the molecule embedding vector.

7. The method of any of examples 1-6 and further including determining multiple molecule embedding vectors for multiple molecule graphs using the embedding model having modified weights and identifying molecule graphs within a threshold distance of each other in the molecule embedding space.

8. The method of any of examples 1-7 wherein each molecule graph includes nodes and edges, wherein the embedding model comprises one or more graph neural network layers that use a message-passing framework, wherein one or more attention layers of the embedding model determine edge weights to be applied to messages received by a receiving node in the graph from one or more sending nodes in the graph, and wherein the molecule features are based in part on the edge weights and the messages.

9. The method of example 8, wherein the edge weights are based on features of the receiving node and the one or more sending nodes.

10. The method of example 9, wherein the edge weights are further based on a weighting coefficient and the one or more attention layers learn how to prioritize different messages based on the back propagation.

11. A machine-readable storage device has instructions for execution by a processor of a machine to cause the processor to perform operations to perform a method. The operations include receiving training data comprising molecule graphs and for each molecule graph, mapping nodes of the graph via a molecule embedding model encoder to an embedding space to generate node embeddings, aggregating the node embeddings to generate a molecule embedding vector, decoding the molecule embedding vector, via an attention based recurrent neural network decoder to generate an output including a character-based representation of the molecule, and using backpropagation of a gradient of an error from a loss function applied to the output of the decoder and a character-based representation of the input molecule graph to modify weights of the molecule embedding model encoder and decoder for mapping of the molecule graph to the embedding space.

12. The device of example 11 wherein the loss function comprises a cross entropy of the output character-based representation and a tokenized representation of the molecule.

13. The device of example 11 wherein the loss function comprises a Connectionist temporal classification function of the output character-based representation and a tokenized representation of the molecule.

14. The device of any of examples 11-13 and further comprising initializing attention weights of the of the attention based recurrent neural network decoder.

15. The device of example 14 wherein the attention weights are initialized using a feed forward layer based on an input token and the molecule embedding vector.

16. A computer implemented method includes receiving, at an embedding model, examples from a training data batch, wherein the examples from the training data batch includes a graph that represents a molecule and a corresponding label, outputting, from the embedding model, molecule features for each example received from the training data batch, wherein the molecule features map to an embedding space, decoding the molecule features for each example to produce a character-based representation of the molecule, calculating a loss between the produced character-based representation and the label, backpropagating gradient of the loss function to the embedding model for each example in the training data batch, and modifying learnable weights of the embedding model based on the back propagation.

17. The method of example 16, wherein the embedding model includes one or more graph neural network layers and one or more attention layers.

18. The method of example 17, wherein the graph includes nodes and edges, wherein the one or more graph neural network layers use a message-passing framework, wherein the one or more attention layers determine edge weights to be applied to messages received by a receiving node in the graph from one or more sending nodes in the graph, and wherein the molecule features are based in part on the edge weights and the messages.

19. The method of example 18, wherein the edge weights are based on features of the receiving node and the one or more sending nodes.

20. The method of example 19, wherein the edge weights are further based on a weighting coefficient and the one or more attention layers modify the weighting coefficient based on the back propagation.