Question: How exactly is the piece level sequence generation autoregressive model connected to the bond completion? How is the fine structure (ie the nodes and edges) of each piece generated so that the bond-completion model can work only on nodes across different pieces?
Evidence: With a molecule G represented as {P i }, { Ẽij } , our model generates {P i } and { Ẽij } in two consecutive phases. The two phases move from coarse to fine granularity.
Piece-level Sequence Generation Given a latent variable z, our model first uses an autoregressive sequence generation model P (P i |P <i , z) to decode a sequence of graph pieces [P 1 , . . . , P n ] 2 . During training, we insert two special tokens "<bos>" and "<eos>" into the begin and the end of the graph piece sequence to indicate the begin and the end, respectively. During testing, the generation stops when a "<eos>" is generated. The sequence model can be RNNs, such as LSTM and GRU. In this work, we use a single layer of GRU and project the latent variable z to the initial state of GRU. The training objective of this stage is to minimize the log-likelihood L P of the ground truth graph piece sequence:
(5) (c) Latent space encoding. We obtain the latent variable z through the graph encoder (Section 3.2). (d) Piece-level sequence generation. A sequence of graph pieces is auto-regressively decoded from the latent variable by a GRU (Section 3.3). (e) Incomplete molecular graph. The generated graph pieces form an incomplete molecular graph where inter-piece bonds are absent. (f) Bond completion. Completion of inter-piece bonds is formalized as a link prediction task for a GNN (Section 3.3). After training, we can directly sample from the latent space to generate molecules.
We generate { Ẽij } non-autoregressively. For the architecture, we use a GNN with the same structure as the graph encoder in Section 3.2 but with different parameters to obtain the representations h v of each atom v in the graph pieces. Given nodes v and u, we predict their connections as follows:
where H θ is a neural network. In this work we adopt a 3-layer MLP with ReLU activation. Apart from the types of chemical bonds, we also add a special type "<none>" to the edge vocabulary which indicates there is no connection between two nodes. During training, we predict both P (e uv ) and P (e vu ) to let H θ learn the undirected nature of chemical bonds. We use negative sampling to balance the ratio of "<none>" and chemical bonds. Since only about 2% pairs of nodes has inter-piece connections, negative sampling significantly improves the computational efficiency and scalability. The training objective of this stage is to minimize the loglikelihood L Ẽ of the ground truth inter-piece connections:
The reconstruction loss of our GP-VAE model is:
To decode { Ẽij } in inference time, the decoder first assigns all possible inter-piece connections with a bond type and a corresponding confidence level. We try to add bonds that have a confidence level higher than δ th = 0.5 to the molecule in order of confidence level from high to low. For each attempt, we perform a valency check and a cycle check to reject the connections that will cause violation of valency or form unstable rings which are too small or too large. Since this procedure may form unconnected graphs, we find the maximal connected component as the result of the generation. We present the pseudo code for inference in Appendix C.
We jointly train a 2-layer MLP from z to predict the scores of target properties using the MSE loss. Denote the loss of the predictor as L prop , the loss function of our GP-VAE is:
where α and β balance the reconstruction loss, the property loss, and the KL divergence D KL between the distribution of z and the prior distribution N (0, I).
Figure 4: Overview of the graph piece variational autoencoder. (a) Piece-level decomposition. Atoms and bonds which belong to different graph pieces are highlighted in different colors. (b) Molecular graph. We inject piece-level information into the molecular graph through atom features. (c) Latent space encoding. We obtain the latent variable z through the graph encoder (Section 3.2). (d) Piece-level sequence generation. A sequence of graph pieces is auto-regressively decoded from the latent variable by a GRU (Section 3.3). (e) Incomplete molecular graph. The generated graph pieces form an incomplete molecular graph where inter-piece bonds are absent. (f) Bond completion. Completion of inter-piece bonds is formalized as a link prediction task for a GNN (Section 3.3). After training, we can directly sample from the latent space to generate molecules.
Answer: We are sorry we made the description too high-level. The generation of graph pieces is similar to the generation of a sentence using RNNs, except that the token is graph pieces instead of words. Since each graph piece is a subgraph, the generated graph pieces then form an incomplete molecular graph where inter-piece bonds are absent. We use a GNN with the same structure as our encoder in Section 3.2 to predict the missing inter-piece bonds on the incomplete molecular graph, which can be formalized as a link prediction task. We have updated the description of our model structure as well as the Fig 5 (Fig 4 in the current version) for better understanding.