TextGraphBART: Unifying Graph and Text with Structure Token

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 07:07:54

Abstract: We propose a novel encoding method called ``Structure Token'' to unify the processing and generation of both graphs and texts with a single transformer-based model. This method allows graphs with text labels to be generated by a series of tokens, enabling both graph and text data to be handled interchangeably. By utilizing structure tokens, our model learns a unified representation, enhancing the ability to process diverse data without requiring extra modules or models. Additionally, the model can be trained like most transformer models with simply cross-entropy loss. To demonstrate the effectiveness of our method, we introduce a pre-training scheme inspired by mBART but adapted to leverage structure tokens. Our model, named TextGraphBART, uses the same architecture as normal Transformer Encoder-Decoder models with small modifications on the input and output to accommodate structure tokens. The evaluations show that this approach achieves comparable results against baseline models of similar sizes on both text-to-graph and graph-to-text generation tasks, without needing specialized loss functions or sampling techniques. These findings suggest that our approach can effectively bridge the gap between textual and structural data representations, and the design of encoding method could offer a new direction for future improvement.

Main paper: 
 introduction: 
 Transformer layers have been proven to work well in several domains beyond text, like audio, image, and even multi-modal data. Some research has also shown that with careful design, transformer layers can extract features from graph data [30, 16] . Graph is a common data structure for representing concepts and relationships. In this work, we focus on a specific type, named text graph, where the concepts and relationships are expressed as texts, such as knowledge graphs and parsing trees.Learning vector representations and generating new text graphs are two essential aspects of text graphs in machine learning. Since texts can be viewed as a chain of words or characters, the text graph then becomes a nested graph. The complexity of handling such a nested graph leads to two major approaches for generating text graphs.The first strand is the multi-stage approach which generates concepts, relationships, and texts in different steps [22, 12] . The process usually involves multiple models that generate different parts of the text graph. For example, Grapher [22] uses the T5 [24] pre-trained model to generate all the concepts and then uses a relation extraction model to predict the relationships between every two concepts. This approach requires the models to generate a complete graph despite the edge sparsity of the target text graph. Therefore, the model includes a special "no-relation" relationship and turns every graph into a complete graph thus requiring extra predictions. Since the relation extraction is Each structure token contains a (sub-)word token with the locational information of that word token in the text graph. (1b) An example of autoregressive decoding with structure token. The procedure is mostly the same as normal text decoding with Transformer model. done on every two concepts, the model does not consider multi-hop relations. Moreover, the model cannot handle the case where two concepts have more than one relation. The second method is the graph linearization approach that fuses the hierarchy in text graph into chain of tokens [1, 10] . This approach treats the text graph as a special text sequence and enables the direct adoption of Language Model (LM) for text graph generation. The idea can also be applied to learning vector representations of text graphs. For example, BT5 [1] convert the text graph into sequence of (subject, relation, object) triples and train T5 to translate between sentence and sequence of triples. Since sequence generation with LM is done in an autoregressive manner, the generation is conditioned on the already generated triples. This behavior allows the model to handle multi-hop relations and the multi-relational case. Meanwhile, the model does not suffer from generating complete graphs because the model can learn to terminate when the generated triples match the target text graph. However, using sequence of triples also introduces extra complexity to the LM. Since the format requires matching the concepts in triples to reconstruct the text graph, there will be duplications of subject or object in the sequence. Thus, the model generates duplicated texts and cannot handle the case where two concepts are represented by the same text but refer to different things. Also, it relies on the model to implicitly learn the connection between two triples with duplication. Furthermore, LM is neither permutation invariance nor equivariance, which means the prediction alters if the generated triples are being shuffled.The goal of this work is to design a new approach that preserves some of the advantages while avoiding the drawbacks of previous approaches. This sets a few desired properties of the new approach. First, the method should be suitable for both representation and generation. It should also consider the cases that cannot be handled by multi-stage and graph linearization approach. Second, the model should be permutation equivariance and perform generation in an autoregressive manner.Last, the method should avoid extra computation, such as the duplication of concepts. To achieve the desired characteristics, we propose the structure token approach, as illustrated in 1. Our method employs a concept we call "Structure Token" which losslessly encodes the text graph into a set of tokens. The token contains a word and a few identifiers for the precise location of that (sub-)word in the text graph. Our model incorporates an unmodified Transformer Encoder-Decoder model [26] with structure token embeddings and a structure predictor for predicting new structure tokens. The text graph is generated autoregressively like regular text generation and the generated structure tokens form a subgraph. Once the generation stops, we can decode the set of structure tokens into the target text graph. A notable difference between our approach and previous approaches is that our Transformer model operates on sets instead of sequences and we view text graphs as nested graphs.To our knowledge, our structure token approach is the first method that can autoregressively generate sub-graphs with multi-token labels without modifying transformers.We validate our structure token approach on text-to-graph (T2G) generation and graph-to-text (G2T) generation tasks. By treating sentence as a special text graph without any edges, the generation tasks become a text-graph-to-text-graph translation problem. Therefore, we present TextGraphBART, a Transformer Encoder-Decoder model pre-trained on text-graph-to-text-graph translation with our structure token approach. The model is evaluated on two publicly available parallel datasets, EventNarrative [8] and WebNLG (2020) [5] , and achieves comparable results using fewer parameters and pre-training data.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The proposed methods and results are described in the Method Section (Section 3) and the Experiments and Results Section (Section 4).Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Recent works for text graph generation primarily focus on reusing pre-trained LMs. BT5 [1] applies the graph linearization approach with T5 [24] pre-trained model. ReGen [10] further proposes a Reinforcement Learning (RL) objective to improve the performance. On the other hand, Grapher [22] uses T5 as an entity extraction model and jointly trains another relation extraction model. It provides two types of implementation: Grapher (Text) and Grapher (Query). The Grapher (Text), which is the state-of-the-art method on WebNLG (2020) [5] dataset, generates entities as a flat sequence, while Grapher (Query) feeds a set of query embeddings to T5 decoder and apply another model to fully decode the outputs to entities. Other works target on the problem of lacking paired datasets for T2G generation. CycleGT [12] apply the cycle-training framework on a G2T model and a multi-stage T2G model. INFINITY [29] apply the cycle-training framework on a single T5 model with graph linearization approach.Most of the works for learning vector representations only focus on G2T generation. GAP [9] proposes a graph-aware attention that first uses the pooling operator to get the features of each text label and then applies a modified attention operator on those features. KGPT [6] provides two types of encoder: graph encoder and sequence encoder. The graph encoder is based on graph attention network [27] that also operates on the pooled text features. On the other hand, the sequence encoder infuses the structure information into the text tokens and feeds them into a transformer model. This approach resembles our structure token approach upon learning vector representations. However, they convert the graph into a dictionary-like format which suffers from a similar duplication problem as graph linearization approach.Our method is essentially derived from the design of TokenGT [16] , which also converts graphs into sets of tokens containing labels and identifiers. However, their idea does not directly fit in our scenario of text graph for two reasons. First, a single graph element (node or edge) in TokenGT needs to be representable by a single token. On the contrary, it would require multiple tokens for an element of text graph because the label is a multi-token text. Second, TokenGT only focuses on representing the graph, while we are interested in graph generation as well.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations are discussed in the Discussion Section (Section 5)Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
structure token: 
 The proposed structure token is a data representation that can losslessly encode all data in a text graph as a set of tokens. Given a text graph G = (N , A) containing a node set N and an arc set A.Each arc is a triple of a head node, an edge, and a tail node. Each graph element (node and edge) is a unique text label S identifiable with an integer ID. This allows different nodes or edges to have the same text label. The full formal definitions can be found in Appendix A.1. In order to convert text graph to structure tokens, we express the node set and arc set into one unified structure of graph elements. Each graph element will be represented by multiple structure tokens. A structure token consists of seven parts: 1. Label: The (sub-)word token of a graph element. 2. Type: A binary indicator specifying whether this graph element is a node or an edge. 3. Token ID: An unique ID for this token. 4. Previous ID: The token ID of previous token. 5. Segment ID: An unique ID for the graph element. 6. Head ID: The segment ID of the head node. 7. Tail ID: The segment ID of the tail node. If the token is part of a node, the head ID and tail ID will just be the segment ID of itself.With these information, we are able to differentiate between structure tokens that are from different parts of the graph. The text graph is converted into a set of structure tokens. Since the IDs can point to a graph element directly, there is no need for duplications like graph linearization approach. We provide the formal definition of structure token in the Appendix A. 10) OneHot2( 0) Orthn( 7) Orthn( 2) Orthn( 6) Orthn( 8 and corresponding structure tokens can be found in Figure 2a . The idea of type, head ID, and tail ID are inherited from TokenGT [16] , which uses identifiers to indicate the connections. We modify their definition and introduce extra identifiers for our text components. The token ID and previous ID are text-level identifiers. The text order is determined by the token ID and previous ID for reconstructing the text label. On the other hand, the segment ID, head ID, and tail ID are graph-level identifiers. For tokens of a specific graph element, the graph-level identifiers of each token will be the same.Furthermore, We add an extra "domain token" to the structure tokens of a text graph to indicate the domain of the graph, like the special language token used in multi-lingual translation [15, 21] . With the domain token, we can specify what kind of data the text graph is holding. For example, since text is treated as a text graph without any edges, we use a "[text]" domain token to indicate that this text graph represents a text. Besides, we use the domain token as the first token of every text label, so the previous ID of the first token of all labels are pointing to the domain token.The structure tokens are transformed into fixed-size high-dimensional vector representations. Each part of the structure token is converted into a vector and concatenated together. Then the vectorized result will be fed to a trainable projection layer for getting the token embedding, as illustrated in Figure 2b . Label and type are converted with one-hot encoding, denoted as OneHot. On the other hand, the IDs need to be handled differently. In order to preserve the graph structure in the tokens with the Transformer model, each ID needs to be converted into orthonormal vectors as proved byTokenGT [16] . We loose the requirement of orthonormality and use a set of orthonormal-like vectors.The dot product value of two different orthonormal-like vector is close to zero or less than some thresholds. These vectors of identifiers enable the attention operation in the Transformer model to be able to aggregate corresponding information through dot product. Each ID would be converted into an orthonormal-like vector through a transform function Orth. We use this transform function to convert the graph-level identifiers directly. On the other hand, we add the vectors of the text-level identifiers together, which allows the attention to aggregate information from neighbor tokens like the absolute position embedding [26] . Details and definitions can be found in Appendix A.3. Unlike position embedding which depends on the location of the tokens in a sequence, the text-level identifiers directly point to the neighbor token no matter their location in the sequence. Meanwhile, sequence orders are defined by IDs and the IDs can be randomly assigned. Therefore, applying any permutation of the input embeddings is equivalent to applying the same permutation on the output hidden states.Text Generation After converting the structure tokens into embeddings, those embeddings are fed into the unmodified Transformer Encoder-Decoder model. Conceptually, our model generates a structure token at each step which contains seven objects. However, we do not really need to generate seven objects at every step. The token ID is unique for every token and we can randomly pick any ID sequence beforehand. Notably, the structure token representation is a set, while the autoregressive Causal Transformer Layer ... generation manner makes the generated tokens resemble a sequence. Although the design of structure tokens enables the possibility of non-monotonic order of text generation, we slightly restrict the generation order of the structure tokens from the same graph element to be ordered and contiguous.With this restriction, we do not need to predict the token ID and previous ID. We can use the same generation scheme of other text generation Transformer model that simply generates the next text token until we are done with this element. Meanwhile, since the graph-level identifiers are the same within a graph element, we only need to predict the graph-level identifiers for the first token of labels.The generation can be further simplified for a single sentence since the graph-level identifiers are merely the token ID of the first token. Thus, text generation with our structure token approach is almost the same as other Transformer-based text generation models.Text Graph Generation For text graph generation, the same methodology applies. We use a structure predictor for predicting the identifiers. The graph-level identifiers are the same within a graph element. The prediction of graph-level identifiers can be done only one time per graph element.Moreover, the type and segment ID can also be omitted because we can tell the values once we get the head ID and tail ID. As a result, our structure predictor only needs to predict the head ID and tail ID. For predicting the IDs, we employ a single causal Transformer layer (a layer of the Transformer decoder), as illustrated in Figure 3 . The causal Transformer layer takes the output of the Transformer model plus the transformed segment ID to produce a hidden vector. The hidden vector will be fed into two projection layers to get a prediction of the head ID and tail ID. To get the ID, we multiply the final hidden vectors with a list of our orthonormal-like vectors, and perform softmax on the multiplication result to get the predictions. With this setup, we can apply the same teacher forcing technique as other Transformer decoders, so the training process is also parallelized.We introduce a pre-training method for our model based on the mBART pre-trained model for multilingual text generation [21] . The pre-training method contains two types of training objectives:the self-supervised objective and translation-like objective, as illustrated in 1 . The model is pre-trained on TEKGEN [2] and GenWiki [14] , and then we fine-tune the pre-trained model on EventNarrative [8] and WebNLG (2020) [5] for evaluating our model on G2T and T2G generation, respectively. The datasets are automatically generated by aligning texts with existing databases, except WebNLG (2020). TEKGEN is a large-scale dataset curated by aligning a subset of the Wikipedia text with Wikidata [28] . GenWiki is another large-scale dataset built on Wikipedia text. The text graphs are collected from DBpedia [3] . EventNarrative is an event-centric dataset that contains text graphs from the EventKG [11] and Wikidata. The text is also a subset of Wikipedia text. WebNLG (2020) is crowd-sourced dataset crafted by human annotators.The text graphs are collected from DBpedia, while the texts are manually written by annotators. It contains 16 categories in the training set and 19 categories (3 extra categories) in the test set. We use the official data split for all the datasets. As for the metrics, we used BLEU [23] , METEOR [4] ,and BERTScore [31] to evaluate the G2T performance, and we use the official evaluation script of WebNLG (2020) to evaluate the T2G performance.Setups Our model is trained in two phases: pre-training and fine-tuning. We initialize our model from scratch and perform the pre-training method. For pre-training, We use the RAdam optimizer [20] with a learning rate of 0.0001. The model is updated with an effective batch size of 256 and being trained for 5 epochs on a single A100 40GB GPU. All fine-tuning experiments are done on a single RTX 3090 24GB GPU. For fine-tuning on EventNarrative, we use the Lion optimizer [7] with a learning rate of 0.00001. The model is updated with an effective batch size of 128 and trained for 20 epochs. For fine-tuning on WebNLG (2020), we use the Adam optimizer [17] with a learning rate of 0.0001. The model is updated with an effective batch size of 128 and trained for 100 epochs.We use an overall hidden size of 512 for our model. The unmodified Transformer encoder and decoder both have 6 layers. Each attention has 16 heads, and we use a hidden size of 32 for self attentions and 64 for cross attentions. The feed-forward layer in Transformer has an input and output hidden size of 512, and the intermediate hidden size is 2048. We use these numbers for the structure predictor as well. For the activation functions, we use the GELU activation function [13] for Transformers and hyperbolic tangent function for the projection layers of structure predictor. During pre-training, we apply the dropout [25] on the attention weights and the residual connections with a dropout rate of 0.1. The model weights are randomly initialized with a mean of 0 and a standard deviation of 0.02.For data processing, we use the same subword tokenizer as T5 which uses the Unigram tokenization method [24, 18] . The tokenizer has a vocabulary of 32100 text tokens, which contain 32000 subword text tokens and 100 reserved special tokens. We use the reserved special tokens for our domain tokens. We compared our model with T5 [24] , BART [19] , and GAP [9] . Both T5 and BART are Transformer Encoder-Decoder models pre-trained on text data and fine-tuned with graph linearization [8] , while GAP modifies the encoder of Transformer Encoder-Decoder model with graph-aware modules for extracting graph features [9] . It is noteworthy that all these models use a similar Transformer decoder. The main difference among TextGraphBART and these models is the way we represent and handle the text graph input.The result is shown in Table 2 . In comparison to T5 and BART, our structure token method achieves better score with fewer parameters than graph linearization approach. Meanwhile, our model is comparable with GAP without modifying the Transformer model. As a conclusion, our structure token representations enabled the Transformer model to capture better features from the text graph than the graph linearization approach.We compare our model with CycleGT [12, 5] , BT5 [1] , and Grapher [22] . BT5 is T5 pre-trained and fine-tuned with graph linearization. On the other hand, both CycleGT and Grapher adopt the multi-stage approach. The CycleGT is a well-known multi-stage approach for text-to-graph generation using cycle training [12] , while Grapher performs supervised learning with a special loss function [22] . Meanwhile, we use the officially released source code of Grapher1 to train a Grapher-small (Text) which has a similar model size (95M) with our model (75M). BothGrapher-small and our TextGraphBART are trained for 100 epochs with the same learning rate and effective batch size.The result is shown in Table 3 . In comparison to CycleGT and Grapher (Query), our simple generation method with structure tokens outperforms models with special training methods. Although our model does not directly match the performance of the large models like BT5 or Grapher (Text), our model is comparable with Grapher-small that has similar model size. Furthermore, we analyze the result by measuring the performance on each category of the WebNLG (2020) test set comparing to Graphersmall. The result is shown in Table 4 . Even though Grapher-small is based on the T5-small pre-trained model, which is trained on an extremely large dataset of 750 GB: the Colossal Clean Crawled Corpus [24] , we can see that our model performs slightly better than Grapher-small on unseen categories (0 samples in training set). In conclusion, our structure token approach can achieve comparable performance on text-to-graph generation under similar model size without using special training methods or loss functions.Ablation Study To investigate the performance contribution of the components of structure tokens,we conducted the ablation study on our structure embedding by fine-tuning our model with the removal of some parts of the embeddings. The model is trained on the WebNLG (2020) with the same setup. The results are shown in Table 5 . In all ablations, the model performance was attenuated as expected. First, the ablation of the token ID and previous ID removes the text order information in the text labels hence the degeneration of performance. Similarly, the head ID and tail ID provide the connectivities of the graph. Removal of this embedding decreases the performance, indicating the importance of the connectivities. On the other hand, the ablation of type and segment ID are not as detrimental as others because the type and segment ID may be inferred from other IDs. Thus, our model is still able to perform albeit less performant. In conclusion, the ablation study showed that all of our structure embedding is important for good model performance.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification: code is provided Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
discussion: 
 The primary objective of this work is to demonstrate the effectiveness of the proposed structure token approach. Due to resource constraints, there are numerous aspects remain unexplored.Scaling Up We believe that TextGraphBART has the potential to achieve better results when scaled up. The backbone of our model is simply the transformer model, like most of the other baselines.Meanwhile, we observed a clear improvement on other baselines when scaling up (e.g. comparing T5-Base and T5-Large for G2T in 2 and Grapher-small and Grapher for T2G in 3). Therefore, we anticipate that transformers on T2G and G2T follow the scaling laws.Model Architecture While we use The Transformer Encoder-Decoder model in our experiments, there are no strict restrictions on the model architecture as long as there are dot product attention operations. The same approach can be applied to encoder or decoder-only model.In the experiments, we only use parallel datasets for pretraining. Since our pre-training scheme contains a T2T path, it's possible to pre-train the model merely with plain-text datasets. On the other hand, we can also incorporate programming language datasets and use syntax parsers to generate the Abstract Syntax Tree (AST) as the text graph for pre-training. However, it is unclear how much the self-supervised objective and translation-like objective affect the downstream performances. 
conclusions: 
 We present a novel approach to the problem of text graph generation leveraging the strength of Transformer models. Our exploration has led to an effective method for structured data representation and generation via structure tokens. In the structure token, we use several identifiers to indicate the connectivities of the graphs and the order of the texts. Then an embedding method for structure token is proposed, allowing the Transformer model to utilize the structural information. We show that the structure token approach can be used to represent and generate both texts and text graphs.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The experiment setups are described in the Experiments and Results Section (Section 4)Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Due to resource constraints, we are unable to perform significance tests.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: The experiment setups are described in the Experiments and Results Section (Section 4)Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: The main result of this work is a new encoding method, which has no societal impact Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: All datasets used are publicly available open datasets.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All dataset and code used are publicly available and cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Let T be the set of all possible text tokens. A text label is defined as a set of pairs containing the text token and positions. We generalize this definition by replacing the position with a contiguous sequence of unique ID. Given an infinite unique ID sequence:I = (id i ) ∞ i=0 where id i ∈ Z + ∧id i ̸ = id j if i ̸ = j.Each id i is a positive integer. We also define id(i) = id i for simplicity. By picking a corresponding ID sequence, we can use any positive integer sequence as the positions. Then a text label S of length l is a set of token-ID pairs defined as:S = {(t i , id j+i ) | 1 ≤ i ≤ l, t i ∈ T } ⊆ T × Z + .We can conditionally specify the start point j ∈ N with S j . Let S be the set of all possible text labels and S j for a specific start point. With the ID sequence I, the positions (p) l p=1 can be replaced with (id j+k ) l k=1 . Then we can union text labels without missing information by picking non-overlapped ID sequences, which is a desired property for the attention operation. A text graph G = (N , A) is composed of a node set N with q nodes and an arc set A with r arcs.The node set N is a set of node labels paired with unique IDs of the nodes, defined as:N = {(N i , n i ) | 1 ≤ i ≤ q, n i ∈ Z + , N i ∈ S} ⊆ S × Z +where N i is the node label and n i is the corresponding node ID. Similarly, an edge set E is a set of edge labels paired with unique IDs, defined as:E = {(E i , e i ) | 1 ≤ i ≤ r, e i ∈ Z + , E i ∈ S} ⊆ S × Z +where E i is the edge label and e i is the edge ID. Notably, the ID used in N and E are disjoint. Then the arc set is defined as:A = {(N h i , E i , N t i ) | 1 ≤ i ≤ r, N h i , N t i ∈ N , E i ∈ E} ⊆ N × E × Nwhere E i is the edge and N h i , N t i is the head node and tail node, respectively.With the setup in A.1, the formal definition of the structure token representation is defined as a set of septuples (tuples with 7 elements). Given a text graph G = (N , A) and its components: nodeN i = (N i , n i ) ∈ N , 1 ≤ i ≤ |N |, arc A j = ((N h j , n h j ), (E j , e j ), (N t j , n t j )) ∈ A, 1 ≤ j ≤ |A|, edge E j ∈ E = {(E j , e j ) | 1 ≤ j ≤ |A|}, and the length of the text labelsl N i = |N i |, l E j = |E j |,we define two sequences:L N = (L N k ) |N | k=1 where L N k = 0, if k = 1 l N k-1 + L N k-1 , if k ̸ = 1 L E = (L E k ) |A| k=1 where L E k = |N | k=1 l N k , if k = 1 l E k-1 + L E k-1 , if k ̸ = 1We assign n i = id(L N i + 1), e j = id(L E j + 1) and specify the start point of each text label such thatN i ∈ S L N i , E j ∈ S L E j .By doing so, we can get the node (or edge) ID from the token IDs of its text label. For each node N i and edge E j , we define the corresponding structure token representationX N i and X E j as:EQUATIONThen the corresponding structure token representation G ′ of the text graph G is defined as:EQUATIONwhere t D ∈ T , id 0 is the ID of domain token and X D = {(t D , 1, id 0 , id 0 , id 0 , id 0 , id 0 )} is the domain token. Each septuple X ∈ G ′ is a structure token containing the label, type, token ID, previous ID, segment ID, head ID, and tail ID. With this definition, we can represent every possible token ID assignment by specifying the unique ID sequence I. On the other hand, since the graph element can be randomly permuted, every possible ordering is also representable with our set G ′ by picking the corresponding unique ID sequence.To convert the structure tokens into embeddings, we use 3 kinds of transform functions. For label and type, we use the one-hot encoding, denoted as OneHot n : A → E n where A is a set with n elements and E n is the standard basis of R n . On the other hand, each ID would first be converted into a d-dimensional orthonormal-like vector with a function Orth d . To get the orthonormal-like vectors, we modify and normalize the sinusoidal position encoding of Transformer [26] with different frequencies. The d-dimensional sinusoidal position encoding PE d at position i is defined as:PE d (i) = d/2 ∥ k=1 pe(i, k) ∈ R d pe(i, k) = sin( i 10000 k/d )∥ cos( i 10000 k/d ) ∈ R 2 (3)where ∥ denotes the vector concatenation. We generalize the definition of PE d with a frequency function f :EQUATIONThen by taking f ′ (k) = 10000 -k/d , the original PE d can be defined with orthonormal-like vectors as:PE d = PE * d {f ′ }. WeEQUATIONwhere norm 2 is the L2 normalization and • denote the function composition. We find that by picking the frequency function -log(k), the generated vectors satisfied the desired properties. In Figure 5 , we generate 1024 vectors by applying Orth on 1 ≤ i ≤ 1024 and compute the cosine similarity of every possible pair. We can see that the similarity values are mostly close to zero.Once we can convert the IDs into orthonormal-like vectors, we use Orth d directly as the transform functions for the graph-level identifiers. For text-level identifiers, we add the vector of token ID and previous ID together. Given two non-domain token a and b with token ID t a , t b and previous ID p a , p b , we have Orth(t a ) • Orth(t b ) ≈ 0 since token ID is unique and Orth is orthonormal-like.Meanwhile, we have p a ̸ = p b for most tokens. The dot product value of a and b become:(Orth(ta) + Orth(pa)) • (Orth(t b ) + Orth(p b )) = Orth(ta) • Orth(t b ) + Orth(ta) • Orth(p b ) + Orth(pa) • Orth(t b ) + Orth(pa) • Orth(p b ) = Orth(ta) • Orth(p b ) + Orth(pa) • Orth(t b ) + Orth(pa) • Orth(p b ) + ε =          Orth(ta) • Orth(p b ) + ε = 1 + ε, if a is the previous token of b Orth(pa) • Orth(t b ) + ε = 1 + ε, if b is the previous token of a Orth(id0) • Orth(id0) + ε = 1 + ε,if both a and b are the first token of text label ε, otherwisewhere ε is a small value around 0. The value will only be meaningful in the dot product if one token is the previous token of the other. The case of first tokens is set to allow exchanging information with the domain token. This can be suppressed by making Orth(id 0 ) = 0.With these designs, we define our structure token vectorize function V ec as:EQUATIONwhere π i (X) denote the i-th element of the septuple X. The vectorized result will be fed into a trainable projection layer Emb : R 4n+|T |+2 → R d to get the structure embedding with d dimensions.NeurIPS Paper Checklistcode available at: https://github.com Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS2024). Do not distribute.https://github.com/IBM/Grapher 


Decision: The paper proposes ``structure token'' to unify the representation and generation of graph and text. The reviewers are consistently negative towards the work, which could be improved by addressing the following aspects: (1) further enhancing performance to reach state-of-the-art (SOTA) levels, (2) updating the generation tool, (3) including more ablation studies, and (4) scaling up and reporting performance.

Reviews: review: paper_summary: paper_summary: - The paper highlights the limitations of two existing methods for generating text graphs: 1. The multi-stage approach does not consider multi-hop relations and cannot handle the case where two concepts have more than one relation. 2. The graph linearization approach introduces extra complexity to the Language Model (LM) and the predictions are altered if the generated triples are shuffled.
- Building on this, the paper proposes the Structure Token method. Specifically, it identifies a graph element (node or edge) using seven parts. Then, it transforms Structure Tokens into embeddings using OneHot and orthonormal-like vectors, which are then input into a Transformer Encoder-Decoder model.
strength_weakness: strength_weakness: - The paper's layout is not aesthetically pleasing, particularly the formatting of Tables 1-3.
- The experiments lack error bars, which diminishes the credibility of the results.
- The experimental results are unsatisfactory and fail to demonstrate the superiority of the model, and the experiments are incomplete.
  - The model's results did not achieve state-of-the-art (SOTA) performance in the G2T/T2G tasks.
  - Lines 257-259 state, "In conclusion, our structure token approach can achieve comparable performance on text-to-graph generation under similar model size without using special training methods or loss functions." The results in Table 3 are nearly identical to those of Grapher-small (Text) T2G. Combined with the absence of error bars, it is challenging to determine whether the similarities are due to errors or model efficacy, making it difficult to convince.
  - Existing results only may demonstrate that the model performs comparably to models with larger parameters under fewer parameters. This does not prove that increasing the number of parameters will enhance performance. A more direct comparison of experimental results is necessary to substantiate this claim. The explanations in Section 5, "Scaling Up," are overly subjective and unconvincing, making the paper seem like a work in progress.
questions: questions: - Please respond to the issues pointed out in the "Weakness" section.
limitations: limitations: Author addressed the limitations.
paper_summary: paper_summary: The paper proposes a new unified graph-text generation framework, TextGraphBART, for the large language model. The paper tries to address both the generation and representation of text and graphs. The paper proposes a new structure token to encode text graphs into a set of tokens. The structured token can encode the text graphs and be decoded into text graphs. Specifically, it consists of seven different embeddings, including position, domain, and text information. The paper pretrained the proposed framework over four different tasks, including text2text, graph2text, text2graph, and graph2graph. The model is pretrained on TEKGEN and GenWIKI and tests on the event narrative and WebNLG. The performance is evaluated on BLEU, METEOR< and BERTScore. The paper compares the model with T5, BART, GAP, CYcleGT, ZBT5, and Grapher. The paper also includes an ablation study and analyzes the future directions.
strength_weakness: strength_weakness: 1. The proposed framework is incremental. Multiple previous papers have used the idea of using different position embeddings to represent the structure information [1,2,3]. The idea of joint text-to-graph and graph-to-text pretraining/generation is also not new [4]. 
2. The experiment is not comprehensive. The paper used eventnarrative for the graph-to-text generation. However, compared to WebNLG  (2020), the EventNarrative has fewer baselines and most of its baselines are outdated. The paper must also add WebNLG(2020) as an additional graph-to-text generation tool to show its contributions. The pretraining datasets and the testing datasets are all Wiki-style datasets. The performance gain may come from pretraining (data leakage) instead of the actual model architecture. The paper needs to include an additional ablation study to show the gain of the eventnarrative/webnlg comes from the model architecture, or also pretraining the proposed baselines on TEKGEN/GenWiki. Otherwise, the scores are not comparable. Additionally, the paper needs to include some of the latest frameworks like [4] since all of the baselines used in the paper are old. The paper also needs to include some human evaluation or qualitative analysis to help readers understand the generation results better. Furthermore, the comparison in Table 4 is not fair, since domain-specific pretraining is more useful than general pretraining [5].
3. The paper puts important information in the Appendix while not reaching the page limit (9 pages). 

[1] Herzig, J., Nowak, P. K., Müller, T., Piccinno, F., & Eisenschlos, J. M. (2020). TaPas: Weakly supervised table parsing via pre-training. ACL 2020.

[2] Wang, Q., Yavuz, S., Lin, V., Ji, H., & Rajani, N. (2021). Stage-wise fine-tuning for graph-to-text generation. arXiv preprint ACL 2021 SRW.

[3] Chen, W., Su, Y., Yan, X., & Wang, W. Y. (2020). KGPT: Knowledge-grounded pre-training for data-to-text generation. EMNLP 2020.

[4] Wang, Z., Collins, M., Vedula, N., Filice, S., Malmasi, S., & Rokhlenko, O. (2023). Faithful low-resource data-to-text generation through cycle training. ACL 2023

[5] Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020). Don't stop pretraining: Adapt language models to domains and tasks. ACL 2020.
questions: questions: See weakness
limitations: limitations: The paper includes a limitation section in the Appendix and a Discussion section in the main paper.
paper_summary: paper_summary: This paper introduces TextGraphBART. Its a new method of encoding graph/text-input by using a structure token. This new token should preserve graph structure as opposed graph linearization or cycle training. This should also allow for the generation of graphs, with accompying text tokens, without making architectural changes to the transformer. It is then verified on both Graph2Text and Text2Graph tasks for commonly used benchmarks such as WebNLG and EventNarrative. It is followed by results that show comparable results to previous works, and a short discussion and conclusion.
strength_weakness: strength_weakness: 1. The work proposes a seamless integration of both text and graph inputs, but only verifies on strictly one-directional modal evaluation: Either text-to-graph or graph-to-text. However, as the authors claim that this is the "first method that can autoregressively generate sub-graphs with multi-token labels without modifying transformers.", a single text-to-graph benchmark feels like a very shallow evaluation.
2.  Models are evaluated on datasets that already work well. This makes it hard to distinguish the added value of such a structure token. Scenarios are sketched where e.g. graph linearization is treated simply as a sequence of tokens, but then it this work should methodologically be verified on datasets where preserving graph structure should be required. This is now not the case.
3.  This is reflected in the results: they are comparable, so the structure token might aswell not be there. The proclaimed difference that this is done with less parameters I would argue is a very weak claim as the difference is not that big.
4.  Finally, I am not convinced that the proposed "structure token" is a truely lossless way of preserving graph structure through-out a transformer. The structure token and its generation is explained in Figure 2, and is claimed to be lossless. However, this is only true in the operations before the embedding leveraging the orthogonal property. After the embedding into a structure embedding, information will (seemingly) be lost about structure in this embedding. From this structure embedding, it cannot be determined what the original structure of the graph was?
5.  In the discussion the authors mention scaling, but there is no clear evidence for this as the authors only tested one model size. This is essentially an unsubstantiated claim.
 6. Why is the domain token needed? This is not further addressed or experimented with.
questions: questions: 1. Why is it that this structure token obsoletes the need for transformer modifications?
2. Why is it interesting that this method obtains comparable results?
3. Line 56: "It should also consider the cases that cannot be handled by multi-stage and graph linearization approach", so what are these cases and how are they not handled by these methods but they are by TextGraphBart?
limitations: limitations: The authors partially address this limitations by motivating the paper in the introduction. However, I dont feel the discussion (or the rest of the work) addresses the actual limitations of this method. What are pros or cons compared to transformer modifications?
paper_summary: paper_summary: This paper proposes a method to integrate the processing and generation of both text and graph data using a single transformer-based model. Structure Token encodes graphs with text labels into a sequence of tokens, enabling the handling of both data types interchangeably. This approach leverages a unified representation within a Transformer Encoder-Decoder model, enhanced to incorporate structure tokens. They evaluate TextGraphBART on text-to-graph (T2G) and graph-to-text (G2T) tasks, demonstrating comparable results to baseline models with fewer parameters.
strength_weakness: strength_weakness: 1. The baseline settings are weak. It lacks of comparing with some strong baselines with advanced graph-structured-aware methods [1][2].
2. The model settings are limited. This paper only try one size of model, and further experiments with larger model sizes are needed.
3. The provided ablation study, though useful, could be more detailed, exploring the interactions and contributions of various components more comprehensively.

Reference: 
[1] Stage-wise Fine-tuning for Graph-to-Text Generation
[2] Self-supervised Graph Masking Pre-training for Graph-to-Text Generation
questions: questions: Many research focuses on reduce the gap between textual and structural data representations, as I mentioned in Weakness. What are your advantages over other methods?
limitations: limitations: Yes. The limitations are discussed in Discussion section.

Comments: comment: Thanks for your response, I will keep my score.
Thank you for your rebuttal. Having read it, I've decided to keep my scores.
Thank you for explanation. I decide to keep my score.
Thank you for your explanation! I decide to keep my score after reading your rebuttal.