Title: value: TextGraphBART: Unifying Graph and Text with Structure Token
Authors: value: Ching-Wen Cheng
PING-CHENG YEH
Venue: NeurIPS 2024
Keywords: value: transformer
text generation
graph generation
text-to-graph
graph-to-text

Abstract: value: We propose a novel encoding method called ``Structure Token'' to unify the processing and generation of both graphs and texts with a single transformer-based model. This method allows graphs with text labels to be generated by a series of tokens, enabling both graph and text data to be handled interchangeably. By utilizing structure tokens, our model learns a unified representation, enhancing the ability to process diverse data without requiring extra modules or models. Additionally, the model can be trained like most transformer models with simply cross-entropy loss. To demonstrate the effectiveness of our method, we introduce a pre-training scheme inspired by mBART but adapted to leverage structure tokens. Our model, named TextGraphBART, uses the same architecture as normal Transformer Encoder-Decoder models with small modifications on the input and output to accommodate structure tokens. The evaluations show that this approach achieves comparable results against baseline models of similar sizes on both text-to-graph and graph-to-text generation tasks, without needing specialized loss functions or sampling techniques. These findings suggest that our approach can effectively bridge the gap between textual and structural data representations, and the design of encoding method could offer a new direction for future improvement.


Review discussion:
1. The introduction of structure tokens is a significant advancement, providing a unified method for processing and generating both text and graph data.
2. The method integrates seamlessly with existing Transformer architectures, requiring only minor modifications, and avoids the need for specialized loss functions or additional modules.
3. Empirical results show that TextGraphBART achieves comparable performance on both T2G and G2T tasks with baselines but with fewer parameters.
1. The baseline settings are weak. It lacks of comparing with some strong baselines with advanced graph-structured-aware methods [1][2].
2. The model settings are limited. This paper only try one size of model, and further experiments with larger model sizes are needed.
3. The provided ablation study, though useful, could be more detailed, exploring the interactions and contributions of various components more comprehensively.
Reference:
[1] Stage-wise Fine-tuning for Graph-to-Text Generation
[2] Self-supervised Graph Masking Pre-training for Graph-to-Text Generation
Many research focuses on reduce the gap between textual and structural data representations, as I mentioned in Weakness. What are your advantages over other methods?
Yes. The limitations are discussed in Discussion section.
No ethics review needed.
Thank you for your insightful review. We acknowledge that there are many research focuses on the textual and structural data representations.
However, they usually fall into the two categories mentioned in the “Introduction” section. Our structure token approach would be the third category, and this paper focuses on proving the proposed approach works under the simplest setting. Essentially, the mentioned “strong baselines with advanced graph-structured-aware methods” and our approach are not competitors. They are somewhat orthogonal and can be applied together.
Thanks for your response, I will keep my score.
1. Paper is generally nice to read
2. The work considers a lot of relevant related work, which they both refer to or base decisions on
3. The build-up of the structure token is well explained
1. The work proposes a seamless integration of both text and graph inputs, but only verifies on strictly one-directional modal evaluation: Either text-to-graph or graph-to-text.
However, as the authors claim that this is the "first method that can autoregressively generate sub-graphs with multi-token labels without modifying transformers.", a single text-to-graph benchmark feels like a very shallow evaluation.
2.  Models are evaluated on datasets that already work well. This makes it hard to distinguish the added value of such a structure token. Scenarios are sketched where e.g. graph linearization is treated simply as a sequence of tokens, but then it this work should methodologically be verified on datasets where preserving graph structure should be required. This is now not the case.
3.  This is reflected in the results: they are comparable, so the structure token might aswell not be there. The proclaimed difference that this is done with less parameters I would argue is a very weak claim as the difference is not that big.
4.
Finally, I am not convinced that the proposed "structure token" is a truely lossless way of preserving graph structure through-out a transformer. The structure token and its generation is explained in Figure 2, and is claimed to be lossless.
However, this is only true in the operations before the embedding leveraging the orthogonal property. After the embedding into a structure embedding, information will (seemingly) be lost about structure in this embedding. From this structure embedding, it cannot be determined what the original structure of the graph was?
5.  In the discussion the authors mention scaling, but there is no clear evidence for this as the authors only tested one model size. This is essentially an unsubstantiated claim.
6. Why is the domain token needed? This is not further addressed or experimented with.
1. Why is it that this structure token obsoletes the need for transformer modifications?
2. Why is it interesting that this method obtains comparable results?
3. Line 56: "It should also consider the cases that cannot be handled by multi-stage and graph linearization approach", so what are these cases and how are they not handled by these methods but they are by TextGraphBart?
The authors partially address this limitations by motivating the paper in the introduction.
However, I dont feel the discussion (or the rest of the work) addresses the actual limitations of this method. What are pros or cons compared to transformer modifications?
No ethics review needed.
We appreciate your thoughtful review. We address your main points below:
About Questions:
1. The necessity of transformer modifications is to incorporate the structural information of the graph into the models. TokenGT [1] shows how a good token design can obsolete the need of transformer modifications for graph data. Our structure token is built on top of TokenGT’s token design, which provides the desired structural information.
2. We think it is interesting even though this method only obtains merely comparable results because it shows many potential future directions in the field. The current trend of language models is extremely focused on generating sequences. Even generating structural data such as JSON data is done with sequences. Our model, while not reaching SOTA performance, still outperforms some models that are many times larger than ours. This shows the potential of rethinking how data is encoded and decoded.
3. Please refer to Line 35-36 and Line 48-49 for the cases not handled by previous approaches.
About Weaknesses:
1. We test our model on both text-to-graph and graph-to-text generation. The one-directional experiments test whether the proposed method works correctly and captures the desired information. If the reviewer can suggest some tasks or datasets that require bi-directional generation, it would be some nice future explorations.
2. We appreciate your suggestion for using that kind of dataset. We chose to evaluate our model on these datasets because they are the common datasets for evaluating text-to-graph and graph-to-text generation models.
3. Please compare our model with T5-Base in Table 2 and Grapher (Query) in Table 3. If the structure token is not there, there will not be a 20% performance difference for those baselines.
4. The “lossless” means each text graph encoded as structure tokens can be decoded back to the original text graph, compared to the graph linearization approach. It does not refer to the neural network would be lossless.
5. The scaling is not a claim. We purposefully put scaling in the “Discussion” section as a limitation of the current work and explicitly state the necessity of further explorations.
6. Please refer to Line 128-133. This allows the model to be trained with the proposed pre-training paradigm in section 3.4
[1] J. Kim, D. Nguyen, S. Min, S. Cho, M. Lee, H. Lee, and S. Hong. Pure transformers are
366 powerful graph learners. Advances in Neural Information Processing Systems, 35:14582–14595,
Thank you for your rebuttal. Having read it, I've decided to keep my scores.
1. The motivation of the paper is clear and the topic of the paper is interesting. The paper investigates an important unified graph-text generation framework for the language model. The idea of structure tokens is interesting and can contribute to future research.
Additionally, the model seems to achieve good results while using fewer parameters.
2. The paper tests on both graph-to-text and text-to-graph datasets. The proposed method achieves better results compared to other baselines. The paper also conducts an ablation study to investigate the contribution of each component. It includes a discussion session to explore the paper's current limitations.
3. The paper provides code and implementation details.
1. The proposed framework is incremental. Multiple previous papers have used the idea of using different position embeddings to represent the structure information [1,2,3]. The idea of joint text-to-graph and graph-to-text pretraining/generation is also not new [4].
2. The experiment is not comprehensive. The paper used eventnarrative for the graph-to-text generation.
However, compared to WebNLG  (2020), the EventNarrative has fewer baselines and most of its baselines are outdated. The paper must also add WebNLG(2020) as an additional graph-to-text generation tool to show its contributions. The pretraining datasets and the testing datasets are all Wiki-style datasets. The performance gain may come from pretraining (data leakage) instead of the actual model architecture. The paper needs to include an additional ablation study to show the gain of the eventnarrative/webnlg comes from the model architecture, or also pretraining the proposed baselines on TEKGEN/GenWiki.
Otherwise, the scores are not comparable.
Additionally, the paper needs to include some of the latest frameworks like [4] since all of the baselines used in the paper are old. The paper also needs to include some human evaluation or qualitative analysis to help readers understand the generation results better.
Furthermore, the comparison in Table 4 is not fair, since domain-specific pretraining is more useful than general pretraining [5].
3. The paper puts important information in the Appendix while not reaching the page limit (9 pages).
[1] Herzig, J., Nowak, P. K., Müller, T., Piccinno, F., & Eisenschlos, J. M. (2020). TaPas: Weakly supervised table parsing via pre-training. ACL 2020.
[2] Wang, Q., Yavuz, S., Lin, V., Ji, H., & Rajani, N. (2021). Stage-wise fine-tuning for graph-to-text generation. arXiv preprint ACL 2021 SRW.
[3] Chen, W., Su, Y., Yan, X., & Wang, W. Y. (2020). KGPT: Knowledge-grounded pre-training for data-to-text generation. EMNLP 2020.
[4] Wang, Z., Collins, M., Vedula, N., Filice, S., Malmasi, S., & Rokhlenko, O. (2023). Faithful low-resource data-to-text generation through cycle training. ACL 2023
[5] Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020). Don't stop pretraining: Adapt language models to domains and tasks. ACL 2020.
See weakness
The paper includes a limitation section in the Appendix and a Discussion section in the main paper.
No ethics review needed.
Thank you for your thorough review. We address your main points below:
1.
While the proposed framework seems to be incremental, we handle many situations that cannot be done with previous approaches. The idea of using position embeddings for structure information is well-known in graph neural networks (GNN).
However, as mentioned in the “Related Work” section, these approaches are only for encoding, whereas our framework focuses on both encoding and decoding. Essentially, most of the encoding methods are not lossless encoding, which means they cannot be used for decoding.
Meanwhile, our structure token is orthogonal to the joint pre-training framework, so we can apply those methods such as cycle training to our model.
2.
About the concern of data leakage due to the wiki-style datasets, it is less of a problem because Wikipedia is also used in other pre-training models like T5/Bart. If there is data leakage, there will not be much performance difference between our model and some baselines.
Furthermore, the comparison in Table 4 cannot be fair because Grapher-small (Text) is based on T5-small that pre-trained on the C4 dataset that is multiple scales larger than our TEKGEN/Genwiki pre-training dataset.
3. We appreciate the mention of the Appendix and the empty space (<1 page). We purposefully put the formal definitions in the Appendix for a better reading experience. We will move some of the information in the Appendix for a better use of the space in the revision.
Thank you for your explanation! I decide to keep my score after reading your rebuttal.
- The method proposed in the paper can avoid extra computation, such as the duplication of concepts.
- The experimental results presented in the paper show that the model performs comparably to models with a larger number of parameters while using fewer parameters, suggesting that the model might perform even better with an increased number of parameters.
- The paper's layout is not aesthetically pleasing, particularly the formatting of Tables 1-3.
- The experiments lack error bars, which diminishes the credibility of the results.
- The experimental results are unsatisfactory and fail to demonstrate the superiority of the model, and the experiments are incomplete.
- The model's results did not achieve state-of-the-art (SOTA) performance in the G2T/T2G tasks.
- Lines 257-259 state, "In conclusion, our structure token approach can achieve comparable performance on text-to-graph generation under similar model size without using special training methods or loss functions." The results in Table 3 are nearly identical to those of Grapher-small (Text) T2G. Combined with the absence of error bars, it is challenging to determine whether the similarities are due to errors or model efficacy, making it difficult to convince.
- Existing results only may demonstrate that the model performs comparably to models with larger parameters under fewer parameters. This does not prove that increasing the number of parameters will enhance performance. A more direct comparison of experimental results is necessary to substantiate this claim. The explanations in Section 5, "Scaling Up," are overly subjective and unconvincing, making the paper seem like a work in progress.
- Please respond to the issues pointed out in the "Weakness" section.
Author addressed the limitations.
No ethics review needed.
We appreciate your detailed review. We address your main points below:
1. We appreciate the mention of the aesthetic of the layout, we will adjust accordingly in the revision.
2. The error bars are not included in the experiments because our model used in each experiment is finetuned from the same pre-trained weights produced by our pre-training method. There are no randomly initialized parameters during the finetuning.
While we do run the same finetuning script 5 times with different random seeds, the scores are identical across 5 runs showing that the randomness in the dataset did not affect the results. The pre-trained weights will also be released with the source code.
3. We acknowledge that the result does not achieve SOTA performance in G2T/T2G tasks.
However, our model achieves the “nearly identical” results using less pre-training data and parameters than Grapher-small (Text).
Besides, our model greatly outperforms CycleGT and Grapher (Query). Combined with the good properties of our method, it demonstrates some interesting features beyond  SOTA performance.
While we “anticipate” the model could scale up, we purposefully put the explanation in the “Discussion” section as a limitation of the current work and explicitly state the necessity of further explorations.
Thank you for explanation. I decide to keep my score.

Meta-review:
role: PC
data: comment: value: The paper proposes ``structure token'' to unify the representation and generation of graph and text. The reviewers are consistently negative towards the work, which could be improved by addressing the following aspects: (1) further enhancing performance to reach state-of-the-art (SOTA) levels, (2) updating the generation tool, (3) including more ablation studies, and (4) scaling up and reporting performance.

Decision: Reject