Title: TextGraphBART: Unifying Graph and Text with Structure Token
Venue: NeurIPS 2024
Accepted: no
Keywords: transformer

text generation

graph generation

text-to-graph

graph-to-text
Abstract: We propose a novel encoding method called ``Structure Token'' to unify the processing and generation of both graphs and texts with a single transformer-based model. This method allows graphs with text labels to be generated by a series of tokens, enabling both graph and text data to be handled interchangeably. By utilizing structure tokens, our model learns a unified representation, enhancing the ability to process diverse data without requiring extra modules or models. Additionally, the model can be trained like most transformer models with simply cross-entropy loss. To demonstrate the effectiveness of our method, we introduce a pre-training scheme inspired by mBART but adapted to leverage structure tokens. Our model, named TextGraphBART, uses the same architecture as normal Transformer Encoder-Decoder models with small modifications on the input and output to accommodate structure tokens. The evaluations show that this approach achieves comparable results against baseline models of similar sizes on both text-to-graph and graph-to-text generation tasks, without needing specialized loss functions or sampling techniques. These findings suggest that our approach can effectively bridge the gap between textual and structural data representations, and the design of encoding method could offer a new direction for future improvement.

Abstract

We propose a novel encoding method called 'Structure Token' to unify the processing and generation of both graphs and texts with a single transformer-based model. This method allows graphs with text labels to be generated by a series of tokens, enabling both graph and text data to be handled interchangeably. By utilizing structure tokens, our model learns a unified representation, enhancing the ability to process diverse data without requiring extra modules or models. Additionally, the model can be trained like most transformer models with simply cross-entropy loss. To demonstrate the effectiveness of our method, we introduce a pre-training scheme inspired by mBART but adapted to leverage structure tokens. Our model, named TextGraphBART, uses the same architecture as normal Transformer Encoder-Decoder models with small modifications on the input and output to accommodate structure tokens. The evaluations show that this approach achieves comparable results against baseline models of similar sizes on both text-to-graph and graph-to-text generation tasks, without needing specialized loss functions or sampling techniques. These findings suggest that our approach can effectively bridge the gap between textual and structural data representations, and the design of encoding method could offer a new direction for future improvement.

1 Introduction

Transformer layers have been proven to work well in several domains beyond text, like audio, image, and even multi-modal data. Some research has also shown that with careful design, transformer layers can extract features from graph data[30, 16]. Graph is a common data structure for representing concepts and relationships. In this work, we focus on a specific type, named text graph, where the concepts and relationships are expressed as texts, such as knowledge graphs and parsing trees. Learning vector representations and generating new text graphs are two essential aspects of text graphs in machine learning. Since texts can be viewed as a chain of words or characters, the text graph then becomes a nested graph. The complexity of handling such a nested graph leads to two major approaches for generating text graphs.

The first strand is the multi-stage approach which generates concepts, relationships, and texts in different steps [22, 12]. The process usually involves multiple models that generate different parts of the text graph. For example, Grapher [22] uses the T5 [24] pre-trained model to generate all the concepts and then uses a relation extraction model to predict the relationships between every two concepts. This approach requires the models to generate a complete graph despite the edge sparsity of the target text graph. Therefore, the model includes a special 'no-relation" relationship and turns every graph into a complete graph thus requiring extra predictions. Since the relation extraction is

...

...

...

id

x N

x N

unmodified Transformer

Encoder-Decoder

autoregressive

token decoding

...

structure encode / decode

structure embed

input projection

structure encode / decode

structure embed

input projection

(a) Model with structure token

...

Transformer Encoder

Akron, Ohio is located ....

encode & embed

input projection

Iteration 0

Decoder & Predictor

[graph]

Iteration 1

country

[graph]

country

{

}

Iteration 2

{

}

[graph]

country

Decoder & Predictor

Akron

Akron

Iteration 3

{

}

Akron

Decoder & Predictor

[graph]

country

Ohio

Ohio

...

Iteration 4

{

}

encode

&

embed

encode

&

embed

encode

&

embed

(b) Autoregressive decoding

done on every two concepts, the model does not consider multi-hop relations. Moreover, the model 35 cannot handle the case where two concepts have more than one relation. The second method is the 36 graph linearization approach that fuses the hierarchy in text graph into chain of tokens [1, 10]. This 37 approach treats the text graph as a special text sequence and enables the direct adoption of Language 38 Model (LM) for text graph generation. The idea can also be applied to learning vector representations 39 of text graphs. For example, BT5 [1] convert the text graph into sequence of (subject, relation, 40 object) triples and train T5 to translate between sentence and sequence of triples. Since sequence 41 generation with LM is done in an autoregressive manner, the generation is conditioned on the 42 already generated triples. This behavior allows the model to handle multi-hop relations and the 43 multi-relational case. Meanwhile, the model does not suffer from generating complete graphs because 44 the model can learn to terminate when the generated triples match the target text graph. However, 45 using sequence of triples also introduces extra complexity to the LM. Since the format requires 46 matching the concepts in triples to reconstruct the text graph, there will be duplications of subject 47 or object in the sequence. Thus, the model generates duplicated texts and cannot handle the case 48 where two concepts are represented by the same text but refer to different things. Also, it relies on 49 the model to implicitly learn the connection between two triples with duplication. Furthermore, LM 50 is neither permutation invariance nor equivariance, which means the prediction alters if the generated 51 triples are being shuffled. 52

The goal of this work is to design a new approach that preserves some of the advantages while 53

avoiding the drawbacks of previous approaches. This sets a few desired properties of the new 54

approach. First, the method should be suitable for both representation and generation. It should also 55

consider the cases that cannot be handled by multi-stage and graph linearization approach. Second, 56

the model should be permutation equivariance and perform generation in an autoregressive manner. 57

Last, the method should avoid extra computation, such as the duplication of concepts. To achieve 58

the desired characteristics, we propose the structure token approach, as illustrated in 1. Our method 59

employs a concept we call 'Structure Token" which losslessly encodes the text graph into a set of 60

tokens. The token contains a word and a few identifiers for the precise location of that (sub-)word 61

in the text graph. Our model incorporates an unmodified Transformer Encoder-Decoder model [26] 62

with structure token embeddings and a structure predictor for predicting new structure tokens. The 63

text graph is generated autoregressively like regular text generation and the generated structure tokens 64

form a subgraph. Once the generation stops, we can decode the set of structure tokens into the 65

66

target text graph. A notable difference between our approach and previous approaches is that our

Transformer model operates on sets instead of sequences and we view text graphs as nested graphs. 67

To our knowledge, our structure token approach is the first method that can autoregressively generate 68

sub-graphs with multi-token labels without modifying transformers. 69

We validate our structure token approach on text-to-graph (T2G) generation and graph-to-text (G2T) 70 generation tasks. By treating sentence as a special text graph without any edges, the generation 71 tasks become a text-graph-to-text-graph translation problem. Therefore, we present TextGraphBART, 72 a Transformer Encoder-Decoder model pre-trained on text-graph-to-text-graph translation with 73 our structure token approach. The model is evaluated on two publicly available parallel datasets, 74 EventNarrative [8] and WebNLG (2020) [5], and achieves comparable results using fewer parameters 75 and pre-training data. 76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

2 Related Work

Recent works for text graph generation primarily focus on reusing pre-trained LMs. BT5 [1] applies the graph linearization approach with T5 [24] pre-trained model. ReGen [10] further proposes a Reinforcement Learning (RL) objective to improve the performance. On the other hand, Grapher [22] uses T5 as an entity extraction model and jointly trains another relation extraction model. It provides two types of implementation: Grapher (Text) and Grapher (Query). The Grapher (Text), which is the state-of-the-art method on WebNLG (2020) [5] dataset, generates entities as a flat sequence, while Grapher (Query) feeds a set of query embeddings to T5 decoder and apply another model to fully decode the outputs to entities. Other works target on the problem of lacking paired datasets for T2G generation. CycleGT [12] apply the cycle-training framework on a G2T model and a multi-stage T2G model. INFINITY [29] apply the cycle-training framework on a single T5 model with graph linearization approach.

Most of the works for learning vector representations only focus on G2T generation. GAP [9] proposes a graph-aware attention that first uses the pooling operator to get the features of each text label and then applies a modified attention operator on those features. KGPT [6] provides two types of encoder: graph encoder and sequence encoder. The graph encoder is based on graph attention network [27] that also operates on the pooled text features. On the other hand, the sequence encoder infuses the structure information into the text tokens and feeds them into a transformer model. This approach resembles our structure token approach upon learning vector representations. However, they convert the graph into a dictionary-like format which suffers from a similar duplication problem as graph linearization approach.

Our method is essentially derived from the design of TokenGT [16], which also converts graphs into sets of tokens containing labels and identifiers. However, their idea does not directly fit in our scenario of text graph for two reasons. First, a single graph element (node or edge) in TokenGT needs to be representable by a single token. On the contrary, it would require multiple tokens for an element of text graph because the label is a multi-token text. Second, TokenGT only focuses on representing the graph, while we are interested in graph generation as well.

3 Method

3.1 Structure Token

The proposed structure token is a data representation that can losslessly encode all data in a text graph as a set of tokens. Given a text graph G = ( N , A ) containing a node set N and an arc set A . Each arc is a triple of a head node, an edge, and a tail node. Each graph element (node and edge) is a unique text label S identifiable with an integer ID. This allows different nodes or edges to have the same text label. The full formal definitions can be found in Appendix A.1. In order to convert text graph to structure tokens, we express the node set and arc set into one unified structure of graph elements. Each graph element will be represented by multiple structure tokens. A structure token consists of seven parts: 1. Label: The (sub-)word token of a graph element. 2. Type: A binary indicator specifying whether this graph element is a node or an edge. 3. Token ID: An unique ID for this token. 4. Previous ID: The token ID of previous token. 5. Segment ID: An unique ID for the graph element. 6. Head ID: The segment ID of the head node. 7. Tail ID: The segment ID of the tail node. If the token is part of a node, the head ID and tail ID will just be the segment ID of itself. With these information, we are able to differentiate between structure tokens that are from different parts of the graph. The text graph is converted into a set of structure tokens. Since the IDs can point to a graph element directly, there is no need for duplications like graph linearization approach. We provide the formal definition of structure token in the Appendix A.2. A real example of text graph

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

[T]

[T]

[T]

[N]

[Domain]

Text Graph

Label

Token ID

Segment ID

Head ID

Tail ID

Previous ID

2

1

10

10

2

1

2

10

2

1

6

10

1

1

2

3

4

1

6

7

10

1

12

8

2

1

3

4

5

6

7

8

11

10

13

9

president of

[E]

president

of

Biden

[N]

United

States

United States

Type

1

1

0

1

11

12

...

...

(a) Example of structure tokens.

Type

Label

Token ID

Segment ID

Head ID

Tail ID

Previous ID

)

Orth

n

(

10

)

OneHot

2

(

0

)

Orth

n

(

7

)

Orth

n

(

2

)

Orth

n

(

6

)

Orth

n

(

8

+

concat

embed

structure

embedding

OneHot

|T |

(

)

of

(b) Converting structure token to embedding.

and corresponding structure tokens can be found in Figure 2a. The idea of type, head ID, and tail ID are inherited from TokenGT [16], which uses identifiers to indicate the connections. We modify their definition and introduce extra identifiers for our text components. The token ID and previous ID are text-level identifiers. The text order is determined by the token ID and previous ID for reconstructing the text label. On the other hand, the segment ID, head ID, and tail ID are graph-level identifiers. For tokens of a specific graph element, the graph-level identifiers of each token will be the same.

Furthermore, We add an extra 'domain token' to the structure tokens of a text graph to indicate the domain of the graph, like the special language token used in multi-lingual translation [15, 21]. With the domain token, we can specify what kind of data the text graph is holding. For example, since text is treated as a text graph without any edges, we use a '[text]' domain token to indicate that this text graph represents a text. Besides, we use the domain token as the first token of every text label, so the previous ID of the first token of all labels are pointing to the domain token.

3.2 Structure Embedding

The structure tokens are transformed into fixed-size high-dimensional vector representations. Each part of the structure token is converted into a vector and concatenated together. Then the vectorized result will be fed to a trainable projection layer for getting the token embedding, as illustrated in Figure 2b. Label and type are converted with one-hot encoding, denoted as OneHot . On the other hand, the IDs need to be handled differently. In order to preserve the graph structure in the tokens with the Transformer model, each ID needs to be converted into orthonormal vectors as proved by TokenGT [16]. We loose the requirement of orthonormality and use a set of orthonormal-like vectors. The dot product value of two different orthonormal-like vector is close to zero or less than some thresholds. These vectors of identifiers enable the attention operation in the Transformer model to be able to aggregate corresponding information through dot product. Each ID would be converted into an orthonormal-like vector through a transform function Orth . We use this transform function to convert the graph-level identifiers directly. On the other hand, we add the vectors of the text-level identifiers together, which allows the attention to aggregate information from neighbor tokens like the absolute position embedding [26]. Details and definitions can be found in Appendix A.3. Unlike position embedding which depends on the location of the tokens in a sequence, the text-level identifiers directly point to the neighbor token no matter their location in the sequence. Meanwhile, sequence orders are defined by IDs and the IDs can be randomly assigned. Therefore, applying any permutation of the input embeddings is equivalent to applying the same permutation on the output hidden states.

3.3 Generation

Text Generation After converting the structure tokens into embeddings, those embeddings are fed into the unmodified Transformer Encoder-Decoder model. Conceptually, our model generates a structure token at each step which contains seven objects. However, we do not really need to generate seven objects at every step. The token ID is unique for every token and we can randomly pick any ID sequence beforehand. Notably, the structure token representation is a set, while the autoregressive

...

hidden

_

state

1:

k

-

1

UniqueID

1:

k

SegmentID

2:

k

+

...

Tail Projection

Head Projection

...

logits

multiply

softmax

masked text

[text]

Transformer encoder

[graph]

[text]

[graph]

text

subgraph

T2T

G2T

G2G

Transformer encoder

Transformer

Decoder

Transformer

Decoder

graph

generation manner makes the generated tokens resemble a sequence. Although the design of structure 159 tokens enables the possibility of non-monotonic order of text generation, we slightly restrict the 160 generation order of the structure tokens from the same graph element to be ordered and contiguous. 161 With this restriction, we do not need to predict the token ID and previous ID. We can use the same 162 generation scheme of other text generation Transformer model that simply generates the next text 163 token until we are done with this element. Meanwhile, since the graph-level identifiers are the same 164 within a graph element, we only need to predict the graph-level identifiers for the first token of labels. 165 The generation can be further simplified for a single sentence since the graph-level identifiers are 166 merely the token ID of the first token. Thus, text generation with our structure token approach is 167 almost the same as other Transformer-based text generation models. 168

Text Graph Generation For text graph generation, the same methodology applies. We use a 169 structure predictor for predicting the identifiers. The graph-level identifiers are the same within a 170 graph element. The prediction of graph-level identifiers can be done only one time per graph element. 171 Moreover, the type and segment ID can also be omitted because we can tell the values once we get 172 the head ID and tail ID. As a result, our structure predictor only needs to predict the head ID and tail 173 ID. For predicting the IDs, we employ a single causal Transformer layer (a layer of the Transformer 174 decoder), as illustrated in Figure 3. The causal Transformer layer takes the output of the Transformer 175 model plus the transformed segment ID to produce a hidden vector. The hidden vector will be fed 176 into two projection layers to get a prediction of the head ID and tail ID. To get the ID, we multiply 177 the final hidden vectors with a list of our orthonormal-like vectors, and perform softmax on the 178 multiplication result to get the predictions. With this setup, we can apply the same teacher forcing 179 technique as other Transformer decoders, so the training process is also parallelized. 180

181

3.4 Pre-Training

We introduce a pre-training method for our model based on the mBART pre-trained model for 182 multilingual text generation [21]. The pre-training method contains two types of training objectives: 183 the self-supervised objective and translation-like objective, as illustrated in Figure 4. The translation184 like objective forces the model to generate tokens depending on the domain token. The self-supervised 185 objective allows us to utilize more datasets without paired data (e.g. plain text datasets or sample 186 subgraphs from a large graph database). By using both kinds of objectives, the effective training data 187 are doubled. Meanwhile, the model is encouraged to learn a more unified representation. With these 188 objectives, we could utilize many different datasets to improve our model. 189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

4 Experiments and Results

Datasets We use four parallel datasets containing both text and text graph for our experiments, as presented in Table 1. The model is pre-trained on TEKGEN [2] and GenWiki [14], and then we fine-tune the pre-trained model on EventNarrative [8] and WebNLG (2020) [5] for evaluating our model on G2T and T2G generation, respectively. The datasets are automatically generated by aligning texts with existing databases, except WebNLG (2020). TEKGEN is a large-scale dataset curated by aligning a subset of the Wikipedia text with Wikidata [28]. GenWiki is another large-scale dataset built on Wikipedia text. The text graphs are collected from DBpedia [3]. EventNarrative is an event-centric dataset that contains text graphs from the EventKG [11] and Wikidata. The text is also a subset of Wikipedia text. WebNLG (2020) is crowd-sourced dataset crafted by human annotators. The text graphs are collected from DBpedia, while the texts are manually written by annotators. It contains 16 categories in the training set and 19 categories (3 extra categories) in the test set. We use the official data split for all the datasets. As for the metrics, we used BLEU [23], METEOR [4], and BERTScore [31] to evaluate the G2T performance, and we use the official evaluation script of WebNLG (2020) to evaluate the T2G performance.

Setups Our model is trained in two phases: pre-training and fine-tuning. We initialize our model from scratch and perform the pre-training method. For pre-training, We use the RAdam optimizer [20] with a learning rate of 0.0001. The model is updated with an effective batch size of 256 and being trained for 5 epochs on a single A100 40GB GPU. All fine-tuning experiments are done on a single RTX 3090 24GB GPU. For fine-tuning on EventNarrative, we use the Lion optimizer [7] with a learning rate of 0.00001. The model is updated with an effective batch size of 128 and trained for 20 epochs. For fine-tuning on WebNLG (2020), we use the Adam optimizer [17] with a learning rate of 0.0001. The model is updated with an effective batch size of 128 and trained for 100 epochs.

We use an overall hidden size of 512 for our model. The unmodified Transformer encoder and decoder both have 6 layers. Each attention has 16 heads, and we use a hidden size of 32 for self attentions and 64 for cross attentions. The feed-forward layer in Transformer has an input and output hidden size of 512, and the intermediate hidden size is 2048. We use these numbers for the structure predictor as well. For the activation functions, we use the GELU activation function [13] for Transformers and hyperbolic tangent function for the projection layers of structure predictor. During pre-training, we apply the dropout [25] on the attention weights and the residual connections with a dropout rate of 0.1. The model weights are randomly initialized with a mean of 0 and a standard deviation of 0.02.

For data processing, we use the same subword tokenizer as T5 which uses the Unigram tokenization method [24, 18]. The tokenizer has a vocabulary of 32100 text tokens, which contain 32000 subword text tokens and 100 reserved special tokens. We use the reserved special tokens for our domain tokens. Each dataset is assigned with a corresponding domain token for the graph data, while all text data from different datasets share the same text domain token. The samples in each dataset are truncated with a maximum length of 128 or 256 text tokens depending on the training stage. A random unique ID sequence is determined for each sample at every epoch. During the pre-training, we randomly

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

assign a unique ID sequence with a maximum value of 512. For the encoder input, we randomly drop 15% of graph elements or tokens depending on the domain.

G2T Results We compared our model with T5 [24], BART [19], and GAP [9]. Both T5 and BART are Transformer Encoder-Decoder models pre-trained on text data and fine-tuned with graph linearization [8], while GAP modifies the encoder of Transformer Encoder-Decoder model with graph-aware modules for extracting graph features [9]. It is noteworthy that all these models use a similar Transformer decoder. The main difference among TextGraphBART and these models is the way we represent and handle the text graph input.

The result is shown in Table 2. In comparison to T5 and BART, our structure token method achieves better score with fewer parameters than graph linearization approach. Meanwhile, our model is comparable with GAP without modifying the Transformer model. As a conclusion, our structure token representations enabled the Transformer model to capture better features from the text graph than the graph linearization approach.

T2G Results We compare our model with CycleGT [12, 5], BT5 [1], and Grapher [22]. BT5 is T5 pre-trained and fine-tuned with graph linearization. On the other hand, both CycleGT and Grapher adopt the multi-stage approach. The CycleGT is a well-known multi-stage approach for text-to-graph generation using cycle training [12], while Grapher performs supervised learning with a special loss function [22]. Meanwhile, we use the officially released source code of Grapher 1 to train a Grapher-small (Text) which has a similar model size (95M) with our model (75M). Both Grapher-small and our TextGraphBART are trained for 100 epochs with the same learning rate and effective batch size.

The result is shown in Table 3. In comparison to CycleGT and Grapher (Query), our simple generation method with structure tokens outperforms models with special training methods. Although our model does not directly match the performance of the large models like BT5 or Grapher (Text), our model is comparable with Grapher-small that has similar model size. Furthermore, we analyze the result by measuring the performance on each category of the WebNLG (2020) test set comparing to Graphersmall. The result is shown in Table 4. Even though Grapher-small is based on the T5-small pre-trained model, which is trained on an extremely large dataset of 750 GB: the Colossal Clean Crawled Corpus

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

(C4) [24], we can see that our model performs slightly better than Grapher-small on unseen categories (0 samples in training set). In conclusion, our structure token approach can achieve comparable performance on text-to-graph generation under similar model size without using special training methods or loss functions.

Ablation Study To investigate the performance contribution of the components of structure tokens, we conducted the ablation study on our structure embedding by fine-tuning our model with the removal of some parts of the embeddings. The model is trained on the WebNLG (2020) with the same setup. The results are shown in Table 5. In all ablations, the model performance was attenuated as expected. First, the ablation of the token ID and previous ID removes the text order information in the text labels hence the degeneration of performance. Similarly, the head ID and tail ID provide the connectivities of the graph. Removal of this embedding decreases the performance, indicating the importance of the connectivities. On the other hand, the ablation of type and segment ID are not as detrimental as others because the type and segment ID may be inferred from other IDs. Thus, our model is still able to perform albeit less performant. In conclusion, the ablation study showed that all of our structure embedding is important for good model performance.

5 Discussion

The primary objective of this work is to demonstrate the effectiveness of the proposed structure token approach. Due to resource constraints, there are numerous aspects remain unexplored.

Scaling Up We believe that TextGraphBART has the potential to achieve better results when scaled up. The backbone of our model is simply the transformer model, like most of the other baselines. Meanwhile, we observed a clear improvement on other baselines when scaling up (e.g. comparing T5-Base and T5-Large for G2T in 2 and Grapher-small and Grapher for T2G in 3). Therefore, we anticipate that transformers on T2G and G2T follow the scaling laws.

Model Architecture While we use The Transformer Encoder-Decoder model in our experiments, there are no strict restrictions on the model architecture as long as there are dot product attention operations. The same approach can be applied to encoder or decoder-only model.

Data and Objective for Pre-Training In the experiments, we only use parallel datasets for pretraining. Since our pre-training scheme contains a T2T path, it's possible to pre-train the model merely with plain-text datasets. On the other hand, we can also incorporate programming language datasets and use syntax parsers to generate the Abstract Syntax Tree (AST) as the text graph for pre-training. However, it is unclear how much the self-supervised objective and translation-like objective affect the downstream performances.

6 Conclusions

We present a novel approach to the problem of text graph generation leveraging the strength of Transformer models. Our exploration has led to an effective method for structured data representation and generation via structure tokens. In the structure token, we use several identifiers to indicate the connectivities of the graphs and the order of the texts. Then an embedding method for structure token is proposed, allowing the Transformer model to utilize the structural information. We show that the structure token approach can be used to represent and generate both texts and text graphs.

The experiment results demonstrated the effectiveness of our method with less data and parameters. 295 Meanwhile, the ablation study further confirmed the importance of various elements of the structure 296 tokens. 297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

References

O. Agarwal, M. Kale, H. Ge, S. Shakeri, and R. Al-Rfou. Machine translation aided bilingual data-to-text generation and semantic parsing. In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+) , pages 125130, Dublin, Ireland (Virtual), 12 2020. Association for Computational Linguistics. URL https://aclanthology.org/2020.webnlg-1.13 .

O. Agarwal, H. Ge, S. Shakeri, and R. Al-Rfou. Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 3554-3565, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.278. URL https://aclanthology.org/2021.naacl-main.278 .

S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives. Dbpedia: A nucleus for a web of open data. In Proceedings of the 6th International The Semantic Web and 2nd Asian Conference on Asian Semantic Web Conference , ISWC'07/ASWC'07, page 722-735, Berlin, Heidelberg, 2007. Springer-Verlag. ISBN 3540762973.

S. Banerjee and A. Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL https: //aclanthology.org/W05-0909 .

T. Castro Ferreira, C. Gardent, N. Ilinykh, C. van der Lee, S. Mille, D. Moussallem, and A. Shimorina. The 2020 bilingual, bi-directional WebNLG+ shared task: Overview and evaluation results (WebNLG+ 2020). In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+) , pages 55-76, Dublin, Ireland (Virtual), 12 2020. Association for Computational Linguistics. URL https://aclanthology. org/2020.webnlg-1.7 .

W. Chen, Y. Su, X. Yan, and W. Y. Wang. KGPT: Knowledge-grounded pre-training for data-to-text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 8635-8648, Online, Nov. 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.697. URL https:// aclanthology.org/2020.emnlp-main.697 .

X. Chen, C. Liang, D. Huang, E. Real, K. Wang, Y. Liu, H. Pham, X. Dong, T. Luong, C.-J. Hsieh, Y. Lu, and Q. V. Le. Symbolic discovery of optimization algorithms. ArXiv , abs/2302.06675, 2023.

A. Colas, A. Sadeghian, Y. Wang, and D. Z. Wang. Eventnarrative: A largescale event-centric dataset for knowledge graph-to-text generation. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , volume 1. Curran, 2021. URL https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/ 2021/file/a3f390d88e4c41f2747bfa2f1b5f87db-Paper-round1.pdf .

A. Colas, M. Alvandipour, and D. Z. Wang. Gap: A graph-aware language model framework for knowledge graph-to-text generation. In International Conference on Computational Linguistics , 2022.

P. Dognin, I. Padhi, I. Melnyk, and P. Das. ReGen: Reinforcement learning for text and knowledge base generation using pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 1084-1099, Online and

Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 345

10.18653/v1/2021.emnlp-main.83. URL https://aclanthology.org/2021.emnlp-main. 346 83 . 347

348

349

S. Gottschalk and E. Demidova. Eventkg: A multilingual event-centric temporal knowledge graph. In Extended Semantic Web Conference , 2018.

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

Q. Guo, Z. Jin, X. Qiu, W. Zhang, D. Wipf, and Z. Zhang. CycleGT: Unsupervised graph-totext and text-to-graph generation via cycle training. In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+) , pages 7788, Dublin, Ireland (Virtual), 12 2020. Association for Computational Linguistics. URL https://aclanthology.org/2020.webnlg-1.8 .

D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv: Learning , 2016.

Z. Jin, Q. Guo, X. Qiu, and Z. Zhang. GenWiki: A dataset of 1.3 million content-sharing text and graphs for unsupervised graph-to-text generation. In Proceedings of the 28th International Conference on Computational Linguistics , pages 2398-2409, Barcelona, Spain (Online), Dec. 2020. International Committee on Computational Linguistics. doi: 10.18653/v1/2020.coling-main.217. URL https://aclanthology.org/2020.coling-main.217 .

M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu, Z. Chen, N. Thorat, F. B. Viégas, M. Wattenberg, G. S. Corrado, M. Hughes, and J. Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics , 5:339-351, 2016.

J. Kim, D. Nguyen, S. Min, S. Cho, M. Lee, H. Lee, and S. Hong. Pure transformers are powerful graph learners. Advances in Neural Information Processing Systems , 35:14582-14595, 2022.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR , abs/1412.6980, 2014.

T. Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 66-75, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1007. URL https://aclanthology.org/P18-1007 .

M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7871-7880, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.703. URL https://aclanthology.org/2020.acl-main.703 .

L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han. On the variance of the adaptive learning rate and beyond. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https: //openreview.net/forum?id=rkgz2aEKDr .

Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics , 8:726-742, 2020. doi: 10.1162/tacl_a_00343. URL https://aclanthology.org/2020.tacl-1.47 .

I. Melnyk, P. Dognin, and P. Das. Knowledge graph generation from text. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 1610-1622, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.findings-emnlp.116 .

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pages 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/1073083.1073135. URL https://aclanthology.org/P02-1040 .

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1-67, 2020. URL http://jmlr.org/papers/v21/ 20-074.html .

N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research , 15 (56):1929-1958, 2014. URL http://jmlr.org/papers/v15/srivastava14a.html .

A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NIPS , 2017.

P. Veliˇ ckovi´ c, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph attention networks. 6th International Conference on Learning Representations , 2017.

D. Vrandeˇ ci´ c and M. Krötzsch. Wikidata: A free collaborative knowledgebase. Commun. ACM , 57(10):78-85, sep 2014. ISSN 0001-0782. doi: 10.1145/2629489. URL https: //doi.org/10.1145/2629489 .

Y. Xu, S. Sheng, J. Qi, L. Fu, Z. Lin, X. Wang, and C. Zhou. Unsupervised graph-text mutual conversion with a unified pretrained language model. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 5130-5144, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.281. URL https://aclanthology.org/2023.acl-long.281 .

C. Ying, T. Cai, S. Luo, S. Zheng, G. Ke, D. He, Y. Shen, and T.-Y. Liu. Do transformers really perform bad for graph representation? In Neural Information Processing Systems , 2021.

T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https:// openreview.net/forum?id=SkeHuCVFDr .

A Formal Definitions

A.1 Preliminaries

Let T be the set of all possible text tokens. A text label is defined as a set of pairs containing the text token and positions. We generalize this definition by replacing the position with a contiguous sequence of unique ID. Given an infinite unique ID sequence: I = (id i ) ∞ i =0 where id i ∈ Z + ∧ id i = id j if i = j . Each id i is a positive integer. We also define id( i ) = id i for simplicity. By picking a corresponding ID sequence, we can use any positive integer sequence as the positions. Then a text label S of length l is a set of token-ID pairs defined as: S = { ( t i , id j + i ) | 1 ≤ i ≤ l, t i ∈ T } ⊆ T × Z + . We can conditionally specify the start point j ∈ N with S j . Let S be the set of all possible text labels and S j for a specific start point. With the ID sequence I , the positions ( p ) l p =1 can be replaced with (id j + k ) l k =1 . Then we can union text labels without missing information by picking non-overlapped ID sequences, which is a desired property for the attention operation. A text graph G = ( N , A ) is composed of a node set N with q nodes and an arc set A with r arcs. The node set N is a set of node labels paired with unique IDs of the nodes, defined as: N = { ( N i , n i ) | 1 ≤ i ≤ q, n i ∈ Z + , N i ∈ S} ⊆ S × Z + where N i is the node label and n i is the corresponding node ID. Similarly, an edge set E is a set of edge labels paired with unique IDs, defined as: E = { ( E i , e i ) | 1 ≤ i ≤ r, e i ∈ Z + , E i ∈ S} ⊆ S × Z + where E i is the edge label and e i is the edge ID. Notably, the ID used in N and E are disjoint. Then the arc set is defined as: A = { ( N h i , E i , N t i ) | 1 ≤ i ≤ r, N h i , N t i ∈ N , E i ∈ E} ⊆ N × E × N where E i is the edge and N h i , N t i is the head node and tail node, respectively.

̸

̸

A.2 Structure Tokens 443

444

445

446

447

448

449

450

451

452

With the setup in A.1, the formal definition of the structure token representation is defined as a set of septuples (tuples with 7 elements). Given a text graph G = ( N , A ) and its components: node N i = ( N i , n i ) ∈ N , 1 ≤ i ≤ |N| , arc A j = (( N h j , n h j ) , ( E j , e j ) , ( N t j , n t j )) ∈ A , 1 ≤ j ≤ |A| , edge E j ∈ E = { ( E j , e j ) | 1 ≤ j ≤ |A|} , and the length of the text labels l N i = | N i | , l E j = | E j | , we define two sequences:

̸

̸

We assign n i = id( L N i +1) , e j = id( L E j +1) and specify the start point of each text label such that N i ∈ S L N i , E j ∈ S L E j . By doing so, we can get the node (or edge) ID from the token IDs of its text label. For each node N i and edge E j , we define the corresponding structure token representation X N i and X E j as:

Then the corresponding structure token representation G ′ of the text graph G is defined as: 453

where t D ∈ T , id 0 is the ID of domain token and X D = { ( t D , 1 , id 0 , id 0 , id 0 , id 0 , id 0 ) } is the domain token. Each septuple X ∈ G ′ is a structure token containing the label, type, token ID, previous ID, segment ID, head ID, and tail ID. With this definition, we can represent every possible token ID assignment by specifying the unique ID sequence I . On the other hand, since the graph element can be randomly permuted, every possible ordering is also representable with our set G ′ by picking the corresponding unique ID sequence.

A.3 Structure Embedding

To convert the structure tokens into embeddings, we use 3 kinds of transform functions. For label and type, we use the one-hot encoding, denoted as OneHot n : A → E n where A is a set with n elements and E n is the standard basis of R n . On the other hand, each ID would first be converted into a d -dimensional orthonormal-like vector with a function Orth d . To get the orthonormal-like vectors, we modify and normalize the sinusoidal position encoding of Transformer [26] with different frequencies. The d -dimensional sinusoidal position encoding PE d at position i is defined as:

where ∥ denotes the vector concatenation. We generalize the definition of PE d with a frequency 467 function f : 468

Then by taking f ′ ( k ) = 10000 -k/d , the original PE d can be defined with PE d = PE ∗ d { f ′ } . We 469 use this generalized position encoding to define the function Orth d : Z + → R d of the IDs for our 470

454

455

456

457

458

459

460

461

462

463

464

465

466

3.0×104

1/10000~(i/d)

-log(i)

2.0×104

1.0×104

0

-0.25

0.00

0.25

0.50

0.75

1.00

orthonormal-like vectors as: 471

where norm 2 is the L2 normalization and ◦ denote the function composition. We find that by picking 472 the frequency function -log( k ) , the generated vectors satisfied the desired properties. In Figure 5, 473 we generate 1024 vectors by applying Orth on 1 ≤ i ≤ 1024 and compute the cosine similarity of 474 every possible pair. We can see that the similarity values are mostly close to zero. 475

̸

Once we can convert the IDs into orthonormal-like vectors, we use Orth d directly as the transform 476 functions for the graph-level identifiers. For text-level identifiers, we add the vector of token ID and 477 previous ID together. Given two non-domain token a and b with token ID t a , t b and previous ID 478 p a , p b , we have Orth (t a ) · Orth (t b ) ≈ 0 since token ID is unique and Orth is orthonormal-like. 479 Meanwhile, we have p a = p b for most tokens. The dot product value of a and b become: 480

where ε is a small value around 0 . The value will only be meaningful in the dot product if one token 481 is the previous token of the other. The case of first tokens is set to allow exchanging information with 482 the domain token. This can be suppressed by making Orth (id 0 ) = 0 . 483

With these designs, we define our structure token vectorize function V ec as: 484

where π i ( X ) denote the i -th element of the septuple X . The vectorized result will be fed into a train485 able projection layer Emb : R 4 n + |T | +2 → R d to get the structure embedding with d dimensions. 486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The proposed methods and results are described in the Method Section (Section 3) and the Experiments and Results Section (Section 4).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in the Discussion Section (Section 5)

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This work does not introduce any new theoretical results Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experiment setups are described in the Experiments and Results Section (Section 4) and code is also provided.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

Answer: [Yes]

Justification: code is provided

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experiment setups are described in the Experiments and Results Section (Section 4)

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Due to resource constraints, we are unable to perform significance tests.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The experiment setups are described in the Experiments and Results Section (Section 4)

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The content does not violate NeurIPS Code of Ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The main result of this work is a new encoding method, which has no societal impact

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: All datasets used are publicly available open datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All dataset and code used are publicly available and cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This work does not release new assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to integrate the processing and generation of both text and graph data using a single transformer-based model. Structure Token encodes graphs with text labels into a sequence of tokens, enabling the handling of both data types interchangeably. This approach leverages a unified representation within a Transformer Encoder-Decoder model, enhanced to incorporate structure tokens. They evaluate TextGraphBART on text-to-graph (T2G) and graph-to-text (G2T) tasks, demonstrating comparable results to baseline models with fewer parameters.\n\nSTRENGTHS:\n1. The introduction of structure tokens is a significant advancement, providing a unified method for processing and generating both text and graph data.\n2. The method integrates seamlessly with existing Transformer architectures, requiring only minor modifications, and avoids the need for specialized loss functions or additional modules.\n3. Empirical results show that TextGraphBART achieves comparable performance on both T2G and G2T tasks with baselines but with fewer parameters.\n\nWEAKNESSES:\n1. The baseline settings are weak. It lacks of comparing with some strong baselines with advanced graph-structured-aware methods [1][2].\n2. The model settings are limited. This paper only try one size of model, and further experiments with larger model sizes are needed.\n3. The provided ablation study, though useful, could be more detailed, exploring the interactions and contributions of various components more comprehensively.\n\nReference: \n[1] Stage-wise Fine-tuning for Graph-to-Text Generation\n[2] Self-supervised Graph Masking Pre-training for Graph-to-Text Generation\n\nQUESTIONS:\nMany research focuses on reduce the gap between textual and structural data representations, as I mentioned in Weakness. What are your advantages over other methods?\n\nLIMITATIONS:\nYes. The limitations are discussed in Discussion section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TextGraphBART. Its a new method of encoding graph/text-input by using a structure token. This new token should preserve graph structure as opposed graph linearization or cycle training. This should also allow for the generation of graphs, with accompying text tokens, without making architectural changes to the transformer. It is then verified on both Graph2Text and Text2Graph tasks for commonly used benchmarks such as WebNLG and EventNarrative. It is followed by results that show comparable results to previous works, and a short discussion and conclusion.\n\nSTRENGTHS:\n1. Paper is generally nice to read\n2. The work considers a lot of relevant related work, which they both refer to or base decisions on\n3. The build-up of the structure token is well explained\n\nWEAKNESSES:\n1. The work proposes a seamless integration of both text and graph inputs, but only verifies on strictly one-directional modal evaluation: Either text-to-graph or graph-to-text. However, as the authors claim that this is the \"first method that can autoregressively generate sub-graphs with multi-token labels without modifying transformers.\", a single text-to-graph benchmark feels like a very shallow evaluation.\n2.  Models are evaluated on datasets that already work well. This makes it hard to distinguish the added value of such a structure token. Scenarios are sketched where e.g. graph linearization is treated simply as a sequence of tokens, but then it this work should methodologically be verified on datasets where preserving graph structure should be required. This is now not the case.\n3.  This is reflected in the results: they are comparable, so the structure token might aswell not be there. The proclaimed difference that this is done with less parameters I would argue is a very weak claim as the difference is not that big.\n4.  Finally, I am not convinced that the proposed \"structure token\" is a truely lossless way of preserving graph structure through-out a transformer. The structure token and its generation is explained in Figure 2, and is claimed to be lossless. However, this is only true in the operations before the embedding leveraging the orthogonal property. After the embedding into a structure embedding, information will (seemingly) be lost about structure in this embedding. From this structure embedding, it cannot be determined what the original structure of the graph was?\n5.  In the discussion the authors mention scaling, but there is no clear evidence for this as the authors only tested one model size. This is essentially an unsubstantiated claim.\n 6. Why is the domain token needed? This is not further addressed or experimented with.\n\nQUESTIONS:\n1. Why is it that this structure token obsoletes the need for transformer modifications?\n2. Why is it interesting that this method obtains comparable results?\n3. Line 56: \"It should also consider the cases that cannot be handled by multi-stage and graph linearization approach\", so what are these cases and how are they not handled by these methods but they are by TextGraphBart?\n\nLIMITATIONS:\nThe authors partially address this limitations by motivating the paper in the introduction. However, I dont feel the discussion (or the rest of the work) addresses the actual limitations of this method. What are pros or cons compared to transformer modifications?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new unified graph-text generation framework, TextGraphBART, for the large language model. The paper tries to address both the generation and representation of text and graphs. The paper proposes a new structure token to encode text graphs into a set of tokens. The structured token can encode the text graphs and be decoded into text graphs. Specifically, it consists of seven different embeddings, including position, domain, and text information. The paper pretrained the proposed framework over four different tasks, including text2text, graph2text, text2graph, and graph2graph. The model is pretrained on TEKGEN and GenWIKI and tests on the event narrative and WebNLG. The performance is evaluated on BLEU, METEOR< and BERTScore. The paper compares the model with T5, BART, GAP, CYcleGT, ZBT5, and Grapher. The paper also includes an ablation study and analyzes the future directions.\n\nSTRENGTHS:\n1. The motivation of the paper is clear and the topic of the paper is interesting. The paper investigates an important unified graph-text generation framework for the language model. The idea of structure tokens is interesting and can contribute to future research. Additionally, the model seems to achieve good results while using fewer parameters.\n2. The paper tests on both graph-to-text and text-to-graph datasets. The proposed method achieves better results compared to other baselines. The paper also conducts an ablation study to investigate the contribution of each component. It includes a discussion session to explore the paper's current limitations.\n3. The paper provides code and implementation details.\n\nWEAKNESSES:\n1. The proposed framework is incremental. Multiple previous papers have used the idea of using different position embeddings to represent the structure information [1,2,3]. The idea of joint text-to-graph and graph-to-text pretraining/generation is also not new [4]. \n2. The experiment is not comprehensive. The paper used eventnarrative for the graph-to-text generation. However, compared to WebNLG  (2020), the EventNarrative has fewer baselines and most of its baselines are outdated. The paper must also add WebNLG(2020) as an additional graph-to-text generation tool to show its contributions. The pretraining datasets and the testing datasets are all Wiki-style datasets. The performance gain may come from pretraining (data leakage) instead of the actual model architecture. The paper needs to include an additional ablation study to show the gain of the eventnarrative/webnlg comes from the model architecture, or also pretraining the proposed baselines on TEKGEN/GenWiki. Otherwise, the scores are not comparable. Additionally, the paper needs to include some of the latest frameworks like [4] since all of the baselines used in the paper are old. The paper also needs to include some human evaluation or qualitative analysis to help readers understand the generation results better. Furthermore, the comparison in Table 4 is not fair, since domain-specific pretraining is more useful than general pretraining [5].\n3. The paper puts important information in the Appendix while not reaching the page limit (9 pages). \n\n[1] Herzig, J., Nowak, P. K., Müller, T., Piccinno, F., & Eisenschlos, J. M. (2020). TaPas: Weakly supervised table parsing via pre-training. ACL 2020.\n\n[2] Wang, Q., Yavuz, S., Lin, V., Ji, H., & Rajani, N. (2021). Stage-wise fine-tuning for graph-to-text generation. arXiv preprint ACL 2021 SRW.\n\n[3] Chen, W., Su, Y., Yan, X., & Wang, W. Y. (2020). KGPT: Knowledge-grounded pre-training for data-to-text generation. EMNLP 2020.\n\n[4] Wang, Z., Collins, M., Vedula, N., Filice, S., Malmasi, S., & Rokhlenko, O. (2023). Faithful low-resource data-to-text generation through cycle training. ACL 2023\n\n[5] Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020). Don't stop pretraining: Adapt language models to domains and tasks. ACL 2020.\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nThe paper includes a limitation section in the Appendix and a Discussion section in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The paper highlights the limitations of two existing methods for generating text graphs: 1. The multi-stage approach does not consider multi-hop relations and cannot handle the case where two concepts have more than one relation. 2. The graph linearization approach introduces extra complexity to the Language Model (LM) and the predictions are altered if the generated triples are shuffled.\n- Building on this, the paper proposes the Structure Token method. Specifically, it identifies a graph element (node or edge) using seven parts. Then, it transforms Structure Tokens into embeddings using OneHot and orthonormal-like vectors, which are then input into a Transformer Encoder-Decoder model.\n\nSTRENGTHS:\n- The method proposed in the paper can avoid extra computation, such as the duplication of concepts.\n- The experimental results presented in the paper show that the model performs comparably to models with a larger number of parameters while using fewer parameters, suggesting that the model might perform even better with an increased number of parameters.\n\nWEAKNESSES:\n- The paper's layout is not aesthetically pleasing, particularly the formatting of Tables 1-3.\n- The experiments lack error bars, which diminishes the credibility of the results.\n- The experimental results are unsatisfactory and fail to demonstrate the superiority of the model, and the experiments are incomplete.\n  - The model's results did not achieve state-of-the-art (SOTA) performance in the G2T/T2G tasks.\n  - Lines 257-259 state, \"In conclusion, our structure token approach can achieve comparable performance on text-to-graph generation under similar model size without using special training methods or loss functions.\" The results in Table 3 are nearly identical to those of Grapher-small (Text) T2G. Combined with the absence of error bars, it is challenging to determine whether the similarities are due to errors or model efficacy, making it difficult to convince.\n  - Existing results only may demonstrate that the model performs comparably to models with larger parameters under fewer parameters. This does not prove that increasing the number of parameters will enhance performance. A more direct comparison of experimental results is necessary to substantiate this claim. The explanations in Section 5, \"Scaling Up,\" are overly subjective and unconvincing, making the paper seem like a work in progress.\n\nQUESTIONS:\n- Please respond to the issues pointed out in the \"Weakness\" section.\n\nLIMITATIONS:\nAuthor addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-13"}