A Branching Decoder for Set Generation
Authors: Zixian Huang
Gengyang Xiao
Yu Gu
Gong Cheng
Venue: ICLR 2024 poster (2024)
Keywords: set generation
keyphrase generation
Primary area: generative models
TL;DR: A novel decoding scheme that can generate a dynamic number of tokens at each time-step and branch multiple generation paths in parallel to reduce inference time.

Abstract: Generating a set of text is a common challenge for many NLP applications, for example, automatically providing multiple keyphrases for a document to facilitate user reading. Existing generative models use a sequential decoder that generates a single sequence successively, and the set generation problem is converted to sequence generation via concatenating multiple text into a long text sequence. However, the elements of a set are unordered, which makes this scheme suffer from biased or conflicting training signals. In this paper, we propose a branching decoder, which can generate a dynamic number of tokens at each time-step and branch multiple generation paths. In particular, paths are generated individually so that no order dependence is required. Moreover, multiple paths can be generated in parallel which greatly reduces the inference time. Experiments on several keyphrase generation datasets demonstrate that the branching decoder is more effective and efficient than the existing sequential decoder.

Published as a conference paper at ICLR 2024
A BRANCHING DECODER FOR SET GENERATION
Zixian Huang, Gengyang Xiao
Yu Gu The Ohio State University Columbus, USA gu mu.8260osu.edi
State Key Laboratory for Novel Software Technology Nanjing University, Nanjing, China maxiann gyxiao ( @mma nju. edu. cn
Gong Cheng
State Key Laboratory for Novel Software Technology Nanjing University, Nanjing, China gehengenju.ent jed.ee.nnuuuuu...mm.
ABSTRACT
Generating a set of text is a common challenge for many NLP applications, for example, automatically providing multiple keyphrases for document to facilitate user reading. Existing generative models use a sequential decoder that gener- ates single sequence successively, and the set generation problem is converted to sequence generation via concatenating multiple text into a long text sequence. However, the elements of are unordered, which makes this scheme suffer from biased or conflicting training signals. In this paper, we propose a branch- ing decoder, which can generate dynamic number of tokens each time-step and branch multiple generation paths. In particular, paths are generated individ- ually so that no order dependence is required. Moreover, multiple paths can be generated in parallel which greatly reduces the inference time. Experiments on several keyphrase generation datasets demonstrate that the branching decoder is more effective and efficient than the existing sequential decoder.
INTRODUCTION
In the past few years, the research of generative models has greatly promoted the development of AI. Pioneering studies, such as those represented by BART (Lewis al. 2020) and T5 (Raffel al.) 2020). demonstrate that tasks with diverse structures can be seamlessly transformed into a uni- fied text-to-text framework. This development has allowed generative models to transition from a task-specific orientation to more versatile, general-purposs capability. The introduction of models such as GPT-3 (Brown et al. (020)) further solidified this trend within both academic and indus- trial spheres. Consequently, there has been a pronounced push towards delving into more nuanced tasks that generative models can address. A prime example is set generation (Zhang et al.] 2019; Madaan et al., 2022), wherein the model is entrusted with generating variable number of target sequences. Such functionality is imperative in contexts such as a news application displaying mul- tiple keyphrases in long document assist reading (Gallina al. 2019) or question answering system offering multiple responses to a user query (Li et al. 2022).
Building the foundational text-to-text paradigm, many current methods for set generation have naturally evolved to adopt the One2Seq scheme (Yuan et al.) 2020 Meng et al. 2021 Madaan al. 2022). Within this scheme, all text from the set concatenated into singular, extended sequence, upon which the model is trained for generation. The One2Seq scheme specifically em- ploys a sequential decoder, generating multiple answers successively in an autoregressive fashion (see [T. While several investigations explore optimizing the concatenation order (Ye et al., 2021
Cao & Zhang 2022). they invariably retain the use of sequential decoder.
Challenges. However, we argue that the prevailing One2Seq approach is not optimal for set gen- eration, presenting limitations during both the training and inference phases of model development.
"Corresoonding author.

---

Published as a conference paper at ICLR 2024
Sequential Decoder
Manchester United <wpp>
Arsenal
<arp>
Manchester City <xpp> Tottenham HotSpur...
Encoder
Manchestar outclassed North London Prenier Leigue poals frenzy, with United thrnhhing A rssmal, 8-2, and City routing Tottenham, 5-1...
Brunching
Decoder
Manchester
Arsenal
Tottenham
United
HotSpur
Figure 1: A comparison of traditional sequential decoder and our branching decoder for case of set generation keyphrase generation task.
The training phase is marked by void in universally recognized canonical sequence concatena- tion order. Distinct sequences, resulting from varied arrangements of set elements, can introduce divergent and conflicting training signals. This inconsistency disrupts the optimization trajectory, potentially limiting the model's performance (Meng et al.] 2021: Ye et al. 202I). During infer- ence, generating a sequence encompassing all target text proves to be considerably time-consuming.
further underscoring the inefficiencies of the current approach (He & Tang 2022).
Our contributions. To address the noted limitations inherent in the sequential decoder, intro- duce a novel decoding scheme, termed One2Branch. This scheme is meticulously crafted to enable the generation of multiple sequences within a set concurrently. As shown in the lower segment of Figure [ our branching decoder generates each keyphrase individually, foregoing the need to con- catenate them into a singular extended sequence. Our novel design circumvents the challenges of ambiguous concatenation order that beleaguer the sequential decoder. In addition, by generating each sequence in the set separately, the model can capitalize on parallel computation, significantly enhancing generation speed. Our methodology is anchored by the branching decoder, adept at sys- tematically delincating multiple paths from a specific input sequence, where each path uniquely signifies a sequence within the target set. A cornersoone of our contribution is the introduction of a unified framework that harmoniously integrates the ZLPR loss (Su et al., 2022) for model train- ing (Section 3.3) and threshold based decoding algorithm for inference (Section B.4). This fusion ensures that, at every decoding step, our model can selectively identify a dynamic set of tokens with logits exceeding a designated threshold (i.e., 0), facilitating the emergence of new generative branches. In a thorough evaluation spanning three representative keyphrase generation benchmarks, One2Branch unequivocally outpaces the established One2Seq techniques, yielding both augmented performance and heightened efficiency. Our methodology, scamlessly integrable with existing gen- erative models, holds potential to redefine the paradigm for set generation using generative models.
Code: http://ppggggcc.....o//o/ooooeae
2 RELATED WORK
Text generation. Text generation has made great progress in many aspects in recent years, such
as different model architectures (Brown et al. 2020 Lewis et al., 2020 Du et al., (02T) and rich
optimization techniques (Li Liang| 2021 Goodwin al. 2020 Houlsby et al., 2019). Existing models are usually based on the text-to- framework (Raffel et al. 2020). I.c. the input and output of the model are unstructured text. Some works adapt the encoder to structured inputs, such as text set (Izacard & Grave, 2021), table (Liu et al.] (019). and graph (Zhao et al., 2020), but their decoder still generates an unstructured text sequence. Some works (Zhang et al. 2011 8 Wen et al.) 2023) use multiple decoders to generate different information, but they essentially adopt multi-view scheme to obtain a single enhanced text sequence rather than multiple text as our branching decoder. Some other works (Vijayakumar et al., 2016 Holtzman et al.) 2020) study decoding strategies to obtain multiple diversified text outputs, but these unsupervised strategies need a manually specified number of generated paths. In contrast, our branching decoder is optimized to generate multiple sequences
2

---

Published as a conference paper at ICLR 2024
in the training stage, and the decoding strategy in the inference stage is consistent with the behavior of training. so as to automatically determine the number of sequences generated.
Set generation. Many tasks in the NLP community can be regarded set generation, such as keyphrase generation (Yaan et al., 2020, Ye et al.) 202T). named entitiy recognition (Tan et al. 2021], multi-label classification Yang et al. 2018 Cao & Zhang 2022), and multi-answer ques- tion answering (Huang et al.] 2023a b). Harly work used the One2One scheme (Meng et al.) (017)) to deal with set generation, which couples each target sequence with its input sequence to form an individual training sample. However, One2One relies on manually setting beam size to generate se- quences, which results in either sacrificing recall to only generate small number of sequences with the highest probability, or sacrificing precision to sample many sequences with a large beam num- ber (Meng et al., 2021). Similar limitations also appear in some work based on non-autoregressive decoders Tan et al., 2021: Sui et al. 202T that can only generate set of fixed size.
Current research on set generation is mainly based on the One2Seq scheme (Yuan et al. 2020), which concatenates multiple sequences into one long sequence. However, since set is unordered, this scheme is easy to introduce order bias during training (Meng et al. [02I]. The main idea to alleviate this problem is to reduce the order bias in training by finding a more reasonable concatena- tion order, which includes designing heuristic rules (Yang et al., 2018), using the reward feedback of reinforcement learning (Yang et al. 2019), and selecting the optimal matching between the gen- erated sequence and the target sequence to calculate the loss (Ye et al. 2021 Cao & Zhang, 2022). Another way enhance One2Seq is data augmentation, which adds sequences representing differ- ent concatenation orders to the training set (Madaan et al. 2022). Although these methods have been verified to alleviate the problem of order bias during One 's training on some tasks, it arguable whether a particular optimal order exists for tasks such as keyphrase generation. Compared with the above work, our One2Branch scheme not only can generate an unfixed number of sequences like One2Seq. but also is not troubled by the disorder of the set like One2Seq.
ONE2BRANCH: A BRANCHING DECODER
3.1 OVERVIEW
Given an input sequence X = .I,...,,8 X₁ with 1 tokens, the goal of set generation to gen- erate a target set Y = .Y,.... Y.) containing target sequences, where each text output s,m, contains m, tokens. A vocabulary V Vi,..., V₂ containing is tokens is predefined and all generated tokens are selected from it. The operator Index(-) is used to map tokens to their index positions in the vocabulary, i.c. Ind) =
Y ......., Y,
Figure gives example of the generation strategy of our branching decoder, containing 3 gen- eration paths that correspond to target sequences. Compared with the existing generative model using a sequential decoder, the branching decoder has two characteristics: (1) generating a dynamic number of tokens at each time-step. and (2) generating multiple target sequences in parallel.
Specifically, different from sequential decoder that can only select fixed beam number of tokens, branching decoder uses a thresholding strategy to allow selecting a dynamic number of tokens, so one or more new generation paths can be branched each time-step. As shown in Figure 2 branching decoder can generate 2-3-3-1 tokens respectively at time-step I to 4. Benefitting from the ability to dynamically branch, multiple target sequences can be generated in parallel. In Figure 2 the branching decoder can use 1-2-3-1 decoders to generate multiple target sequences in parallel at time-step to 4. Compared with the traditional sequential decoder that needs to use a total of 12 time-steps (including 2 separating tokens) to generate all target sequences, the branching decoder only needs to use time-steps since multiple sequences can be generated parallel.
3.2 ARCHITECTURE
Our One2Branch scheme uses the classic encoder-decoder architecture, but the output of the encoder can be fed into multiple parameter-shane sharing decoders to generate parallel. A stacked transformer encoder first encodes the input sequence to obtain its hidden states representation:
HE Encoder(X).
3

---

Published as a conference paper at ICLR 2024
Time-stpp
Time-tipp
Time-step
Time-sepp
Generated Text
BA B
BA
BA BCA
BCA
BA
BA
bos
BCC
BCA
BCA
C.B
Decoder
Decoder
Decoders
Number
Figure 2: An example of branching decoder for set generation. It uses 0 as the threshold to select nodes (blue) each time-step and generates the set ("B A", CA, "CB").
where H [xdd and d denotes the dimension of representation.
At the 1-th time-step, for the i-th generated path with / - 1 generated tokens P the representa- tions of the 1-st 1-th time-steps be obtained as:
S..t, H = Recoder(PP...-1 H
(2)
where Decoder(:,)) is an autoregressive decoder composed of stacked transformer, S E R and Hi,, 1xd contain generation scores and representations tively.. For the f-th token, its generation scores S E Si,, is vector containing the score of each token in the predefined vocabulary V. For the tokens with a score greater than the threshold, they will be selected as generated tokens and each of them will branch new path the next time-step. In the following, we will introduce our training strategy, which enables a fixed threshold to decode dynamic number of tokens and branch out to multiple paths generate in parallel.
tokens on the i-th path, respec-
3.3 TRAINING
Our training strategy consists of: sharing decoder, learning threshold, and negative sequence.
Sharing decoder. The branching decoder should generate multiple unordered target sequences in parallel, adopt sharing decoder way to train the model. As shown in Figure 3 each target sequence is independently input into decoder with shared parameters using teacher forcing manner obtain the generation score of each token in the vocabulary, which implemented by Equation 2
S. H. Decoder(((bos)). Yi). HE.,
where S, E (((||+1)* and H, E (1X are the generation scores and representations of all the |Y| + 1 tokens on the i-th path, respectively, [ is the operator of concatenation, and (bos) is special token used to represent the beginning of generated sequence.
Although each sequence is input to decoder independently. their label sets are related to each other. As illustrated in Figure 3 when the decoder inputs the common prefix (bos) of all the target sequences, because both "B" and "C" are tokens that can be generated next, the positive label sets are 2 Index( (B",, "C"}) at time-step 1 for every decoder i. Similarly, when the decoder inputs the common prefix "(bos) B" of the target sequences "B A" and "B C A" the positive 1 and i 2. For each
(B",,
label sets are 2 Index( ("A", "C"}) at time-step 2 for decoders target sequence at each time-step. the negative label set is obtained by removing the positive label set from the vocabulary, i.e. 2i Endex(()))[2)
4

---

Published as a conference paper at ICLR 2024
Positive
ThresoodO
Negative labels
Shared Demder
Shared Decader
Shared Decoder
Shared Decoder
[m]
A
242
bos
Terget Sequenoes
Negalive Sequancce
Figure 3: An example of One2Branch scheme's training strategy. For the target set {"B A", "B CA, "C B"), One2Branch uses three decoders with shared parameters to learn from each target sequence in the teacher forcing manner. A negative sequence "A B" is also used to train. For each input token, the scores positive tokens in the vocabulary are optimized to be greater than the threshold 0, and the scores of the remaining negative tokens optimized to be less than O.
If two target sequences and Y, have a common prefix up to time-step t but differ at time-step t + 1, then the positive label sets of the two sequences at time-step t separately include Y1,+1 and Ya.+1 This approach of constructing labels does fix the number of positive and negative labels, which requires the model to learn to predict a dynamic number of tokens at each time-step.
Learning threshold. In order realize the selection of dynamic number of tokens at each time- step to branch multiple paths, we expect that the model learns threshold to qualify tokens that can be selected. Benefiting from the work of Su et al. (2022) used in multi-label classification task, we introduce the ZLPR loss to generative model to learn the threshold at O:
L = log(1 + Σ
8 log(
8,
1=1
f=1
(4)
where
Σ WEnt
exp( Si,(-S,t..))
exp exp(S.,,,))
WEN
where n is the size of the target sequences set, m; is the length of target sequence Y, and Si... E R the generation score of token V at time-step supervise model training, both s, 8, E (0, +oo] will be optimized to be minimal, meaning the scores of position tokens in Ω will be optimized toward positive infinity away from O, and the scores of negative tokens in Ω- will be optimized toward negative infinity away from 0, which enables using the threshold in the inference stage.
the i-th path. When using Equation to
Negative sequence. Recall that a common way to train decoder is to calculate the loss from positive sequences using teacher forcing, which often leads to exposure bias and affects model gen- eralizability (Bengio et al. 2015). To address it, we incorporate loss from negative sequences to supervise the model in distinguishing between positive and negative sequences.
From the first incorrect token of negative sequence, for all the subsequent tokens, construct their label sets 25,, = Ø, S = In)ex(VV.. As Figure shows, "A B" a negative sequence; "A" and "B" in the second and third time-steps negative tokens with empty positive label sets.
In practice we train branching decoder in two stages. In the first stage, only target sequences are used. In the second stage, first use the model trained in the first stage predict on the training set and collect the incorrectly generated sequences having the highest scores as hard negatives, and then further train branching decoder with both target sequences and negative sequences.
5

---

Published as a conference paper at ICLR 2024
Algorithm Decoding
Input: X, step"aa h.min, k-max
1: F, CF 2, R. // IF: the set of finished paths. Cf: the scores of IF
2: U,C + {(bos)) J. {0} /I U: the set of unfinished paths. CV; The scores of U,
3: H +- Encoder(X)
4: t+1
5: while U is not empty AND S stepmax do
6:
IB C < 2, Ø // B: the set newly branched paths. CP: The scores of B.
7:
S, H Decoder(UU, Hb) Calculate the generation scores of all the paths in U in batch.
8:
for +- to j-max do k-thLargest( S.,)) Return the k-th largest score St and its path index and vocabulary index w at time-step t.
9:
Stt,w.
if S.,, > or le S R.min then (.add( V " C add(C) Stt,,)))
12:
F, C + Eoth (B, C, C+) // If' path has generated the (cos), it is finished path. U,CU + B,C,
14:
++t+1
15:
return F,, CF
3.4 DECODING
Using the above training strategy, illustrated in Figure 2 the branching decoder can select an unfixed number of tokens with a generation score greater than O at each time-step in the inference phase, and dynamic branch out new paths to continue generating in parallel at the next time-step. However, this naive decoding strategy may suffer from insufficient generation-bbanchingg branching out too few paths, or excessive generation-bbanchhinn branching out too many paths to fit in the memory. In order to avoid such extreme cases, set the minimum number of explored paths ammin and the maximum number of generated paths ).max, and rely on the average score of the entire path instead of the score of the current token smooth the result.
We show the decoding algorithm branching decoder in Algorithm I For an input sequence X, we first Equation (1) to obtain its representation H R (line 3), and then iterate up to step max time-steps to parallelly generate paths (lines 5-14). In each iteration, the representation H is repeated [U] times and fed to Equation (3) with the set U to obtain the scores of all possible generated paths E |=*|U|*a (line 7). In order to avoid potential memory overflow caused by generating too many paths, we only generate new paths from the h-max highest-scored tokens among all n x candidate tokens at time-steps (lines 8-9). Then, the candidate tokens V whose generation scores Si,,w are greater than 0 or are among the top-k-min are appended to path U, and added to the set IB as new paths, and their corresponding generation scores are added to set C (lines 10-11). The paths that end with the terminal symbol (cos) are moved into the set I, their corresponding scores are moved to CF (line 12), and the rest will continue to generate in the next time-sepp (line 13). Finally, we filter out the paths whose average score less than to obtain the final generated sequences.
4 EXPERIMENTAL SETUP
4.1 DATASETS AND EVALUATION METRICS
Keyphrase generation (KG) classic task of set generation with rich experimental data. We selected three large-scale KG datasets KPTimes (Gallina et al., 2019), and StackEx (Yuan et al., 2000),, which are from the fields of science, news and online forums, respectively. Each dataset contains not only keyphrases that are present in a document but also those that are absent. Dataset statistics are shown in Table
main experimental data: KP20k (Meng et al. (017).
Following Chen et al. (220)), used macro-averaged Fi @5 and Fl @ to report the generation performance of present and absent keyphrases. When the number of predicted keyphrases is below 5, F1 @ first appends incorrect keyphrases until 5 and then compares with ground-truth calculate

---

Published as a conference paper at ICLR 2024
Table 1: Dataset statistics. # KP, KP], and % Abs KP refer to the average number of keyphrases per document, the average number of words that each keyphrase contains, and the percentage of absent keyphrases, respectively. All of them are calculated over the dev set.
Dataset
Field
Train
# Dev
# Test
# KP
KPP
% Abs KP
KP20k
Science
509 K
20 K
20 K
5.3
2.1
39,8
KPTimes
News
259 K
10 K
20 K
5.0
2.2
56.4
StackEx
Forum
298 K
16 K
16 K
2.7
1.3
46.5
Fl score defined by Yuan et al. 2020)) FI @M@MM is the version of FI @5 F1@5
without appending incorrect keyphrases, which compares all predicted keyphrases with the ground-truth compute F1 score.
4.2 IMPLEMENTATION DETAILS
We implemented our One scheme based on the code of huggingface transformers 4.12.5 and used T5 Raffel et al. 22200) as backbone. For the two stages of training, we trained 15 epochs the first stage, and 5 epochs in the second stage. We set step 20 to ensure that step"lax is greater than the length of all keyphrases on all dev sets.
We set -mm 60 to ensure that A is greater than all numbers of keyphrases on all dev set. We tuned mmi on each dev set from to 15 to search for the largest sum of all metrics, and the best Amin was used on the test set. On all three datasets, the best performance was achieved with I.min = 8.
We followed the setting of Wu al. (2022) using batch size 64, learning rate 1e 4, maximum sequence length 512, and AdamW optimizer. We used three seeds {0, 1, 2 and took the mean results. We used gradient accumulation 64, trained One2Branch based on T5-Base (223 M) single RTX 4090 (24 G), and trained One2Branch based on T5-LLarge (738 M) on two RTX 4090. For inference, we ran both base and large versions on single RTX 4090. Unless otherwise stated, the results of our baseline methods came from the work of Wu et al. (2022).
We also implemented our Ore2Branch scheme based MindSpore 2.0.
5
EXPERIMENTAL RESULTS
5.1 MAIN RESULTS: COMPARISON WITH SEQUENCE DECODER
We presented the comparison of generation performance, inference throughput and GPU mem- ory usage between the One2Branch scheme based on branching decoder and the One2Seq scheme based on sequential decoder. The implementation of One2Seq was trained with Present-Absent concatenation ordering, using a delimiter (sep) concatenate the present keyphrases and the absent keyphrases, which has been seen as an effective ordering in previous works (Yuan et al. 2020 Meng et al. 2021). We experimented the throughput and GPU memory usage with batch size on a single RTX 4090, and used greedy search to decode One2Seq (see results in Table 2
Generation performance. One2Branch outperforms One2Seq all F1 scores on absent keyphrase and more than half of Fl scores present keyphrase.
For absent keyphrases, they are completely unordered, and is difficult for One2Seq to give a reasonable concatenation order to avoid bias during training. while One2Branch is label order agnostic model which can deal with this problem. The results Table 2 support our motivation; compared with One2Seq. One2Branch improves by up to 3.5, 11.6, and 10.2 in Fl @5 and to 3.6, 6.3, and 9.5 in F1 @M on three datasets. It shows that Branhh has better set generation capabilities.
For present keyphrases, they may not be unordered set in some concatenation ordering, which potentially reduces the risk of One2Seq suffering from order bias. For example, the cases whose present keyphrase number is only one are ordered for the present part, and their proportions in the three datasets are 16.7%, 15.6% and 38. 1%. Despite this, Bre2Branch still performs better overall, outperforming One2Seq at least 3.5 in F1 @ and least 1.7 in F1 @M
due to the Present-Absent
https: tttp://ttttttttooe www.hh/.wwo//oggggggaea////ee/ee http://s///tttooc/oogggghhtottte

---

Published as a conference paper at ICLR 2024
Table 2: Comparison with the sequential decoder of the One2Seg scheme. The reported results are the average scores of three seeds. The standard deviation of each FI score is presented in the subscript. For example, 33.6; means an average of 33.6 with a standard deviation of 0.1. We reported the throughput and GPU Memory in inference stage with batch size 1.
Present
Absent
Throughput (example/s)
GPU Memory
Backbone
F1 @5 F1@M
FI
@M
KP20k
Onc2Seq
33.61
38,8,
1.70
3.40
2.5
1,714
TS-Base
One2Branch
36.31
35.2a
4.71
5,61
6.8 (2.7x)
1,858 M
One2Seq "
34.32
39.30
1.7o
3.50
1.6
33972 M
TS-Large
One2Branch
36.71
38.40
5.20
7.1a
5.1 (3.2x)
3,632
KPTimes
One2Seq
34.62
49.22
15.3:
24.2:
3,0
1,534
TS-Base
One2Branch
38.22
51.11
25.7:
28.7,
7.5 (2.5x)
2,084 M
One2Seq
36.60
50.81
15.71
24.11
1.7
3,944 M
La-Lerge
One2Branch
40.12
52.52
27.3g
30.4:
4.4 (2.6x)
3,682 M
StackEx
One2Seq
28.71
56.11
9.40
21.61
6.6
1,476 M
T5-Base
One2Branch
29.9,
55.01
19.67
31.1g
10.6 (1.6x)
1,500
One2Seq
30.52
58.0g
10.61
23.92
3.7
3,486 M
T5- La-Large
One2Branch
30.31
56.01
20.52
32.01
6.3 (1.7x)
3,472 M
on KPTimes. It is worth noting that KPTimes has the largest proportion of absent keyphrase com- pared the other two datasets (shown in Table [ which may bring greater challenges the train- ing of One2Seq with unordered set, resulting in overall performance lagging behind One2Branch One2Branch performs comparably to One2Seq on KP20k and StackEx with better F1 @5 but lower Fl @M. On these two datasets, One2Branch generally performs better when the target set is larger.
Throughput. One2Branch has significant advantages over One2Seq in inference speed. Benefiting from the parallel decoding capability of the branching decoder, the advantages of One2Branch are more obvious when the target set is larger. Recall the statistics in Table [ KP20k has the highest average number of keyphrases, One2Seq based on T5-Large). StackEx has the lowest average number of keyphrases, so the speedup on this dataset is lower than the other datasets, but still least 1.6 times faster than One2Seq. Impressively, One2Branch based on La-Large may even be faster than One2Seq based on T5-Base (5.1 vs. 2.5 KP20k and 4.4 vs. 3.0 KPTimes), even though the former has 2-3 times as many parameters as the latter.
witnesses the largest speedup accordingly (3.2 times faster than
GPU memory usage. Although the higher throughput of One2Branch comes from generating mul- tiple paths in parallel at the same time, their GPU memory usage is comparable, and One2Branch even uses less memory on the large version. There are two factors that cause this phenomenon. Firstly, the common prefix of multiple sequences is only generated once, while One2Seq needs to generate each one independently. Secondly, the sequence generated by One2Seq needs interact with the previously generated sequences, which will increase the usage of GPU memory.
5.2 COMPARISON WITH SOTA MODELS
Following the work of Wu et 2022, we compared with categories of sate-off-hheart keyphrase generation models. The first category is to study better model structures and mechanisms to improve sequential decoder. CatSeq Yuan et al.) (020) integrates copy mechanism (Meng et al., (017) into the model, and ExHiRD-h (Chen et al. 202T) further improves it by using an exclusion mechanism to reduce duplicates. Ye et al. (2021) first introduces Transformer (Vaswani al. 2017) to One2Seq model and then proposes SetTrans with order-agnostic training algo- rithm for sequential decoder. The second category is to use powerful pre-trained models such as BART (Lewis al. 2020) and T5 (Raffel et al.) 2020), and further pre-train in in-domain data, such as KeyBart Kulkarni et al. (2022). Baws Wu et al, (2022). SciBart Wu et al.] (2022). The comparison results reported in Table For the absent keyphrases, One2Branch exceeds all
8

---

Published as a conference paper at ICLR 2024
Table 3: Comparison with state-of-the-art One2Seq models.
KP20k
KPTimes
StackEx
F1 525
FM@M
F
F1@M
F105
FI@M
Present
CatScq
29.1
36.7
29.5
45.3
ExHiRD-h
31.11
37.40
12.1
45.2;
28.
54,82
Transforner
33.31
37.62
30.2a
45.3c
30.8c
55.42
SeiTrans
35.60
39.12
35.6G
46.34
35.8.
56.7s
BART-Base
32.22
38.83
35.91
49.9q
$7.11
BART-Large
33.24
39.22
37.318
$1.0ss
31.2,
57.8s
TS-Base
33.6;
38.80
34.6,
49.2g
28.7:
56.1
TS-Large
34.32
39.30
36.60
50.8,
30.52
58.0s
KeyBART
32.51
39.82
37.8a
58.9;
SciBART-Base
34.11
39.62
34.8,
48.8,
30.4g
57.64
SciBART-Lagee
34.7a
41.5.
35.34
49.7g
30.9g
57.82
NewsBARTI
32.4g
38.72
35.4g
49.8
30.7a
57.50
Onc2Branch (TS-Base)
36.31
33.22
38.22
S1.11
20.9,
35.01
One2Branch (T5-Large)
36.7:
38.40
40.1
$2.52
30.31
56.01
Absent
CatSeq
1.5
3.2
15.7
22.7
ExHiRD-h
1.6g
2.5g
13.42
16.5
10.11
15.5,
Transformer
2.22
4.64
17.11
23.11
10.4;
18.72
SetTrans
3.51
5.81
19.8g
21.92
13.9:
20.70
BART-Base
2,21
4.2g
17.1a
24.9h
11.7c
24.9a
BART-Large
4.7g
17.61o
24.4p
TS-Base
1.7g
3.4p
15.31
24.2:
9.40
21.61
TS-Large
1.7g
3.50
15.71
24.11
10.6.
23.92
KeyBART
2.6
4.7,
18.0;
25.52
13.0g
27.1s
SciBART-Base
5.24
17.2a
24.62
24.2s
SciBART-Lang
3.12
5.7g
17.2s
25.72
12.6,
26.7:
NewsBART-Buse
2.21
4.4g
17.6g
26.11
12.18
25.74
One2Branch (TS-Base)
4.71
5.61
25.71
28.76
19.67
31.16
One2Branch (T5-Large)
5.2g
7.10
27.3%
30.4,
20.5,
32.0.
One2Seq baselines on all three datasets, demonstrating its superior set generation capabilities. For the present keyphrases, One2Braach is also among the best on two datasets.
5.3 ADDITIONAL EXPERIMENTS
We conducted supplementary experiments, detailed in the appendix. We analysed the effectiveness of training with negative sequences in Section A. and the influence of the hyperparameter a-min in Section A.2 Case study presented in Section A.3 to analyze the characteristics of One2Branch concretely. To more comprehensively assess One2Branch's performance, we experimented with four -iittribiitiintninnnnnn datasets (Section(A.4) and question answering dataset (Section(A.5). These experiments reinforce the advantages of One2Branch as discussed in Section 5.1
CONCLUSION
In this paper, we propose a new decoder called branching decoder, which can generate a set of se- quences in parallel, in contrast with existing decoders that successively generate all the sequences in single concatenated long sequence. In the experiments, the branching decoder performed impres- sively in both set generation performance and inference speed.
As new paradigm for decoders, although we have demonstrated its promising effectiveness and efficiency in this work, more in-depth exploration is still needed. Currently, the branching decoder is implemented based on the existing pre-trained model with inconsistent training methods, which may limit the capability of branching decoder. It would be helpful to explore pre-training methods that better fit the branching decoder. In addition, whether branching decoder can achieve consis- tent performance on different generation architectures, such as GPT-style causal language model, deserves further experimental analysis.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the NSFC (62072224) and the CAAI-Huawei MindSpore Open Fund.
REFERENCES
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28: Annual Conference Neural Informa- tion Processing Systems 2015, December 7-12. 2015, Montreal, Quebec, Canada, pp. 1171-1179, 2015. URL e (ww.://6777777777222222222222310
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam. Girish Sastry. Amanda Askell, Sandhini Agar- wal, Ariel HerberttVosss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https::/prroceed.ngg... newr.ig......i..pp CC/ wttpsscppppppppepepapapaapappp/a 1 1145c0d6bf 4b4 96741 42ss html
42f Abst
Jie Cao and Yin Zhang. Otseq2set: An optimal transport enhanced sequence-to-se tooeence-to-eet model for ex- treme multi-label text classification. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 5588-5597. As- sociation for Computational Linguistics, 2022. doi: 10.18663/v1/20222.mmms www.:222.mmllp-mmm main.377. URL emn main.377
https://dois org. 011.:/0.00000011111/110100000000 126633/1//2000
Wang Chen, Hou Pong Chan, Piji Li, and Irwin King. Exclusive hierarchical decoding for deep keyphrase generation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguis- tics, ACL 2020, Online, July 5-10, 2020, pp. 1095-1105, Association for Computational Linguis- tics, 2020. doi: 10. 10.18653/v1/2022.acl
wai..min..nmmmmmmmmmmmeee 103.
2020. ac1-mallaal ma ww..aanin....mm.mmm..mm......m.. 103.
Wang Chen, Piji Li, and Irwin King. A training- and reference- free summarization evaluation metric via centrality-weighted relevance and self-referenced redundancy. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing, ACI/JJNNPP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 404-414, Association for Computational Linguistics, 2021. doi: 10.18653/v acl-long.34, URL httt:tt:: /tt::///oiiiio org. 18w.1886333320 wwt-::21..cloooaaaa..aa
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. All NLP tasks are generation tasks: A general pretraining framework. CoRR, 1bs//103..0060, 2021. URL
:tttps//aarxe wwg.//////a......ooo abs 2200. 10360
Ygor Gallina, Florian Boudin, and Béatrice Daille. Kptimes: A large-scale dataset for keyphrase generation on news documents. In Kees van Deemter, Chenghua Lin, and Hiroya Takamura (eds.), Proceedings of the 12th International Conference on Natural Language Generation. INLG 2019, Tokyo, Japan, October 29 November I, 2019, pp. 130-135. Association for Computa- tional Linguistics, 2019. doi: 2211653/v1/W19-660 org/w19-86177
URL https: http://////lllathhll/cooo ogy.
Travis R. Goodwin, Max E. Savery, and Dina summarization with adaptive multi-task fine-tuning. In Trevor Cohn, Yulan He, and Yang Liu
Towards zero shot conditional
10

---

Published as a conference paper at ICLR 2024
(eds.), Findings of the Association for Computational Linguisties: EMNLP 2020. Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pp- 2215-3226, Association for Computational Linguistics, 2020. doi: 10.
URLhttps:
7/doi aedoi.or//0... 0. 11113/61/5200222 f nd wgs-emmnl....m .29
Yuxin He and Buzhou Tang. Setgner: General named entity recognition as entity set generation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.). Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022. pp. 3074-3085. Association for Computational Linguis- tics, 2022. doi: 10. http://ww12222emmm/// 1omnlp-main.200 main.200. URL https://doi. org. 1ttp://0....001111.88888 v1/2022 emn 1 ma iin.annn...mm 2.0
Ari Holtzman, Jan Buys. Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL. htt.ppss://ppeereeieeww
met/forune wttprmmmmmirrrgoooe FygGQyrFVH
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th Interna- tional Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2799. PMLR, 2019. wewprooeeednngg.......... mw.. www.:mmw.....r..e.ommmeeeeeeeeee wwwpss/vsr...... w9wp///llouoooo sby 9a. html
URL
Zixian Huang, Jiaying Zhou, Chenxu Niu, and Gong Cheng. Spans, not tokens: A span-centric model for multi-span reading comprehension. In Ingo Frommholz, Frank Hopfgartner, Mark Lee, Michael Oakes, Mounia Lalmas, Min Zhang, and Rodrygo I.. T. Santos (eds.), Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21-25, 2023, pp. 874-884. ACM, 2023a. doi: org 10. 1eleg//..111555111 001144533537788 06155600.36656655
10. 10.11455/58337003665666
URLbttss:
Zixian Huang, Jiaying Zhou, Gengyang Xiao, and Gong Cheng. Enhancing in-context learning with answer feedback for multi-span question answering. CoRR, abs/2306.04508, 2023b. doi: 10. 48550/ARXIV.2306. 04508.
http://ti...rg/....// 48550/ hwtp//aaivvvv.. 04508
Gautier Izacard and Edouard Grave Distilling knowledge from reader to retriever for question answering. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
net.
Forum?id NTEZ 6wysdb
Mayank Kulkarni, Debanjan Mahata, Ravneet Arora, and Rajarshi Bhowmik. Learning rich rep- resentation of keyphrases from text. In Marine Carpuat, Maric-Catherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.), Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 891-906. Association for Computational Linguistics, 2022. doi: 10. 10.18653/w1/2022.ff. http5///ww/22222.nnnggaaaaaa fww.inngnnnnnall..77777
18653/v1/2022 findings Sea.ddingn--aacl.666 naac1.67
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-sequeeee pre- training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 7871-7888 Association for Computational Linguistics, 2020. doi: (0.18653/v1/20200 main. 703. URL
https: //ooi woo.orgg//lo 10.133/16/2200000 aal-mmallaa wm..i.n.....mm....... 703
Haonan Li, Martin Tomko, Maria Vasardani, and Timothy Baldwin. Multispanqa: A dataset for multi-span question answering. Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.). Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA. United States, July 10-15, 2022, pp. 1250-1260. Association for Computa- tional Linguistics, 2022. doi: 10.18653/v 1/2032/1/22222n/// www.l-main.n...ee main.90. URL https: //ooi. /tddi.oo//// 10. w1118855v/22000 naacl ma 90

---

Published as a conference paper at ICLR 2024
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li. and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume I: Long Pa- pers), Virtual Event, August 1-6, 2021, pp. 1882-4597. Association for Computational Linguis- tics, 2021, doi: 10.18653/v1/2222 acl.oo0.......ooo..33333.. URL 2021. acl long..........o , 353
Tianyu Liu, Fuli Luo, Qiaolin Xia, Shuming Ma. Baobao Chang, and Zhifang Sui. Hicrarchical encoder with auxiliary supervision for neural table-to-text generation: Learning better represen- tation for tables. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA. January 27- -eebuuary 2019. pp. 6786 6793. AAAI Press, 2019, doi: 10. 30aiiv33i01.33000
URL
doi www.:oi....rrgggooo 160 aaai. v 33101.111000 330.:....11111111111133131111111 33016786
Aman Madaan, Dheeraj Rajagopal, Niket Tandon, Yiming Yang, and Antoine Bosselut. Conditional set generation using seq2seq models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.). Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 4874-4896. Association for Computational Linguisties, 2022. doi: 10. 10011863/11/0222.com 202553///202..mmmlppp 324. URL bttps://dol1 www/oo.....oooo.ooooooo.o.... 18653/ v1 2022. emr.1p main.. 324
Rui Meng. Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. Deep keyphrase generation. In Regina Barzilay and Min-Yer Yen Kan (eds.). Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics, ACI. 2017, Vancouver, Canada, July 30- August 4, Volume 1: Long Papers, pp. 582-592, Association for Computational Lin- doi: 10.18653/w1/117-1053 URL ottp://////..../////.//////
guísties, 2017.
P17- 117-1054]
Rui Meng, Xingdi Yuan, Tong Wang, Sanqiang Zhao, Adam Trischler, and Daqing He. An empiri- cal study on neural keyphrase generation. In Kristina Toutanova, Anna Rumshisky, Luke Zettle- moyer, Dilek Hakkani-Tün Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.). Proceedings of the 2021 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 4985-5007, Association for Computational Linguistics, 2021. doi: 10.18653/v 1/2021 walaacllmain... 396. URL /ttpp:://oi. /ttt::///bl..og//00B886///////1 haacl-ma in. 396
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. J. Mach. Learn. Res., 21:140:1-140:77, 2020. URL http://///// /ttt:://jmr... jmlr.org/
papers/v21/20 274.html
Jianlin Su, Mingren Zhu, Ahmed Murtadha. Shengfeng Pan, Bo Wen, and Yunfeng Liu. ZLPR: A novel loss for multi-label classification. CORR, w2wps/2228800955...22 2022. doi: 10.48550/arXiv.2200. 02955. URL htt.:ssss///ooooor/
40w40/555552.. 120. 02955
Dianbo Sui, Chenhao Wang, Yubo Chen, Kang Liu, Jun Zhao, and Wei Bi. Set generation net- works for end-to-end knowledge base population. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. EMNLP 2021, Virtual Event / can Republic, 7-11 November, 2021, pp. 2550-9660. Association for Computational Linguistics, 2021. doi: 10.
Punta Cana, Domini-
URL httpthttt///ott /ttt:p/://ooioor/11.1.113333 0065//0.10//1113111
2021 eww www.emmww..mmmmmmmmmmmmm..mmmmmm 760
Zeqi Tan, Yongliang Shen, Shuai Zhang, Weiming Lu, and Yueting Zhuang. A sequence-to-set network for nested named entity recognition. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, 202 URL
International Joint Conference
Canada, 19-27 August 2021, pp. 3936 3942. org, 2021. doi: 2021/542
https: / .org 249.324963322 cc..2/2//53222225353333
12

---

Published as a conference paper at ICLR 2024
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 2017. URL httpp::/pprcceddggggge neurips......i.................. wc/pppper2017/aaassss (033ee44554/ddeeeet0oooo3eee
Abstract. html
Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural sequence models. CoRR, abs/1610.02424, 2016. URL
1610. 02424
Yuqiao Wen, Yongchang Hao, Yanshuai Cao, and Lili Mou. An equal-size hard EM algo- rithm for diverse dialogue generation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openzeevew net hett/ppaf2.cKK33EEEYYt-
Di Wu, Wasi Uddin Ahmad, and Kai- Kai-Wei Chang. Pre-trained language models for keyphrase gen- eration: A thorough empirical study, CoRR, abs/2212.10233,, 2022. doi: 10.48550/arXiv2221.. 10233.
2212.10233
10233
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang. Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event Austria, May 3-7, 2021. OpenReview.net, 2021. URL
Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, and Houfeng Wang. SGM: sequence generation model for multi-label classification. In Emily M. Bender, Leon Derczynsii, and Pierre Isabelle (eds.), Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pp. 3915-3926. Association for Computational Linguistics, 2018. htthttts:/// http://aacaantoooogoooooo wttpo/loooyoo//1--111331
Pengcheng Yang, Fuli Luo, Shuming Ma, Junyang Lin, and Xu Sun. A deep reinforced sequence- to-set model for multi-label classification. In Anna Korhonen, David R. Traum, and Lluis Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 5252- 5258. Association for Computational Linguistics, 2019. doi: (0.18653/41/p19-1555 URL https://do11 www.ooo.o/......... 11151//1//11111155551
Jiacheng Ye, Tao Gui, Yichao Luo, Yige Xu, and Qi Zhang. One2set: Generating diverse keyphrases as a set. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.). Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th Interna- tional Joint Conference on Natural Language Processing, ACI/IJCNIP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pp.
Association for Computational Lin- aac.:/222..loooo..335...33 URL https:://oi.org/e/ee
guistics, 2021. doi:
11/2021.aa1-1220
354
Xingdi Yuan, Tong Wang. Rui Meng, Khushboo Thaker, Peter Brusilovsky, Daqing He, and Adam Trischler. One size does not fit all: Generating and evaluating variable number of keyphrases. In Dan Jurafskk, Joyce Chai, Natalic Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp.
Association for Computational Linguistics, 2020. doi: 10.18653/v1/2000 acl-main.711 URL httlt: /tw. 10 1et..11155///000 acl acl-main.
Xiangwen Zhang, Jinsong Su, Yue Qin, Yang Liu, Rongrong Ji, and Hongji Wang. Asyn- chronous bidirectional decoding for neural machine translation. In Sheila A. Mcllraith and Kilian Q Weinberger (eds.), Proceedings of the Thirry- Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18).
13

---

Published as a conference paper at ICLR 2024
Table 4: The ablation study of negative sequence. Avg denotes the average FI @ and FI @I@M.
KP20k
KPTimes
StackEx
Backbone
FI@5
FI @@
F1@5
FI@M
F1@5
@I@M
Present
One2Branch
36.3
35.2
38.2
51.1
29.9
55.0
TS-Base
w/o negative
34.5
36.4
35.5
51.0
33.3
51.2
One2Branch
36.7
38.4
52.5
30.3
56.0
T5- T5-Large
w/o negative
36.5
38.2
39.2
52.4
33.4
52.3
Absent
One2Branch
4.7
5.6
25.7
28.7
19.6
31.1
T5-Base
w/o negative
4.3
5.9
26.7
27.2
22.9
24.0
One2Branch
5,2
7.1
227.3"
30.4
20.5
32.0
La-Large
w/o negative
7.0
27.7
29.2
23.3
25.2
New Orleans, Louisiana, USA, February 2-7, 2018, pp. 5698 5705. AAAI Press, 2018. URL htttps://ww..ett www..aaaa.com org/ /ttpg//c//i//e/// index.ndx..... htap:/AAA//A TWW.T/AAA118//11 /AtpAAAAAA/////////////ee///////
Yan Zhang. Jonathon S. Hare, and Adam Prügel-Bennett. Deep set prediction networks. In Hanna Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 3207 -217, 2019. 2019. URL httpo//tt://////oeeeiioooo hewripsss..n....i............... www.:hwwrrr.....ppppaeeee cttppper/cpp/////pea 2019/ 2019/hash/Br 6e 9wd0 5eea000aaee2e555eeeee2eeaaaaaaa0 www.::22ataaaaatttttto...
Chao Zhao, Marilyn A. Walker, and Snigdha Chaturvedi. Bridging the structural gap between encoding and decoding for data-to-text generation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Associ- ation for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 2481-2491. As- sociation for Computational Linguistics, 2020. doi: httpp://oo..orr/.0.66/////000..0
maw.acllmmai..2444. URL
A APPENDIX
A.1 ABLATION STUDY: TRAINING WITH NEGATIVE SEQUENCES
A unique feature of One2Branch is its training with negative sequences. In this section, delve into the effectiveness of this approach. For a fair comparison, we trained the checkpoints from the first stage for additional 5 epochs without using negative sequences. Table rreports the ablation results. The incorporation of negative sequences resulted in performance improvement in most cases (16 out of 24). Among them, the absent keyphases of StackEx showed the most significant increase on F1 @ M, recording 7.1 for the base version and 6.8 for the large version. Although using negative sequences induces drop of 8 indicators, the technique holds promise. Exploring it further, for instance, through dynamic negative sequence sampling as suggested by Xiong et al. (2021). might offer enhancements in the future.
A.2 INFLUENCE OF THE MINIMUM NUMBER OF EXPLORED PATHS
To prevent Ore2hranch from inadequate generation, we set minimum for explored paths, denoted as imi as detailed in section 3.4 Instead of requiring score strictly greater than every time- step, allow the average generation score to be above O. This adjustment, meaning larger min allows the branching decoder to explore a broader generation space. Figure shows the average number of keyphrases (# KP) generated by OnanBranch; For StackEx, when mii is greater than 5, # KP changes slowly, indicating that a larger exploration space is not necessary. For KP20k and KPTimes, # KP is still growing trend when Amin = 15.
gunin increases, # KP also increases.
In order to further analyze the effect of min on generation performance, we presented the Fl score of Branch the test sets in F Figure As Amin increases, the F1 @5 of both present
14

---

Published as a conference paper at ICLR 2024
30
20
#KP
10
0
0
10
5
15
k min
K
min
Figure 4: The average number of keyphrases generated by One2Branch-Baar (left) and One2Branch- Large (right) under different g-min, Red (dot), blue (square), and orange (triangle) refer to the KP20k, KPTimes, and StackEx dataset, respectively.
F395
Present F1 @5
Present FI @M
Absent FI@5
Absent Fi M
Figure 5: The keyphrase generation performance of One2Branch (T5-Base) under different mmin Red (dot), blue (square), and orange (triangle) refer to the KP20k, KPTimes, and StackEx dataset, respectively.
and absent keyphrases gradually increases and then flattens, indicating that larger exploration space helps generate correct sequences for the samples with more target keyphrases. From the performance on KPTimmes and StackEx, we note that as min increases, @ M first increases and then decreases. This suggests that moderately increasing mii enhance exploration and generation. However, excessively large mmin can introduce noise.
A.3 CASE STUDY
Table presents case from the test of KP20k generated by One2Branch-Large with kmmin 1, 8, and 15. Compared with miin 1, a larger min can help the model generate more insightful keyphrases. For example, when g.min 2 One2Branch generates the absent keyphrase "cryptogra- phy" which only obscurely expresses in the document.
There are many similar expressions in the generated keyphrases. For example, the gener- ated sequences ("message authentication code", "message authentication". "authentication", "mac scheme", "mac") have the same meaning and some of them have the same prefix. One2Branch would generate multiple semantically similar sequences when it is difficult to determine the optimal one. This problem may be alleviated by adding a deduplication module. In addition, an exces- sively large min can lead to overly broad generation. For example, when min 15, the generated keyphrase "digital signature" is irrelevant to the document. Therefore, a suitable min is integral to Bre2Branch.
15

---

Published as a conference paper at ICLR 2024
Present
Present
Absent
Absant
$2.99
k
k
k
Present @1@5 @
Present FI @ M
Absent F1 F1@5 @
Absent FI@M
Figure 6: The keyphrase generation performance of One2Branch (T5-Large) under different Ammin. Red (dot), blue (square), and orange (triangle) refer to the KP20k, KPTimes, and StackEx dataset, respectively.
Table 5: A case generated by One2Branch from KP20k. The keyphrases generated correctly are marked blue.
Construct message authentication code with one way hash functions and block ci- phers. We suggest an mac scheme which combines hush function and an block cipherin in order. strengthen this scheme prevent the problem of leaking the lermediate value between hash function and block cipher by additional random bits. The requirements the used hash function loosely. Security proposed scheme heavily dependent the underlying block cipher. This scheme is efficient software implementation for processing long messages and has clear security properties.
Document
Present: {one way hash function, block cipher, mac}
Ground-truth
Absent: [cryptography]
kmin =
[block cipher, message outhentication code, message authentication, one way hash function)
Generated KPs
(block cipher, message authentication code, message authentication, way hash function, hash function, authentication, authentication code, mac scheme, cryptog- raphy, security proof, mac)
min =
Generated
{block cipher, message authentication code, message authentication, one way hash function, hash function, authentication, authentication code, mac scheme, cryptog- raphy, security proof, digital signature] }
miin
Generated KPs
Table 6: Statistics of keyphrase generation (KG) and question answering (QA) datasets. # Trg, (Trg), and % Abs Trg refer the average number of target sequences per document, the average number of words that each target sequence contains, and the percentage of absent target sequences, respectively. All of them are calculated over the dev set.
Dataset
Task
# Test
# Trg
[Trg
Abs Trg
Inspec
KG
500
5.3
2.0
37.1
Krapivin
KG
400
9.8
2.5
26.4
NUS
KG
211
5.9
2.2
44.3
SemEval
KG
100
14.7
2.4
57.4
MSQA
QA
653
2.9
3.0
A.4 OUT-ORI-DITRIBUTIION KEYPHRASE GENERATION DATASETS
Following Wu et al. (2022), we used the One2Branch model trained on KP20k to evaluate on four out-of-distribution datasets. Some statistics of these datasets shown in Table
Table reports the comparsion of One2Branch and One2Seq in term of generation performance, inference speed, and GPU memory usage. Similar the results in Section 5.I] One2Branch has outstanding performance in the generation absent keyphrase and inference speed. It comparable and alternately leads with One2Seq present keyphrases.
16

---

Published as a conference paper at ICLR 2024
Table 7: Comparison with the sequential decoder of the One2Seq scheme on out-of-domain keyphrase generation datasets. The reported results are the average scores over three seeds. The standard deviation of each Fl score is presented in the subscript. For example, 33.61 means an average of 33.6 with standard deviation of 0.1. We reported the throughput and GPU Memory in inference stage with batch size I.
Present
Absent
Throughput (example/s)
GPU Memory
Backbone
F1 @5 FI@M
FI @5@5
@M@M
Inspec
One2Seq
28.8s
33,9s
1.11
2.0g
3.2
1,698 M
TS-Base
One2Branch
30.4g
33.7a
2.61
3.42
8.7 (2.7x)
1,652 M
One2Seq
29.51
34.3,
1,1s
2.16
2.1
3,948
TS-Large
One2Branch
32.21
34.21
3.10
4.0p
7.2 (3.4x)
3,516 M
Krapivin
One2Seq
30.2g
35.02
2.32
4.34
2.8
1,716 M
TS-Base
One2Branch
27.5g
26.9s
5.0:
6.13
9.0 (3.2x)
1,710 M
One2Seq
31.52
35,9;
2.31
4.57
1.9
3,960
T5-Large
One2Branch
30.31
31.31
5.0p
7.91
6.7 (3.5x)
3,614
NUS
One2Seq
38.88
44.0,
2.70
5.1s
3.2
1,712 M
TS-Base
One2Branch
39.12
39.74
5.67
6.5g
9.3 (2.9x)
1,706 M
Öne2Seq
39.84
43.86
2.53
4.26
1.9
33,44 M
T5-Large
One2Branch
41.61
43.60
6.0p
8.30
6.8 (3.6x)
3,592
SemEval
One2Seq
29.5
32.616
1.44
2.0s
1,712
TS-Base
One2Branch
29.32
30.64
2.5,
2.95
8.7 (2.4x)
1,618
One2Seq
29.710
32.1
1.51
2.0s
1.9
3,974 M
T5-Large
One2Branch
28.20
31.51
2.70
3.41
7.1 (3.7x)
3,560
In Table we reported the comparsion of One2Branch with state-of-the-art One2Seq models. It can be seen that One2Branch achieves a significant lead in absent keyphrase generation compared to all baselines. For present keyphrase, SetTrans is still the best performing model, thanks to its ability of capturing the dependencies between generated sequences.
A.5 QA PERFORMANCE
Besides the keyphrase generation datasets, we also experimented with One2Branch on a multi- answer question answering dataset MSQA (Li al.. (022). Each sample in MSQA includes question with at least two answers, and document containing all answers is given. Some dataset statisties reported in Table
For the implementation of both One2Branch and One2Seq. we searched the best learning rate from (Ie-4, 3c-4) and trained models 50 epochs. The maximum sequence length was to 2048 to ensure that all documents are not truncated. For One2Seq. we concatenated the answers in the order of their occurrence in the document. We followed Li al. (202) use two metrics. Exact Match (EM) measures the F1 score between generated answers and gold-standard answers, which requires an exact match between answer texts. Partial Match (PM) generalizes EM by using the length of longest common substring score each predicted answer based on its nearest ground-truth answer.
We experimented with MSQA in two configurations, with and without the given document. When using the document, MSQA becomes extraction task, extracting all answers from the document. is worth noting that the dataset provides the order in which the answers appear in the document. This is longer unordered set generation task, which allows the model generate answers in order. When the document not used, MSQA becomes a fully absent generation task, so that models need to generate answers from its parameters.
The comparison results of One2Branch and One2Seq on MSQA are reported in Table One2Branch lags slightly behind One2Seq in the configuration of using document, with maximum
17

---

Published as a conference paper at ICLR 2024
Table Comparison with state-of-the-art One2Seq models on four out-of-distribution datasets. All of the models were trained on the KP20k dataset.
Inspec
Krapivin
NUS
SemEval
FI@5
FI@M
FI@5
FI@M
@I@5
FI@M
FI@5
@M
Present
CatSeq
22.5
26.2
26.9
35.4
39.7
ExHiRD-h
25.44
29.1a
28.64
30.8;
30.4
28.218
Transformer
28.87
33.3a
31,4,
36.57
37.8G
42.9,
28.8s
32.1s
SetTrans
29.13
32.81
33.510
37.511
39.9%
44.622
32.2,
34.2
BART-Base
27.0s
32.37
27.0s
33.68
36.61
42.4s
27.1
32.121
BART-Large
27.611
33.3g
28.42
34.73
38.0s
43.511
27.412
31.1
T5-Base
28.85
33.95
30.23
35.02
38.88
44.04
29.516
32.616
T5-Large
29.51
34.3a
31.52
35.9h
39.8a
43.8a
29.710
32.1
KeyBART
26.83
32.5a
28.76
36.514
37.37
43.01n
26.0s
28.94
SciBART-Base
27.510
32.8
28.2s
32.91
37.37
42.114
27.0s
30.48
SciBART-Large
26.112
31.7
27.111
32.412
36.418
40.912
27.9
32.012
NewsBART- Bass
26.210
31.7
26.2s
32.315
36.9s
42.410
26.42.
30.42.
One2Branch (T5-Base)
30.46
33.7g
27.5g
26.9;
39.12
39.74
29.32
30.64
One2Branch (TS-Large)
32.21
34.21
30.31
31.31
41.61
43.6u
28.2a
31.51
Absent
CatSeq
0.8
2.0
2.8
ExHiRD-h
1.11
1.62
2.23
3.3a
1.64
2.1G
Transformer
1.2g
2.31
3,32
6.34
2.5.
4,4g
1.62
2.24
SetTrans
1.91
3.01
4.51
7.2g
3,7
5.517
2.22
2.92
BART-Base
1.01
1.72
2.83
4.98
2.64
4.2g
1.61
2.12
BART-Large
1.53
2.41
3.11
5.12
3.1s
4.8g
1.93
2.43
T5-Base
1.11
2.0g
2.32
4.3d
2.70
5.13
1.41
2.0g
T5-Large
1.13
2.16
2.34
4.57
2.53
4.2G
1.51
2.0g
KeyBART
1.42
2.32
3.62
6.46
3.14
5.57
1.64
2.2g
SciBART-Bas.
1.62
2.84
3.34
5.48
3.31
5.32
1.8,
2,21
SciBART-Large
1.52
2.62
3.41
5.63
3.2s
5.07
2.66
3.38
NewsBART-Baa
1.01
1.82
2.42
4.5g
2.4g
4.0g
1.6
2.22
One2Branch (TS-Base)
2.61
3.42
5.01
6.13
5.67
6.5g
2.5
2.95
One2Branch (TS-Large)
3.10
4.0o
5.00
7.91
6.0p
8.3p
2.70
3.4
Table 9: Comparison with the sequential decoder of the One2Seq scheme on the multi-span question answering dataset (MSQA). The reported results are the average scores over three seeds. The stan- dard deviation of each F1 score is presented in the subscript. For example, 72.34 means average of 72.3 with standard deviation of 0.4. We reported the throughput and GPU Memory in inference stage with batch size 1.
Dev
Test
Throughput (example/s)
GPU Memory
Backbone
EM
PM
EM
PM
w/ doe (extraction task)
One2Seq
72.3
84.0,
71.5s
84.1g
5.3
2,220 M
T5-Base
One2Branch
71.4:
83.22
70.2s
82.9g
9.0 (1.7x)
2,060 M
One2Seq
74.62
85.9,
74.77
86.4;
3.2
4,986
T5- T5-Large
One2Branch
74.7g
85.73
73.36
85.5G
5.7 (1.8x)
4,640 M
(generation task)
One2Seq
16.44
35.62
16.0+
35.4s
7.2
2,346 M
T5-Base
OranBhanch
19.9.
36.5;
20.9.
37.2,
8.9 (1.2x)
1,480 M
One2Seq
18.22
37.34
18.12
37.31
"4.8
3,878 M
La-ger
One2Branch
21.62
38.8.
23.3s
39.7;
8.0 (1.7x)
3,352 M
18

---

Published as a conference paper at ICLR 2024
gap of 1.4 the EM of the test set. This may be benefit of One2Seq in which later generated sequences can interact with previously generated sequences. Moreover, this is not a set generation task because the order of the answers is known. However, One2Branch has obvious advantages in inference speed. The inference time of One2Branch based on Lare is even better than One2Seq based T5-Base, when the former has significant advantage in QA performance.
One2Branch achieves more significant lead in the configuration of without using document, with maximum gap of 5.2 the EM of the test set. This demonstrates the potential of One2Branch broader range of set generation tasks beyond keyphrase generation.
19


Reviews:
[{"review_id": "QQtPWHJxCV", "reviewer": "Reviewer_JQ4Q", "summary": "Often generative models use a sequential decoder that generates a single output sequence. For the tasks where multiple outputs are possible (for example, a set generation problem), a popular way is to train a sequential decoder concatenating all outputs in a long sequence. This setup suffers from several limitations. \n\nThe paper introduces a branching decoder that can generate branch-out multiple generation paths. One very interesting contribution of the paper is the integration of the ZLPR loss (Su et al., 2022) that allows a threshold-based decoding algorithm (instead of heuristic approaches) for inference. During decoding the branching decoder generates new generative branches identifying  a dynamic set of tokens with logits exceeding a threshold (0 in this case). \n\nThe experiments are mainly done on keyphrase generation tasks focusing on three different datasets focusing on different domains and keyphrase lengths (avg 1.3 to 2.2 words on average). The results demonstrate that the branching decoder performs considerably better than the sequential decoders.\n\n--\n\nThank you for your response. Thanks for including Diverse beam search comparisons. I have updated my reviews accordingly.", "questions": "It would be great to hear the authors’ response to two main weaknesses raised above.\n\nMinor: Also, in section 3.4, why do we need the minimum number of explored paths (k^min) given we are using the threshold-based decoding algorithm?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The novel branching decoder that allows generating multiple target sequences in parallel, generating a dynamic number of tokens at each time-step. \n\nThe integration of the ZLPR loss during training allows 1) training with both target and negative sequence, and 2) a threshold-based decoding algorithm (instead of heuristic approaches) for inference.\n\nSolid results showing the advantages of the branch decoders for keyphrase generation tasks. \n\nThe codes are made available.", "weaknesses": "I believe that there are two major weaknesses in the paper. Addressing them would improve the impact of the paper significantly.\n\nFirstly, the sota comparisons are focused on pretrained models and sequential decoders. Diverse decoding and generation methods (see some relevant papers below)  would also potentially be good for set generation tasks, and it would be interesting to know how branching decoders perform against them. \n\nThe other major weakness of the work is that the authors solely focus on the keyphrase generation problem, where the keyphrases are 1-3 words long which is very small compare to natural generation tasks. Natural language generation is inherently a set generation problem, for example, there could be multiple paraphrases for a sentence (paraphrase generation), same questions can be asked differently (question generation) and summary can be written differently (summarization). It would be very interesting to see how branch decoders perform on some of these tasks (see some relevant papers below). \n\nRelevant papers:\n\nhttps://doi.org/10.18653/v1/D19-1308\n\nhttp://arxiv.org/abs/1703.06029\n\nhttps://www.aclweb.org/anthology/2020.findings-emnlp.218\n\nhttps://arxiv.org/pdf/2105.11921.pdf\n\nhttp://arxiv.org/abs/1611.08562\n\nhttps://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17329", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ic8XXvsDZS", "reviewer": "Reviewer_KcLG", "summary": "This paper tackles the challenge of set generation which has been mainly formulated as a One2Seq problem. Bypassing the order bias induced in the latter, the authors propose instead a branching decoder that generates sequences in parallel that meet certain requirements. Experiments on keyphrase generation show the effectiveness of their method compared to sequential decoders.", "questions": "- I am not familiar with the multi-decoder technique, can you elaborate on what \"a decoder with shared parameters\" means or provide a reference ? In the text, you mention that \"each sequence is input to a decoder independently\" does that mean there are multiple decoders with shared parameters ? or it is the exact same decoder that the inputs are fed into, although independently ?\n- What loss is the model trained on ? Is it only optimized on the ZPLR loss ? If so, what happens if only positive labels are considered ? How does your method compare with simply selecting the highest scoring sequences instead of selecting sequences that have positive scores in average ? During decoding, the $k_{min}$ highest-scored tokens are guaranteed to be selected at each iteration anyway. Does your proposed method work without the ZPLR loss ? \n- Is your negative sequence a contribution ? It would be insightful to have a pointer towards some negative contrastive learning literature that you drew inspiration from and indicate the differences.\n- What value of $k_{min}$ is chosen for the computation of throughput / GPU memory usage ? How do they vary with different values of $k_{min}$ ?\n\n**Typo**\n- There seems to be a typo in the figure 2: dotted arrow should go from B to CB.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and the presentation is clear.\n- Experiments on keyphrase generation demonstrate strong performance on all datasets compared to SOTA as well as better OOD capabilities.\n- Additional experiments show promises for other generation tasks e.g. MSQA.\n- Ablation study is conducted on the impact of negative sequences during training.", "weaknesses": "see questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NO8o3I8oK8", "reviewer": "Reviewer_wNPk", "summary": "This paper proposes a branching decoder, which can generate a dynamic number of tokens at each time-step and branch multiple generation paths. In particular, paths are generated individually so that no order dependence is required. Moreover, multiple paths can be\ngenerated in parallel which greatly reduces the inference time. The experiments are promising", "questions": "NA", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "*  The overall idea and motivation are novel for set generation tasks.\n* Thorough experiments demonstrate clear benefits over established approaches.\n* The training method and inference algorithm are well-designed.\n* Strong results on multiple datasets make a compelling case for the branching decoder.", "weaknesses": "* Little ablation to analyze the impact of different design choices.\n* Limited analysis of how performance varies across different set sizes and domains.\n* The factorization of decoder embedding seems unnecessary given modern hardware.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2uExzP4Bcd", "reviewer": "Reviewer_4BdG", "summary": "The paper considers set generation with seq2seq models. The proposed solution dubbed One2Branch is a combination of parallel decoding and the ZLPR loss which allows for a dynamic number of hypotheses. For training, a stepwise version of the ZLPR loss is optimized for both positive and negative label sequences. For inference, path-level scores are used with min/max number of paths. One2Branch is shown to outperform direct linearization (One2Seq) on keyphrase generation.", "questions": "While parallel decoding is more efficient, does it not also lose the benefit of autoregressive reasoning as well? It seems reasonable to expect that a sufficiently powerful One2Seq model may ultimately perform better than a more specialized set generation model.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Substantially novel approach to seq2seq set generation\n- Well-performing", "weaknesses": "- The method yields a task-specific model specialized for set generation (i.e., it loses the generality of seq2seq, in contrast to One2Seq).\n- The improvement is not very large, since the baselines are already pretty good.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)