Dependency Parsing is More Parameter-Efficient with Normalization
Authors: Paolo Gajo
Domenic Rosati
Hassan Sajjad
Alberto Barrón-Cedeño
Venue: NeurIPS 2025 poster ()
Keywords: semantic dependency parsing
normalization
biaffine attention
Primary area: applications

Abstract: Dependency parsing is the task of inferring natural language structure, often approached by modeling word interactions via attention through biaffine scoring. 
This mechanism works like self-attention in Transformers, where scores are calculated for every pair of words in a sentence. However, unlike Transformer attention, biaffine scoring does not use normalization prior to taking the softmax of the scores. In this paper, we provide theoretical evidence and empirical results revealing that a lack of normalization necessarily results in overparameterized parser models, where the extra parameters compensate for the sharp softmax outputs produced by high variance inputs to the biaffine scoring function. We argue that biaffine scoring can be made substantially more efficient by performing score normalization.
We conduct experiments on semantic and syntactic dependency parsing in multiple languages, along with latent graph inference on non-linguistic data, using various settings of a $k$-hop parser.
We train $N$-layer stacked BiLSTMs and  evaluate the parser's performance with and without normalizing biaffine scores.
Normalizing allows us to achieve state-of-the-art performance with fewer samples and trainable parameters.
Code: https://github.com/paolo-gajo/EfficientSDP

Dependency Parsing
is More Prramteer-fffcient with Normalization
Paolo Gajo University of Bologna
Domenic Rosati Dalhousic University
paolo. gajo20umibo.
Domenic, Reaaiiiccooati
Hassan Sajjad
Alberto Barrón-Cedeño
Dalhousie University
University of Bologna a. barron@uniiaa it
HSaj jad@dal...
Abstract
Dependency parsing is the task of inferring natural language structure, often ap- proached by modeling word interactions via attention through biaffine scoring. This mechanism works like self-attention in Transformers, where scores calculated for every pair of words a sentence. However, unlike Transformer attention, biaffine scoring does not use normalization prior to taking the softmax of the scores. In this paper, we provide theoretical evidence and empirical results revealing that lack of normalization necessarily results in overparameterized parser models, where the extra parameters compensate for the sharp softmax outputs produced by high variance inputs the biaffine scoring function. We argue that biaffine scoring can be made substantially more efficient by performing score normalization. We conduct experiments on semantic and syntactic dependency parsing multiple languages, along with latent graph inference non- -ingiiigcii data, using various settings of k-hop parser. We train N-layer stacked BiLSTMs and evaluate the parser's 's performance with and without normalizing biaffine scores. Normalizing allows us to achieve state-of-the-ar performance with fewer samples and trainable parameters. Code: https:// /ttpa://githie ///ggtttub... www.paoooopppeeceooooooooooooolo wwwpooogggjjooffffffoo fiinttsSS
Introduction
Dependency parsing (DP) consists in classifying node labels i, (words), edges ei (relations), and edge labels riy (relation types) of dependency graph 31. A popular model for this task, introduced by Dozat and Manning (77] entails modeling word interactions as a fully connected graph via biaffine attention. Despite its simplicity, a number of models using this biaffine transformation require more parameters than necessary, due to what we identify as a lack of normalization of its outputs. We hypothesize that this overparameterization is caused by the high variance of the outputs. which extra parameters help mitigate. After showing that variance can be reduced through normalization, we demonstrate that similar or better performance can be obtained for DP with fewer parameters and training samples.
g
[3] [4]
In this task, consider graph G = (V, E) comprising set of nodes V, connected by set of edges E, with each edge Gij E connecting pairs of nodes (V;, Uz) E V. In latent graph inference (LGI) for dependency graphs, sentence of [V] words modeled | [V] graph via biaffine scoring [7]
X W(W
XXQ(XWx) T QKT,
X E R
Observe that this is equivalent to the unnormalized scores used in [38] 's self-attention: QKT/ /v where Attention(Q. (Q, K, V) Softmax (QKT/v/A /vde)) V and d, the output size of the keys.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The main difference resides in the fact that Transformer attention scores are scaled by a = 1/ dd As explained in [8). this scaling done because high variance inputs to the softmax function will result in large values dominating the output (e=*2) and small values decaying (e**1). The downstream effect is exploding and vanishing gradients. To see how this works, observe that the score s between any query and key vector is the result of their dot product q:h: If we assume that these vectors are zero mean and unit variance, then the variance is Var(s) = dg. Finally, we get our normalization factor, which ensures that each entry in the score matrix has standard deviation of I, since Std vd.. Consequently, lower input variance will result in more stable outputs.
We observe that there is no consistency in the literature on the use of this scaling term for DP. Most works do not use this scaling term [3] 7 & [3 [4], with the exception of [37] who use [38] self-attention out of the box. In this paper, we carry out a series of experiments on semantic and syntactic DP tasks, along with non-linguistic LGI tasks, using a wide range of architecture ablations. We extend the work of Bhatt et al. B which represents the state of the art on dependency graph parsing in NLP.
Following the state of the art, we train stacked BiLSTMs and biaffine classifier to infer the latent dependency graph connecting the words of a sentence. We find that, when not normalizing the scores produced by the biaffine transformation, model performance drops in terms of micro-averaged Fi-measure and attachment score (26.. In particular, increasing the amount of layers produces normalization effect by reducing the variance of the output scores. Using score normalization, we find that in some cases similar or better performance can be obtained by reducing the amount of trained BiL.STM parameters by as much as 85%.
Our contributions thus forde-fold. (i) We show the impact of normalizing the output of the biaffine scorer in relation to the architectural changes in a DP model. (ii) We show DP models can obtain better performance with substantially fewer trained parameters. (iii) We provide new method that substantially improves scores and obtains state-of-the-art performance for semantic (2] 46 and
syntactic 48 dependency parsing.
The rest of the paper is structured as follows. Section 2 presents an overview of the literature concerming DP, with specific focus on how to infer the adjacency matrix of dependency graph. Section 3 explains why layer depth can compensate for normalization. Section 4 provides an overview of the datasets, models, evaluation, and other experimental details. Section 5 illuttrates how normalization mitigates overparameteriizaton. Finally, Section draws conclusions and provides avenues for future research.
2 BBcckrruud
LGI involves predicting all, or a proper subset of, the edges between the nodes of a graph [5| 19 39] For example, one could carry out LGI over molecule graphs [33] to predict the bonds between atoms or the proximity of pixel clusters in an image (9).
In the context of NLP, DP is a task concerned with inferring the dependency graph of a sentence 31 Depending on the nature of the nodes and relations of the graph, it can be described as semantic
(SemDP) 3 8 14 37 or syntactic (SynDP) 7 1 48). In this work, we tackle both tasks, but focus more specifically on SemDP, which can be thought of as comprising two sub-tasks: named entity recognition (NER) ( and relation extraction (RE) [3 [4]. They can be either approached in a pipeline as separate objectives 5 47 50 53 or by jointly training a single model on both
sub-tasks [3 4
14 49). In the context of deep learning models trained end-to-end, three main
paradigms are used: encoder-based models [3] 7 8 13
43) decoder-only Transformers, and seq2seq encoder-decoder
more commonly known as large language models (LLMs)
Transformers 17 20
In this work, we focus on encoder-style models, since they currently achieve the best performance [4 37 and are much more parameter efficient than LLM-based solutions. These models approach entity (node) prediction analogously to NER, while edges and relations are handled via MLP projection
on DP tasks [
node-pair features [16] 28 29 54 or via attention -based edge and relation
inference 3
g
13
14
37 Current state-of-the-art models, exemplified by K. are based on
the biaffine dependency parser introduced by (7), which uses stacked Bil STM layers on top of
embeddings from pre-trained language model.

---

N
30
45
9
40
10
200
100
600
800
1000
Figure 1: Effective rank p W)) reduces over training epochs we increase N of BiLSTM layers.
3 Layer depth can compensate for normalization
In order to reduce the number of parameters used for biaffine scoring, we observe that the softmax function is sensitive to inputs with high variance: large values dominate the output (e**1) and small values decay (c**1). ). This causes drop in the downstream task performance, due to some values dominating the probability outputs in the score matrix, as well as exploding and vanishing gradients. Typically, contemporary architectures based on 7 do not employ normalization, but still able to perform well on DP tasks.
We explain this discrepancy using insights from the theory of implicit regularization [1. which contains the result that, as layer depth increases, the effective rank of the weight matrices is reduced during gradient descent. Our claim is that reduction in the rank of the weight matrices causes corresponding decrease in the variance the input to the softmax function.
Result 1 (Singular Value Dynamics Under Gradient Descent []. Minimization of a loss function L(W) with gradient descent using weight matrices W Raxn fassuming small learning rate and initialization close to the origin). An -layer linear neural network leads the singular values or of W to evolve in the number of iterations by:
0r(1 + 1) a,(1) - - u.(v (1,
where (L(((()) RR V, are the corresponding right and left singular vectors, and .,- is the Frobenius inner product. This implies that, as the number of layers increases, the rank of the weight matrices decreases, since the smallest singular values decay.
While Result I is stated for deep linear neural networks for the matrix factorization task, it has been validated empirically as applying to deep non-linear neural networks [34] (2).. We validate this finding for the biaffine scoring setting in Figure I which shows the inverse proportionality between the effective rank p(W) (36 the weights BiLSTM and the number N of its layers.
Claim (Monotonic Increase of Output Variance with Rank), Let R. be a random vector with covariance matrix K Cov(X) RRxn, and let A e Rmxn be fixed matrix with singular value decomposition (SVD):
min(m.n)
A =
Σ
T.u.V
where 02 2 0. Define the best rank-r approximation of A by truncated SVD as:
A Σ T...V.
m denote the image of X under the rank-r linear map. Then, the total variance of Y,, as measured by the trace of its covariance matrix, increases monotonically with r:
LetY, A
tr(Cov(Y.)) & tr((Cov(Y++))

---

Proof. Let Ky == Cov(XX E R=*" denote the covariance matrix of the random vector X. Since covariance matrices are symmetric and positive semi-definite (PSD), we have K 2 0. Let A, -i=1 au be the rank-r truncated SVD of A, the covariance matrix of Y, : A, is:
Cov(Y,) = 2.,
To evaluate the total variance compute the trace:
tr (rCov((Y,)) tr (A, 22r
Using the eyclic property of the trace (ur( (t((ACC) tr(BCA)) and symmetry of Kar we get:
tr (A,
tr (K.n A, A.)
Now define the matrix M, == A,A, R=x", which is PSD. As r iccrasss, A, includes more
terms its truncated SVD, so we have:
M, Σ oivi M,+1 M. + 1
Thus:
M,+1 2 M,-
Since K,z 2 and since the trace product PSD matrices respects Loewner order (i.e., A < B, then tr(CA) & tr(CB) for all C 2 0), conclude:
tr rgK M,) & Mr+k),
which implies:
tr (Cov(Y,)) < (Cov(Y,+2)). Therefore, as the rank r increases, the total variance Y, increases.
The implication of Claim D is that the pre-softmax score matrix from a shallow network without normalization will have higher variance than a deeper network because of Result[ Based on this finding, we propose to remove BiLSTM layers and use normalization -rather than having greater layer depth- in order to develop more parameter-efficient method.
Experimental setting
4.1 Data
To train and evaluate our models, we use four SemDP datasets along with the 2.2 version of the Universal Dependencies English EWT treebank (enEWT) [24] and SciDTB [48] for SynDP. Table [ summarizes datasets statistics and reports their entity and relation class annotations.
As regards SemDP, ADE 12 is a medical-domain dataset comprising reports of drug adverse-effect reactions. Each sample contains a single relation, "adverseEffect." which links a disease drug. CoNLL04 (35 contains news texts and is annotated with the classic named entities {per, org, loc) and relations ry é workkFor, kill, orgBasedIn, liveln, locIn). ADE and CoNLL04 are characterized by relations which are not complex enough to form connected graphs of considerable size. SciERC [21 a dataset compiled from sentences extracted from artificial intelligence literature. It is arguably the most challenging dataset used in this study, since most of the entities in the validation and testing partitions are not assigned an entity class. This makes it hard to train tag embeddings that can help to infer edges. ERFGC [46] is dataset comprising "flow graphs" parsed from culinary recipes. The semantic dependency graphs of these recipes are directed and acyclic, with single root. Differently from 3, and as advised directly by [46] (personal correspondence), we ignore the "_" edge labels present in the corpus, since they link discontinuous parts of phrasal verbs and are thus irrelevant. For ADE, CoNLL04, and SciERC
the splits provided in 4 ERFGC is not available online;
obtained it by contacting the authors [46].
As regards SynDP, we use enEWT 24 to compare directly against 13
results. While many is closed-access and prohibitively expensive. Instead, we use SciDTB [88. discourse analysis dataset comprising 798 abstracts extracted from the ACL Anthology
works evaluate SynDP on the Penn Treebank 30
It was processed for the syntax dependency parsing task using Stanza [32
We neglect ACEM and ACE05 42 two popular datasets for RE, due to their prohibitive cost.
com http://www........o...../....ttt
https://aclattbooogyyooo

---

Table Size of the train/dev/test splits and entity/relation classes for each dataset. Relations
Data
Entities
ADE
disease, drug
adverseEffect
2,563 / 854 / 300
CoNLL04 922 / 231 / 288
organization, person, location
kill, locatedIn, workFor, orgBasedIn, liveln
SciERC
21
generic, material, method, metric, otherSciTerm, task
usedFor, featureOf, hyponymOf, evaluateFor, partOf, compare, con- junction
1,366 187 397
EREGC 242 / 29 29
food, tool, duration, quantity, action- ByChef, Action,Action, actionBy- Food, actionByTool, foodState, tool- State
agent, target, indirectObject, tool- Complement, foodComplement, foodE4, foodPartOf, foodSet, toolEq. toolPartOf, actionFq. timingHead other
enEWT
24
xPOS tags
UD relations
10,098 1,431 1,427
SciDTB
48
xPOS tags
UD relations
2,567 / 814/881
From these two datasets, consider the xPOS tags (e.g., noun, preposition, determiner) and the syntactic dependencies (c.g., sentence root, object, adverbial modifier) to respectively be the entities and relations of the dependency graphs to infer.
In order show that the normalization effect is innguage-independen in Appendix A we also carry out SynDP experiments using Universal Dependencies [251 datasets in six other languages. Finally, we also conduct experiments on three non-linguistic datasets: PCQM-Contact [TO, CIFARIO Superpixel [9], and QM9 33 PCQM-Contact and QM9 are datasets which contain graphs of molecules, while CIFAR I0 Superpixel contains superpixel clusters constructed from CIFAR10 images. We use these datasets to further show that the observed effects of normalization are not limited to linguistic dependency parsing. but are a general phenomenon carrying out inference of fully connected graphs. We report the results of this additional experiment in Appendix B
4.2 Model
We adopt the architecture of (3] in our work, schematized in Figure 2 It can be subdivided into four main components: encoder, tagger, parser, and decoder. The input is tokenized and passed through BERT-like encoder, where token representations are averaged into | word-level features x; R Optionally, additional features can be obtained by predicting the entity classes of each word with tagger, composed by single-layer BiLSTM d, followed by a classifier:
hg tag = o(x;), h tag ER
tag Softmax(MLP" (h!"))) Y: E y;
where T is the set of word tag classes. The tagger's predictions are then converted into one-hot vectors and projected into dense representations by another MLP, such that dag (1rmmb(1ryy)) (y:")) These new tag embeddings are concatenated with the original BERT output and sent to the parser.
In the parser, an optional N-layered BiLSTM & produces new representations h, = (ee tag + xx),
which are then projected into four different representations:
MLP(edge- head)
(h:,
ef MLp(edge-dept) (h;),
MLP dept) (h:),
r MLP(rel- (h:).
The edge scores s edge and relation scores spel are then calculated with the biaffine function f: fxx,, W)
x Wxx + x,
*Using lokenlevel representations resulted in much lower performance in preliminary experiments.
5

---

input
parser
BERT
BiLSTM N
Score norm
MLP (ohae
tagger BiLSTM &
biaffine scorer
MLP tag classifier
tags
score decoder
tag- lag-to-dense MLP
tagger
edges
edge labels
Figure 2: Dependency parsing diagram. Dashed components the targets of ablation experiments.
odge 8 = glodgo) e We), We E Rdxixd
spd p(ra)
W, E p
where R is the set of relation classes, i... the possible labels applied to an edge. In Appendix C following [13]. we extend the architecture and instead feed the representations produced by the BERT
encoder into L E {0, 1, 3) pairs of biaffine and graph attention network (GAT) 40) layers. This the influence of normalization when using k--opp parser.
allows us
Finally, in the decoder, the edge scores are used conjunction with the relation representations r and rf obtain the final predictions. During training, we do greedy decoding, while during inference, we use Chu-Liu/Edmonds maximum spanning tree (MST) algorithm [ to ensure the predictions are well-formed trees. This especially useful with big dependency graphs, since greedy decoding more likely to produce invalid trees as size increases. When doing greedy decoding, an edge index (i.e.. adjacency matrix) a, arg max, Sij
edge is produced by taking the argmax of the attention
scores 8; edge across the last dimension. The edge index is then used to select which head relation representations ri to use to calculate the relation scores sre f(rr: rá. W), W Rdx [dx/ The relations are then predicted as r, arg max, sij When using MST decoding, edge and relation scores are combined into single energy matrix where each entry represents the score of a specific head-dependent pair with its most likely relation type. This energy matrix is then used in the MST algorithm, producing trees with single root and cycles. Prior to energy calculation, edge scores and relation scores are scaled so that low values are squished and high values are increased, making the log softmax produce a hard adjacency matrix.
The model is trained end-to-end jointly on the entity, edge, and relation classification objectives:
M
m
Liag
1
Vi,t
tag
log
|
Ledge
log p (1
edge 1),
[V]
(R]
Lrel
1 (V
1)
Σ
High log (o())))).))
1-1--1
L >1 Leag + A2 + Lret).
Losses are calculated based on the gold tags, edges, and relations. We set A 0.1 and X2 = 1 as hyperparameters because the tagging task much simpler than predicting the edges, since the same top performance is always achieved regardless of any other selected architecture hyperparameters.

---

Table Main setting hyperparameter ranges. & Parser BiLSTM. biaffine layer.
Component
Hyperparameter
Values
Encoder
Freeze BERT
V BERT (V,x)
Tagger BiLSTM d
E {V, x]
Tagger
Concat. tag embeds.
e, E [V, xX)
& num. layers
Ne {0. 1, 2,33 ho {100, 200, 300, 4003 dMpP E {100, 300, 500) L E (V,x) pur [U.N]
e hidden dim.
MLP(edge) output dim. V
Parser
LayerNorm
MLP(edge) and f(edge) init.
Score scaling
a {1, va
Following the usual approach for SynDP 13 14, when training on enEWT and SciDTB use an oracle -the gold tags- and do not predict the POS tags ourselves. Since in this case we only focus on training the edge and relation classification tasks, we set λ₁ 0.
4.3 Hyperparameters
We experiment with a range of hyperparameters for the encoder, tagger, and parser, as listed in Table 2 We use BERT base [6] our pre-trained encoder, which we keep frozen throughout the whole training run in our main setting.
As regards the tagger, set Lg = 1 and he 100, in [3. with weights initialized with Xavier uniform distribution. Appendix D.I] we ablate the o BiLSTM and the use of the tag embeddings e to assess their impact on overall performance. With relation to the parser, for all hyperparameter combinations of (N, he, dMLP), uniform distribution (Iper ~ Lf) and no LayerNorm LNg. In Appendices D.2 to D.5 ablations over the parser hyperparameters indicated in Table
experiments by initializing its weights with Xavier
we conduct
We train our models for steps and evaluate on the development partition of each dataset every 100 steps. For enEWT and SciDTB, we train for 5k steps with 1k-stte validation intervals to make our results more comparable with the state of the art [13]. We apply early stopping at 30% of the total steps without improvement and choose the best model based on the top performance the development split. In Appendix D.6, we extend the training to 10k steps and fully fine-tune a variety of small and large pre-trained language models show time-wist test performance trends more clearly. We set the learning rate at including ablations,
10 10-3 when the encoder is kept frozen. all settings, AdamW [8] the optimizer and batch size of 8.
We use B original architecture as our baseline. uses frozen BERTNase model as encoder and trains all of the components showed in Figure Following the best results obtained by (7), they use three BiL.SMM layers the parser with hidden size of 400, while the four MLPs following stacked BiLSTM have output size of 500 for the edge representations and 100 for the relations.
4.4 Evaluation
Following
measure tagging and parsing performance on ADE, CoNLL04, SciERC, and ERFGC in terms of micro-averaged Fm-muure In addition, for enEW and SciDTB we use unlabeled (UAS) and labeled (LAS) attachment score (26). To corroborate the validity of our results, we train and evaluate each setting, including ablations, with five random seeds. We report mean and standard deviation for the F,, UAS, and LAS metries, averaged over the five runs. To the significance of our results, we use the one-tailed Wilcoxon signed-rank test 45
5 Results and discussion
Table shows the results of our experiments for the labeled edge prediction task, with the first row using the same hyperparameters [3] (hg 400, dMLP 500). We also use these hyperparameters
7

---

Table 3: Micro-F, (SemDP) and LAS (SynDP) for the labeled edge prediction task. Best in bold.
Model
N
ADE
CoNLL04
SciERC
ERFGC
enEWT
SciDTB
1
0.653
0.56610.019
0.257=0.024
0.701 10.000
0.804
10.006 0.915 20.002
0.541
10.021 0.399
-0.024 0.147
10.049
0.548
20.010 0.559
#0.005
0.729
20.004
1
0.657 20.011
0.556
=0.021
0.282
10.009
0,676
20.010 0.771
10.006 0.892
20.002
0.667 10.011
0.573
=0.025
0.273
10.010
0.694
20.010
0.796
10.006
0,910
20.002
Ours
3
0.662 10.027
0.562
=0.021
0.299
10.023
0.705
20.011 0.804
10.006
0.915
10.002
0
0.567
#0.014 0.438
=0.033
0.181
+0.027 0.612
#0.008 0.646
+0.002
0.796
#0.002
0,668 10.017
0.597
00.015
0.299
10.019 0,692
10.000 009 0.789
10.003
0,904
1.0002
Vd
0.676
10.019
0.596
000014
0.312
10.011
0.699
10 009
0.805
1.0003
0.916
100000 002
0.686 10.025
0.602
00.017
0.320
10.013 0.708
10. 008
0.807 0.000
0.919 100001
ADE @ (200, 100)*
CONTL04 @ (400, 300)*
SciERC @ (300, 300)*
0.70
0.60
0.30
0.65
0.25
0.50
0.60
0.45
0.15
0.55
0.40
EREGC @ (400, 300) *
EnEWT (400, 500)**
SciDTB (400, 500)*
0.80
0.90
0.75
0.65
0.85
0.70
0.60
0.65
0.80
0.60
0.75
0.55
0.35
2
2
2
Figure 3: Micro-F, (SemDP) and LAS (SynDP) vs N (0, at 2k training steps. Red = norm; blue raw. *Performance increase with normalization is statistically significant (p 0.01).
for enEWT and SciDTB, since our ablation studies only concerns SemDP due SynDP being less challenging. The lower half uses the best combinations for ADE (200, 100), CoNLL04 (400, 300), SciERC (300, 300), and ERFGC (400, 300), chosen based mean performance across these four datasets. For brevity, the performance on the tagging and unlabeled edge prediction tasks are reported in Appendix E Details the used computational resources be found in Appendix
Overall, normalizing biaffine scores provides an evident performance boost at all BiLSTM depths. In particular, for ERFGC we beat the state-of-the- ohethht-artttt art performance achieved by (3. Most of the performance gain obtained by adding the first layer, with additional ones yielding diminishing returns. In other words, the performance boost provided by score normalization is highest in the absence of the implicit normalization provided by extra parameters (N > O). general, the top normalization can be matched or
performance obtained by using three BiLSTM layers and surpassed with a single BiLSTM layer, when using score normalization. Taking into account the lower values for hy and dyn., this represents reduction in trained parameters of up 85%.
As laid out in Section 3 score variance tends decay with deeper BiL.TMM stacks. This in turn produces a converging trend as N increases. When looking at Figure this is especially evident

---

BERT buse 1p 0.01)
DeBERTabus abu (p 2.001)
0.4
0.4
0.3
0.3
0.2
0.1
0.0
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
BERT large (p = 0.75)
DeBERT Marrgg (p x 0.01)
0.4-
0.4
0.3
0.2
0.2
0.0
0.0-
0
2000
4000
6000
8000
10000
0
2000
4000
6000
8000
10000
Figure 4: Test performance on SciERC (micro imiero-averaged Fy-measure vs number of training steps). The p-values indicate greater performance with normalization (one-tailed Wilcoxon signed-rank test).
for ERFGC, enEWT, and SciDTB, for which the beneficial effect of score normalization shrinks smoothly with higher values of N. Although the same cannot be said for ADE, CoNLLO4, and SciERC, the performance boost is still statistically significant across all layer depths (p < 0.01).
As regards SciERC, normalizing scores without any BiLSTM layers (N 0) produces 23% increase in performance with a strong reduction in standard deviation. SciERC arguably has the hardest dependency graphs to parse, due to the little overlap between training and testing entities, which makes it difficult to leverage tag embeddings, In addition, it is characterized by complex semantic dependencies. Therefore, when dealing with hard tasks. In particular, normalizing scores helps mitigate the higher variance of the model's 's predictions for this challenging dataset. Conversely, the lack of score normalization exacerbates already uncertain predictions. For SciERC, this makes the trend observed in Figure 3 more unstable. Indeed, the performance first drops N 2 and then increases which does not happen for the other datasets.
reckon normalizing biaffine scores be especially important
again at N
Compared existing approaches which do not scale the biaffine scores, fewer parameters thus be trained to obtain similar performance. Therefore, our results indicate that parsers using raw biaffine scores are likely overparameterized, compared to the performance they could achieve by using score normalization.
In addition, score normalization increases sample efficiency by accelerating convergence. Figure displays the performance on SciERC's test set for four models during full fine-tuning. As the figure shows, the speedup in convergence is statistically significant when normalizing scores. This shows how score normalization can be beneficial even when fully fine-tuning models with ~10 parameters, given a hard task which forces the model to make uncertain predictions.
6 Conclusions
In this work, we explored the effect of scaling the scores produced by biaffine transformations when predicting the edges of dependency graph. We have demonstrated, both theoretically and empirically, that the score variance produced by lack of score scaling hurts model performance

---

when predicting edges and relations. In addition, our theoretical work and experiments highlight strong relationship between the number of trained layers and their intrinsic normalization effect.
Departing from a state-of-the-art architecture for semantic dependency parsing, we were able to both semantic and syntactic dependency parsing six datasets. On
improve its performance ERFGC, a dataset of directed acyclic semantic dependency graphs compiled from culinary recipes, our approach beats the state-of-the-ar performance achieved by Bhatt al. 3. Moreover, our results showed that a single BiLSTM layer can be sufficient to match or surpass the results of state-of-the-art architectures with a decrease trained parameters of up to 85%. In the case of SciERC, a challenging dataset for semantic dependency parsing, we found that the performance boost was particularly great when training the parser with three BiLSTM layers. Moreover, for this challenging dataset, we find that scaling the predictions of the biaffine scorer can accelerate convergence speed even when fully fine-tuning models in the 100 to 400M parameter range. addition, for three of the datasets we also observed that the performance obtained with normalized and raw scores converged smoothly as the number of trained layers increased. This supported our claim that stacking BiL.TMM layers mainly serves the purpose of producing an implicit regularization, and that this effect can be obtained by normalizing the scores, without any extra parameters, In additional experiments, we corroborated our findings using data in languages other than English 225 as well as non-linguisti data, for example using a dataset of molecule graphs [33]. Using GAT layers, we further confirmed our hypothesis that GNN-based iterative adjacency refinement [33 also benefits from normalization.
In future work, we plan to extend our experiments to large-scale graph inference tasks which have been limited by the lack of parameter-efficien methods, such as long-form discourse parsing tasks. With regard to score scaling, we also wish to verify whether the positive effects of this normalization can carry over to other tasks, beyond latent graph inference. Finally, further explorations in model efficiency are warranted, given our findings, e.g., via pruning of model parameters.
Acknowledgments
We acknowledge the support of the Killam Foundation, the Natural Sciences and Engincering Research Council of Canada (NSERC), RGPIN-2002-03944, Canada Foundation of Innovation (CFI) and Research Nova Scotia. Advanced computing resources are provided by ACENET, the regional partner in Atlantic Canada, the Digital Research Alliance of Canada, and the Vector Institute.
Paolo Gajo carried out this work while visiting Dalhousie University and working with the Hyper- matrix team. His work is funded by the BU (NextGenerationEU) with funds made available by the National Recovery and Resilience Plan (NRRP), Mission 4, Component 1, Investment 4.1 (M.D. 118/2023).
References
[1] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit Regularization in Deep Matrix Factorization. H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, Buch E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1607.06650 2016
[3] Dhaivat J. Bhatt, Seyed Ahmad Abdollahpouri Hosseini, Federico Fancellu, and Afsanch Fazly. End-to-end Parsing of Procedural Text into Flow Graphs. In Nicoletta Calzolari, Min- Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 5833-5842, Torino, Italia, May 2024. ELRA and ICCL.
[4] Zhen Bi, Jing Chen, Yinuo Jiang, Feiyu Xiong, Wei Guo, Huajun Chen, and Ningyu Zhang. CodeKGC: Code Language Model for Generative Knowledge Graph Construction, January 2024. arXiv:2304.0904 [cs].
10

---

[5] Yee Seng Chan and Dan Roth. Exploiting antactico-semanti structures for relation extraction. In Proceedings of the 49th annual meeting of the association for computational linguisties: human longuage technologies, pages 551-560, 2011.
[6] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[7] Timothy Dozat and Christopher D. Manning. Deep Biaffine Attention for Neural Depen- dency Parsing. In International Conference on Learning Representations. arXiv, March 2017. arXiv:1611.011 1.01734 [cs].
[8] Timothy Dozat and Christopher D. Manning. Simpler but More Accurate Semantic Dependency Parsing. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volune 2: Short Pupers), pages 4844900, Melbourne, Australia, July 2018. Association for Computational Linguistics.
[9] Vijay Prakash Dwivedi, Chaitanya K. Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. Journal of Machine Learning Research, 24(43):1 2023.
[10] Vijay Prakash Dwivedi, Ladislav Rampášek, Mikhail Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long range graph benchmark. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022), 2022.
[11] Jack Edmonds. Optimum branchings. Journal of Research of the National Bureau of Standards Section Mathematics and Mathematical Physics, 71B(4):233, October 1967.
[12] Harsha Gurulingappa, Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin Hofmann- Apitius, and Luca Toldo. Development of benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports. Journal of Biomedical Informatics, 45(5):885- 892, 2012. Text Mining and Natural Language Processing in Pharma- cogenomics.
[13] Tao Ji, Yuanbin Wu, and Man Lan. Graph-based Dependency Parsing with Graph Neural Networks. In Anna Korhonen, David Traum, and Lluís Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2475-2485, Florence, Italy, July 2019, Association for Computational Linguistics.
[14] Shu Jiang, Zuchao Li, Hai Zhao, and Weiping Ding. Entity-Relation Extraction as Full Shallow Semantic Dependency Parsing. IEFF/ACM Transactions on Audio, Speech, and Language Processing. 32:1088 12:1008-1099 2024.
[15] Anees Kazi, Luca Cosmo. Scyed- Ahmad Ahmadi, Nassir Navab, and Michael M. Bronstein. Differentiable Graph Module (DGM) for Graph Convolutional Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):1606-166) February 2023. Conference Name: IEEE Transactions Pattern Analysis and Machine Intelligence.
[16] Eliyahu Kiperwasser and Yoav Goldberg. Simple and Accurate Dependency Parsing Us- ing Bidirectional LSTM Feature Representations. Transactions of the Association for Com- putational Linguistics, 4:313-327, July 2016. _eprint: pdf//oi/10.1112/her a 00101/156741/facl 00101.pdf.
[17] Tianyu Liu, Yuchen Eleanor Jiang, Nicholas Monath, Ryan Cotterell, and Mrinmaya Sachan. Au- toregressive structured prediction with language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 993-1005, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
11

---

[18] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization, January 2019. arXiv:1711.05101 [es, math].
[19] Jianglin Lu, Yi Xu, Huan Wang, Yue Bai, and Yun Fu. Latent graph inference with limited supervision. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[20] Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. Unified structure generation for universal information extraction. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers), pages 5755-5772, Dublin, Ireland, May 2022. Association for Computational Linguisties.
[21] Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Ellen Riloff. David Chiang, Julia Hockenmaier, and Jun 'cchi Tsujüi, editors. Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pages 3219-3232, Brussels, Belgium, Detober-Novemble 2018. Association for Computational Linguistics.
[22] Alexis Mitchell et al. Ace 2004 multilingual training corpus Idc2005:09. Web Download, 2005.
[23]
David Nadeau and Satoshi Sekine. A survey of named entity recognition and classification. In Satoshi Sekine and Elisabete Ranchhod, editors, Recognition, classification and use, pages 3-28. John Benjamins Publishing Company, 2009.
[24] Joakim Nivre, Mitchell Abrams, Željko Agié, Lars Ahrenberg, and Lene Antonsen. Universal dependencies 2.2, 2018. LINDATICLARIAH digital library at the Institute of Formal and Applied Linguisties (ÚFAL), Faculty of Mathematics and Physics, Charles University.
[25]
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri. Thierry Declerek, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034 4043, Marseille, France, May 2020. European Language Resources Association.
[26] Joakim Nivre and Chiao-Ting Fang. Universal dependency evaluation. In Proceedings of the NoDaLiDa 2017 Workshop on Universal Dependencies (UDW 2017), pages 86 (6-95, 2017.
[27] Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, RISHITA ANUBHAI, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. Structured prediction as translation between augmented natural languages. In International Conference on Learning Representations, 2021.
[28) Wenzhe Pei, Tao Ge, and Baobao Chang. An Effective Neural Network Model for Graph-based Dependency Parsing. In Chengqing Zong and Michael Strube, editors, Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 313-322, Beijing, China, July 2015. Association for Computational Linguistics.
[29] Hao Peng, Sam Thomson, and Noah A. Smith. Deep multitask learning for semantic dependency parsing. In Regina Barzilay and Min- Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2037-2048, Vancouver, Canada, July 2017. Association for Computational Linguistics.
[30] Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. The Penn Discourse TreeBank 2.0. In Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, and Daniel Tapias, editors, Proceedings of the Sixth International Conference on Language Resources and Evaluation (8)) Marrakech, Morocco, May 2008 European Language Resources Association (ELRA).
12

---

[31] Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D Manning. Universal dependency parsing from scratch. arXiv preprint arXiv:1901.104553 2019.
[32] Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020.
[33] Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp, and O. Anatole von Lilien- feld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 111):440022, August 2014. Publisher: Nature Publishing Group.
[34] Noam Razin and Nadav Cohen. Implicit regularization in deep learning may not be explainable by norms. Advances in neural information processing systems, 33:21 112-117-41187, 2020.
[35] Dan Roth and Wen-Tau Yih. A linear programming formulation for global inference in natural language tasks. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) HLT-NAACL 2004, pages 1-8, Boston, Massachusetts, USA, May May 7 2004, Association for Computational Linguistics.
[36] Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606-610. IEFE, 2007.
[37] Wei Tang, Benfeng Xu, Yuyue Zhao, Zhendong Mao, Yifeng Liu, Yong Liao, and Haiyong Xie. UniRel: Unified representation and interaction for joint relational triple extraction. Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7087-7099, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties.
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[39] Petar Veličković, Lars Buesing, Matthew Overlan, Razvan Pascanu, Oriol Vinyals, and Charles Blundell, Pointer Graph Networks. In Advances in Neural Information Processing Systems, volume 33, pages -232-2244, Curran Associates, Inc., 2020.
[40] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
[41] David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. Entity, relation, and event extraction with contextualized span representations. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing GEMNLP-IJCNLAS pages 5784-5789 Hong Kong, China, November 2019. Association for Computational Linguistics.
[42] Christopher Walker et al. Ace 2005 multilingual training corpus Idc2006106. Web Download, 2006.
[43] Jue Wang and Wei Lu. Two are better than one: Joint entity and relation extraction with table-sequence encoders. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 1706-1721, Online, November 2020. Association for Computational Linguistics.
[44] Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang. Xin Zhang, Shen Huang, Pengjun Xic, Jinan Xu, Yufeng Chen, Meishan Zhang, Yong Jiang, and Wenjuan Han. ChatlE; Zero-Shot Information Extraction via Chatting with ChatGPT, May 2024. arXiv:2302.10205 [cs].
[45] Robert F Woolson. Wilcoxon signed-rank test. Encyclopedia of biostatistics. 8, 2005.
13

---

[46] Yoko Yamakata, Shinsuke Mori, and John Carroll. English Recipe Flow Graph Corpus. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5187-5194, Marseille, France, May 2020. European Language Resources Association.
[47] Zhiheng Yan, Chong Zhang, Jinlan Fu, Qi Zhang, and Zhongyu Wei. A partition filter network for joint entity and relation extraction. In Marie-Francin Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 185-197, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[48] An Yang and Sujian Li. SciDTB: Discourse dependency TreeBank for scientific abstracts. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 444-449, Melbourne, Australia, July 2018. Association for Computational Linguistics.
[49] Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charnois. An autoregressive text-to- graph framework for joint entity and relation extraction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19477-19487, 2024. Issue: 17.
[50] Dmitry Zelenko, Chinatsu Aone, and Authony Richardella. Kernel methods for relation extraction. Journal of machine learning research, 3(Feb):1083-1106 2003.
[51] Bowen Zhang and Harold Soh. Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction, April 2024. arXiv:2404003668 [cs].
[52] Dan Zhao. Combining explicit and implicit regularization for efficient learning in deep networks. Advances Neural Information Processing Systems, 35:3024 -0034, 2022.
[53] Zexuan Zhong and Danqi Chen. frustratingly easy approach for entity and relation extraction. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Turt Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 50-61, Online, June 2021. Association for Computational Linguistics.
[54] Hao Zhu, Yankai Lin, Zhiyuan Liu, Jie Fu, Tat-Seng Chua, and Maosong Sun. Graph neural networks with generated parameters for relation extraction. In Anna Korhonen, David Traum, and Lluís Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1331-,339, Florence, Italy, July 2019, Association for Computational Linguistics.
14

---

Table 4: Samples per
train/dev/test partition for each non non-English dataset.
Dataset
Train
Dev
Test
URL
Arabic
6,075
909
680
Chinese
3,996
500
500
http:///tthhl..ccc/////////////t ISDT
Italian
12,161
538
467
Japanese
6,824
493
518
http:://gttttiicco////////s/tllt GSD
Spanish 13,821
1,607
1,666
httpp://ggtttti.cco/cre/r/e/eltr AnCora
Wolof
1,149
436
453
Table 5: SynDP results for different languages, with and without score normalization.
N
AR
CH
IT
JP
ES
WO
0
0.538
i 0.005
0.395 i 0.007
0.563 i: 0.006
0.493
1: 0.010
0.554
0.004
0.252 1 0.007
1
1
0.723
+ 0.088
0.653
+ 0.077
0.792 + 0.003
0.812
= 0.008
0.775
0.003
0.525
+ 0.806
2
0.745
+ 0.004
0.710
+ 0.005
0.826
+ 0.003
0.844
0.005
0.807
0.002
0.587
+ 0.007
3
0.748
+ 0.004
0.717
+ 0.007
0.832 + 0,002
0.849
= 0.008
0.808
= 0.002
0.614
+
0
0.609
+ 0.003
0.479
+ 0.007
0.633 +: 0.002
0.585
= 0.006
0,620
= 0,002
0.305
+ 0.005
1
0.737
+ 0.007
0.693
0,003
0.820
0.04
0.838
0.044
0,801
0.000
0.556
0.006
<2
2
0.758
+ 0.003
0.736
1 0.005
0.842
+ 0.004
0.859
= 0.004
0.822
= 0,001
0.613
1 0.008
3
0.759
0.005
0.742
0.006
0.845 0.002
0.859
0.083
0.823
0.002
0.633
+ 0.005
A
Non-English data results
In this appendix, we carry out experiments on the SynDP task on six non-English datasets, all downloaded from Universal Dependencies The datasets are listed in Table 4 and comprise the following languages: Arabic, Chinese, Italian, Japanese, Spanish, and Wolof. The results of the experiments reported in Table 5 Since the data is in different languages, encoder we use mBERT, the multilingual version of BERT All model hyperparameters are kept the same as in the main setting.
Similarly to the English results observed for enEWT and SciDTB, the addition of BiLSTM layers increases the performance. Furthermore, additional layers have diminished boosting effect on the performance. This shows that the benefits of normalization are independent of the language of which the dependencies are being parsed.
B Non-linguistic results
In this appendix we lay out the unlabeled latent graph inference experiments we carried out on three non-linguistic datasets: PCQM-Contact [10]. CIFARIO Superpixel [9]. and QM9 [33]. PCQM- Contact and QM9 are molecule datasets, while CIFARIO Superpixel is a dataset of superpixel clusters constructed from CIFARIO. 10. Nodes in QM9 and PCQM-Contact respectively contain 9 and 11 features describing each atom in a molecule. In CIFARIO Superpixel, nodes contain the 3D RGB features of each pixel cluster. In QM9 and CIFAR10 Superpixel, concatenate the 3D and 2D spatial coordinates of the nodes to their features. Therefore, the resulting node feature dimensionalities of the three datasets are: dPcQM 9, deIFARIO 5, and dQM9 14.
We feed the graph node features into I., E (1,2,3) GAT layers and provide them with a fully- connected adjacency matrix without any edge attributes. Although the adjacency is fully connected, meaning each node can attend to all others in a single step, stacking L., layers refines the globally aggregated representations over 1-3 iterations. The processed node features are passed into the biaffine layer to produce edge predictions. In this case, we pass the scores into a sigmoid function
https://hugginngfacc.u ct wtwtr///oooobbbbbbbbbtebbbbeeeee http://bert-bbaeemllllllaaeeee cased
15

---

Table 6: Results for the non-linguistic datasets at varying depths of GAT layers.
Ly
PCQM-Contact
CIFAR10 Superpixel
Q.99
0.241 10.044
0.407 10.110
0.861 100008
1
2
0.217
20.019
0.528 20.134
0.876 20.006
0.245
20.034
0.751 20.014
0.877
20.008
0.288
20.090
0.531 20.088
0.886 00.001
vd
1
2
0.209
20.028
0.585 10.121
0,880
0.237
10.019
0.703
10.048
0.881
00.01
Lag
X;
biaffine scorer
biaffine scorer
BiLSTM & N
X
GAT Y
MLP
8
Figure 5: Extended parser architecture, featuring L, e 1,,3, pairs of biaffine and GAT layers.
because we are not trying to predict an arborescence with the maximum spanning tree algorithm like in dependency parsing. We use 0 BiLSTM layers to isolate the effect of the GAT layers with respect to any other network components. Scores averaged over three different seeds.
The unlabeled F1 scores reported in Table 6 In all cases, the results show that the gap in performance large at L and decreases (or flips) with additional layers. This corroborates the findings from the other settings and is an especially important result considering the phenomenon holds even for such a small network, consisting of just 100-160k parameters.
C GAT results
In this appendix, we extend the main-setting parser, following [3], and carry out k-hop parsing experiments by passing the node representations through L,, {0, 2,3) 3} pairs of biaffine and GAT layers. These allow the model to encode higher-order dependencies before the final biaffine layer. Figure 3 the extended architecture. We keep N 0 to isolate the effect of the GAT layers.
As Table showss, the results of this architecture are consistent with the ones obtained using the main- setting 1-hop parser. with the performance increasing across the board when using normalization. As regards the number of GAT layers, performance increases when using I.y E (1,2) layers, compared to T., 0, but drops again for all datasets T.y with normalization and T.y E {0, 1} layers. is particularly interesting to notice that for enEWT the top performance without normalization is obtained at L, 0. while I., 1 provides better performance when normalizing biaffine scores. Overall, the usefulness of multi-hop dependency parsing for SemDP is scarce if compared with the SynDP results, where higher-order dependencies provide more useful information for the predictions.
In general, the best performance is obtained
D Ablations
This appendix provides a collection of ablation experiments for various components and settings of the model used in the main experiments.
D.1 Tagger
Table reports the results for the best values of hy and dMLP (chosen as described in Section ablating over ó (V, x} and tag (V, x]. The first quarter of the table is equivalent to Table
16

---

Table 7: Micro-F, (SemDP) and LAS (SynDP) for the labeled edge prediction task at varying depths of Ly (0, 2, GAT layers. Best in bold.
Lyy
ADE
CoNLL04
SciERC
ERFGC
enWWT
SciDTB
0
0,517
20.038
0.526
20.046
0.123
20.050
0.536
20.013
0,610
20.009
0.727
20.006
0.509
20.022
0.493
20.037
0.039
20.020
0.599
20.008
0.583
20.011
0.731
20,009
2
0,509
20.025
0,485
20.033
0.029
10.041
0,611
11 10.015
0.540
20.012
0.710
20.013
0.487
+0.040
0.481
=0.087
0.000
+0.000
0.585
+0.009
0,511
#0.015
0.700
#0.010
0,556
10.037
0.574
0.0028
0.156
100031
0.607
100009
0.671
10.007
0.778
100006
0.587
+0.015
0.550
20.036
0. 0.148
+0.028
0.639
+0.007
0.696
=0.005
0.809
#0.008
V/
0.534
+0.023
0.563
=0.029
0.128
+0.029
0.637
+0.007
0.657
=0.007
0.795
+0.008
3
0.543
+ +0.024
0.514
00002
0.071 1 +0.021
0.616
+0.006
0.596
00.010
0.770
+0.007
ADE
CoNLL04
ScIRC
0.8
0.75
0.4
0.70
0.0
0.3
0.65
0.4
0.2
0.60
ERFGC
enEWT
SciDTB
0.8
0.90
0.6
0.8
0.85
0.5
0.80
0.1
2
3
6
10
6
7
8
9
10
0
1
2
8
9
10
Figure 6: Micro-F, (SemDP) and LAS (SynDP) vs E ,
10) at 20k training steps. Best
per-dataset hyperparameters (hg. dMLe) with Φ = and e; tag
= v. Red = norm; blue raw.
Using both the tagger BiLSTM and tag embeddings average produces the best results. This sensible, since a better tagger produces better tag embeddings, which in turn help inform the edge and relation classification tasks. The model in this case obtains the best performance CoNLL04, and ERFGC. Its top performance for SciERC (0.320) is also close to the overall peak performance (0.324), obtained without using the tagger BiLSTM. On average, the mean performance is considerably higher when combining the positive contributions of the BiLSTM and the tag embeddings.
ADE,
D.2 Additional BiLSTM layers
Figure 6 depicts performance against the number of BiL layers N at 20k training steps. Com- pared with the main setting's 24 steps, we increase the amount of training because lower amounts resulted in very high F, standard deviations with few N 6 layers in preliminary experiments. With a shallow stack BiLSTM layers (N < 6), performance is greater when normalizing scores. However, at deeper depths performance becomes very unstable for CoNLL04, ERFGC, and enEWT. Nonetheless, is evident that score normalization beneficial, with deeper stacks of BiLSTM layers reducing the effect of normalization, supporting our main- claims and experiments.
As observed Section adding trainable layers seemingly allows the parameters to scale the variance of the scores to compensate for the lack of explicit normalization. As reported in Appendix D.6. similar behavior can be observed with fully fine-tuned models, where normalization yields diminishing benefits the more parameters train. This behavior points trade-off between the use of our
17

---

Table 8: Tagger ablations with best hypeparaamterrs (hy, dMLP). Best in bold, second underlined.
a
Φ
ADE
CoNLLO4
SciERC
ERFGC
Mean
( dMPP)
(200, 100)
(400, 300)
(300, 300)
(400, 300)
0,541 #0.021
0.399
+0.024
0.147
+0.049
0.548 +0.010
1
0,657 20.011
0.556
10.021
0.282
10.009
0.676
10.010
0.515
0.667 10.011
0.573
10.025
0.273
0.694 10.010
0.662 10.027
0.562
10.021
0.299
10.023
0.705
10.011
0.567 10.014
0,438
10.033
0,181
10.027
0.612
008
1
0.668 10.017
0.597
10.015
0.299
10.019
0.692
1: 00.009
0.541
va
0.676 20.019
0.596
20.014
0.312
10.011
0.699
20.009
0.686 20.025
0.602
20.017
0.320
20.013
0.708
20.008
0.543 20.013
0.402
20.028
0.162
20.019
0.554
20.008
1
0.674 20.011
0.527
20,026
0.275
10.013
0.677
0.517
0.672 10.019
0.575
10.009
0.293
10.012
0.689
10.011
0.657
0.583
10.011
0.301
0.697
0.563 10.011
0.443
10.019
0.188
10.0066
0.612
1:0 100.033
0.653 20.023
0.565
#0.027
0.299
+0.020
0.687
#0.006
0.534
va
0.672 20.017
0.592
+0.020
0.307
20.008
0.702
20.005
0.661 20,022
0.593
20.017
0.305
#0.014
0.708
20.007
0.550
0.387
20.030
0.157
10.012
0.553
20.006
0.667 20.022
0.532
20,036
0,273
20.013
0.673
20.000
0.514
0.665 +0.00
0.565
+0.024
0.278
+0.027
0.694
#0.013
0.676 +0.021
0.558
+0.051
0,288
+0.012
0.706
#00000
0.578 #0.022
0.437
+0.030
0.187
+0.014
0.611
#0.008
1
0.651 +0.032
0.559
+0.029
0.301
+0.007
0.684
#0.003
0.534
vi
0.673 +0.029
0.582
00.017
0.310
+0.014
0.701
#0.008
0.659 20.019
0.586
+0.026
0.324
#0.019
0.708
20.012
0.547 20.019
0.402
#0.038
0.156
#0.029
0.549
20.015
1
0.651 20.017
0.552
#0.012
0.288
10.008
0.680
+0.007
0.516
0.664 #0.031
0.566
#0.012
0,288
+0.017
0.693
#0.011
0.663 +0 008
0.561
+0.014
0.291
+0.012
0.703
+00.007
0.566 +0.014
0.431
+0.015
0.182
+0.023
0.606
#0.012
0.655 #0.015
0.567
t0.010
0.286
+0.018
0.678 #0.013
0.534
vd
0.678 10.014
0.591
10.026
0,307
10.007
0.702 10.0007
0,684
0.595
10.017
0.321
0.698 10.011
technique and the amount of stacked BiL.STM layers, given dataset and given amount of training steps.
D.3 MLP output dimension
As shown in Table without normalization, increasing the output dimension of the two MLPs responsible for projecting edge representations leads to a decrease in performance. This is in contrast with the behavior observed when using normalization, where performance is essentially independent from the output dimension. This is in line with our claim that higher variance in the edge scores causes lower performance. Normalizing the scores reduces the variance, which makes performance stable even at high output dimensions. Once again, these results show the relationship between score variance and performance, with equivalent performance being achievable with fewer parameters.
D.4 BiLSTM hidden size
As Table shows, the hidden size of the BTL does not have visible effect on performance. This is especially the case when applying score normalization, which produces smaller standard
18

---

Table 9: Performance in terms of F. measure when ablating over different output dimensions for the parser's MLPs
v. eag ). Best in bold.
dMLP
ADE
CoNLL04
SciERC
ERFGC
(N, hp)
(3, 200)
(3, 400)
(3, 300)
(3, 400)
100
0.662
+0.027
0.579
=0.030
0.302
#0.018
0.709
+000008
300
0.658 +0.018
0.562 #0.021
0.299
#0.023
0.705
#0.011
500
0.657
10.018
0.566
10.019
0.281
10.016
0.701
10.009
100
0,686
10.025
0,586
10.013
0.318
10.017
0.712
1:0.008
vd
300
0.680
10 018
0.602 10.017
0.320
100013
0.708
1:0 00.
500
0.685 10.019
0,600 20.015
0.31 100009
0.707
100000
Table 10: Performance in terms of Fy-measure when ablating over different hidden sizes for the parser's stacked BiLSTMs (Ø V, gog v). Best in bold.
ADE
CoNLL04
SciERC
EREGC
(N, dMLP)
(3, 100)
(3,3
(3, 300)
(3, 300)
100
0.682 #0.021
0.580
+0.031
0.291
#0.021
0.693 #0.011
200
0.662 +0.027
0.582
+0.006
0.286
#0.032
0.703 #0.007
1
300
0.674
10.029
0,585
10.026
0.299
20.023
0.703
100006
400
0.663 10.032
0.562
10.021
0.289
20. 038
0.705 10.011
100
0.685 10.020
0.600
20.018
0.302
#0.013
0.698
10.006
200
0.686
+0.025
0.610
+0.018
0.314
#0.019
0.702
+0.008
vd
300
0.678
+0.012
0.599
+0.012
0.320 #0.013
0.711
+00005
400
0.674 10.011
0.602 10.017
0.306 10.016
0.708 1.0.08
deviations. As result, not only do models perform better with biaffine score normalization, but performance is also less dependent on the hidden size of the BiLSTM encoders. This supports our claim that score normalization is a useful technique to obtain the same performance with lower parameter counts, since the models tend to perform comparably, despite lower BiLSTM hidden sizes.
D.5 LayerNorm and parameter initialization
In this section, we analyze the effects of using a Xavier normal initialization (Tpor ~ N for the weights of the two projections MLP(edge-head) and MLp(egge-dept) along with the edge biaffine layer f(rdge) (- W2). We also apply a LayerNorm function LN 2 for each of the BiLSTM layers.
As reported in Table 11 average the best results are obtained with the base setting, i.e., with par l and L. x. Using LayerNorm can provide performance boost for some datasets, especially with uniform initialization. However, it makes performance unstable for some. As matter of fact, when using LayerNorm with three BiLSTM layers on CoNLL04 and SciERC, the model often breaks and is not able converge. Based average performance, then, the best models are obtained by scaling biaffine scores (a 1/vd) and initializing the parser with uniform weight distribution, without any LayerNorm in between the stacked BiLSTM layers.
D.6 Full fine-tuning and sample efficiency
In this section, we present the results obtained when fine-tuning BERT hase, DeBERTapass BERT and DeBERTatarge over 10k steps. In our previous settings, we only used a frozen BERT base: whose last hidden states we fed input to the tagger and parser. This evaluation allows to test the effectiveness of our approach in a setting in which the number of learnable parameters is unconstrained. In this experiment, we use different learning rates for the base models (7 1 x 10 4) and the large models (1) 3 x 10-5) and we apply gradient norm clipping with | maz 1.0. In
targes
littps://dook http://cw....ooroooooo///ooaoa/a htt.://waeeeem..........t..toooo www.hhtt init www.iaavieeee. pormall
19

---

Table 11: Ablation over the best hyperparameters (he, dMLP) and Φ and e, V, using LayerNorm layers LÑg {v, x } and Xavier uniform/normal initialization 1prr [ } for the parser. The first quarter of the table is equivalent to Table Best in bold, second- best underlined.
Ipar
LN a
ADE
CoNLL04
SciERC
ERFGC
Mean
(hes dMLP)
(200, 100)
(400, 300)
(300, 300)
(400, 300)
0.541 +0.021
0.399
+0.024
0.147 #0.049
0.548 +0.010
1
0.662 10.027
0,562
10.021
0,299
10.023
0.705
10.011
0.515
0.657 10.011
0,556
10.021
0.282
10.009
0,676 10.010
0.667 10.011
0,573
10.025
0,273
10.010
0.694 10 010
0.567 #0.014
0,438
#0.033
0.181
#0.027
0.612 +0 00.
0,668 10.017
0,597
10.015
0.299
10.019
0,692
10
0.541
Vd
0.676 10.019
0,596
20.014
0.312
10.011
0.699
1: 009
u
0.686 +0.025
0.602
10.017
0.320
10.013
0.708
1: 008
0.545 +0.018
0,399
#0.024
0.151
#0.014
0.548 +0.010
0.680 +0.014
0,554
#0.042
0.265
#0.028
0.686
+0.007
0.468
0.679 +0.011
0.578
+0.033
0.260
+0.020
0.715
+0.012
0.680 10.013
0.117
10.261
0.060
10.133
0.569 10.318
0.567 +0.014
0.433
#0.029
0.181
#0.027
0.614
+0.004
1
0.664 +0.017
0.564
#0.037
0.282
#0.029
0.686
+0 004
0.503
vi
0.697 +0.022
0.623
#0.019
0.300
#0.042
0.702
#0.011
0.675 10.019
0.239
20.327
0.111
10.152
0.703
10 006
0.545 $0.017
0,415
20.014
0.155
#0.019
0.558
#0.009
0.667 +0.014
0,543
#0.017
0.275
+0.020
0.680
+0.015
0.520
0.674 +0.025
0.576
+0.023
0.272
#0.014
0.699
+0.006
0.672 +0.022
0.580
#0.022
0.297
+0.019
0.705
+0 0.0.006
0.570 +0.013
0.454
#0.006
0.181
+0.023
0.613
+o 0000007
0.671 +0.015
0.578
+0.031
0.299
+0.030
0.685
+0.010
0.538
Va
0.671 +0.031
0.593
#0.016
0.301
+0.019
7.704
+0 0000007
W
0.681 +0.024
0.590
#0.014
0.315
#0.009
0.702
+0.011
0.545
0,415
10.014
0.155
+0.019
0.558
=0.000 40.009
1
0.679 +0.012
0.582 20.012
0.259
#0.010
0.680
20.016
0.469
0.668
0.580
0,284
+0.024
0.711
20.005
0.679 +0.018
0.000
00.000
0).000
+0.000
0.711
+0.009
0.570 10.013
0.454
10.006
0.181
10.023
0.613
10.007
1
0.675 10.019
0,578
10.022
0,280
10.022
0.682
10 006
0.512
Vd
0.687 10.021
0.609
10.012
0.308
10.012
0.702
10.011
0.678
+0.000
0.476
0.000
+0.000
0.701
+0 2.0.006
20

---

Table 12: Results for the fully fine-tuned models (hp- = 400. hour 500, Φ == V. and e
Model
a
ADE
CoNLL04
SciERC
EREGC
Mean
BERT BERTbase base
0.748 +0.028
0.613
#0.019
0.414
#0.011
0.726
+0.006
0.321
vd
0.731
10.025
0.629
10.022
0.412
10.016
0.726 10.006
0.321
BERT BERTaarge
0.748
10.016
0.700
#0.019
0,473
#0.011
0.750
#0.006
0.340
vd
0.777
20.011
0.697
#0.014
0.446
#0.025
0.748
+0.010
0.341
DeBERTapase
0.754
#0.013
0.674
#0.014
0.429
#0.015
0.751
+0.002
0.332
0.761
10.021
0.700
20.013
0.425
20.019
0.751
40.010
0.338
DeBERTatorge
1
0.786
10.010
0.739
10.016
0.478
10.019
0.763
10.006
0.352
vd
0.794 10.015
0.747
10.013
0.476
10.010
0.763 10.007
0.353
the case of the large models, we also use a cosine schedule with warm-up over 6% of the steps. We adopt these measures because during early trials we experienced sudden mid-run gradient explosions. In this case, we do not use any downstream BiLSTMs and only vary whether we use biaffine score normalization in the parser.
Table [2] reports the performance terms of micro-averaged F measure for the best models, picked based on top validation performance, evaluated every 100 steps. As the table shows, normalizing the scores generally does not increase top performance when fully fine-tuning these models. This is in line with our theoretical results, since tuning all of their - 100 400M parameters can compensate for the lack of biaffine score normalization.
In order to have stronger indication of whether score normalization also works in this setting, we study the performance on the test set versus the amount of training steps. Figure and Figure respectively visualize the performance of the base and large models on the test set in terms of aveeree F1 me-mearr over 10k training steps. We still use early stopping, which why some of the series are cut short before reaching 10k steps.
As Figure shows, one-tailed Wilcoxon signed-rank test finds the difference in performance throughout the training to be significantly higher when normalizing scores. Since the average test performance of the models is similar with and without normalization, the gap between the two curves being statistically significant indicates faster convergence with normalization.
For both BERT base and DeBERTabase the difference in convergence speed and performance is statistically significant on ADE, CoNLLO4, and SciERC. The same is true for BERT only on ADE. However, the effect is statistically significant with DeBERTatorge for all datasets. This indicates our score normalization approach can produce positive effects in terms of sample efficiency also when fine-tuning models with hundreds of millions of trainable parameters.
E Full main-setting results
Table 13 reports the
results for the best hyperparameter combinations for all three tasks of predicting tags, edges, and relations of the semantic dependency graphs. The first part of Table [3 identical to the bottom of Table 3 Since we already discussed the performance for labeled edges in Section we forgo discussing them again in this appendix.
As regards unlabeled edges, using
normalization improves mean performance for all datasets, similarly to relations. This is expected, as unlabeled edges directly influence the prediction of the edge labels, together with entity class prediction. As we have already observed for relations, the performance boost provided by normalization is particularly great for SciERC at = 0.
It is also interesting to notice that for CoNLL04, the performance for the unlabeled edge prediction task only slightly higher than that of the relations. fact, as regards ADE, the opposite is true, with the labeled performance seemingly being higher. Due to the high variances, it would therefore seem that for these two datasets there essentially no difference difficulty between the labeled and unlabeled edge prediction tasks. In the case of ADE, this makes perfect sense, since there is
21

---

ADE (p < 0.01)
CoNLL04 (p < 0.01)
0.70
0.6-
0.65
0.60
0.1
0.55
0.50
SciERC (p 0.01)
EREGC (p 1.00)
0.4
0.6
0.3
2000
4000
6000
8000
10000
2000
4000
6000
8000
(a) BERT
ADE (p < 0.01)
CoNLLAM 0.00
0.7
0.6
0.6
0.4
0.5
0.2
2000
2000
6000
8000
100000
2000
4000
5000
8000
10000
SGiRC (p 0.01)
EREGC (p 0.70)
0.4
0.6-
0.3
0.2
0.1
0.2-
0.0
2000
2000
6000
8000
100000
2000
40001
5000
S1000
10000
(b) DeBERTabase
Figure 7: Performance in terms of F1- meeasure vs the number of training steps for the base models on SemDP datasets. Red = norm; blue = raw. The p-values refer to the performance being greater with score normalization (one-tailed Wilcoxon signed-rank test).
only possible relation. For CoNLLO4, the amount of relation types is rather limited well; therefore, it is sensible for the performance to also be similar in this case. Indeed, when looking at SciERC and ERFGC, the performance gap between labeled and unlabeled tasks is considerable. This a strong indication that CoNLLO0 is thus found somewhere the middle, where the number of possible relations only slightly affects labeled performance.
As regards enEWT and SciDTB, the performance on the prediction edges (UAS) and relations (LAS) is also rather similar. Since in this case the predictions involve syntactic rather than semantic relations, the similar performance hints at relations being easier to predict in SynDP than in SemDP.
In the tagging task, very high standard deviations can be observed. In the case of ADE and CoNLL04, this could be caused by our choice of Å = 0.1 being too low a value. Regarding SciERC, as already mentioned, most of the entities in the validation and test sets are not found in the training set. This means not possible to train the model to recognize them, which results in the abysmal tagging performance reported in Table B This makes it challenging to train good tag embeddings which can
22

---

ADE (p < 0.01)
CoNLL04 (p = L.00)
0.6
0.4
0.2
0.2
0.01
0.0
4000
6000
10000
2000
8000
SciRC (p 0.75)
EREGC (p 0.66)
0.4
0.6
0.3
0.0
2000
4000
6000
8000
10000
2000
4000
600D
8000
10000
(a) BERT lange
ADE (p < 0.01)
CONLLO4 (p=00.001
0.8
0.8
0.6
0.4
0.4
0.2
0.0
0.0
0
2000
2000
6000
8000
10000
2000
4000
5000
8000
10000
SciRC (p 0.01)
ERFGC (p < 0.01)
0.1
0.4
0.2
0.2
0.0
0.0
2000
2000
6000
8000
10000
2000
40001
6000
S000
(b) DeBERTaterge
Figure 8: Performance in terms of F -measure vs the number of training steps for the large models on the SemDP datasets. Red = norm; blue raw. The p-values refer to the performance being greater with score normalization (one-tailed Wilcoxon signed-rank test).
help inform the edge and relation prediction tasks. Surprisingly. we notice that for CoNLL04 and ERFGC the tagging performance is also seemingly higher with score normalization. However, the overlaps of the standard deviations are too high to be able to make any claims on the matter.
F
Computational resources
We ran all of our experiments on cluster of NVIDIA H100 (96GB of VRAM) and NVIDIA L40 (48GB of VRAM) GPUs, one run per single GPU. When freezing the BERT base encoder and only training the BiLSTMs and the classifiers, each training and evaluation run took -5-7 minutes, depending on the number of BiLSTM layers. For the main setting, finding the best hyperparameters involved training and testing 620 models, for a total ~600 GPU hours. the full fine-tuning setting (Appendix (.66). training and evaluation took ~1 hour for each of the 40 base models and ~2-3 hours for each of the 40 large models, for an additional ~ 140 GPU hours.
23

---

Table 13: Micro-Fi (SemDP) and LAS (SynDP) on all tasks (Ø = vv. tag V).
Best in bold.
Metric
a N
ADE
CoNLL00
SciERC
ERFGC
enEWT
SciDTB
(hep, dMLP)
(200, 100) (400, 300)
(300, 300)
(400, 300)
(400, 500)
(400, 500)
0 0.541 +0.021 0.399
=0.024
0.147
#0.049
0.548
#0.010
0.559
+0.005
0.729
#0.004
1 0.657 #0.m1 0.556
20.021 0.282
#0.009
0.676
#0.010
0.771
+0.006
0.892
#0.002
2 0.667 10.011 0,573
10.00
0.273
10.010
0.694
10.010
0.796
10.006
0,910
10.002
rels.
0.662 10.027
0,562
20.021
0.299
10.023
0.705
20.011
0.804
10.006
0.915
10.002
0.567 20.014
0.438
20.033
0.181
20.027
0.612
20.008
0.646
#0.002
0.796
:20.002
0.668
0.597
=0.015
0.299
10.019 0.692
0.789
40.008
0.904
100002
0.676 +0.019
0.596
=0.014
0.312
+0.011
0.699
+0.009
0,805
+0.003
0,916
+t0.002
0.686 +0.025
0.602
=0.017
0.320
+0.013
0.708
+0.000 000
0.807
#0.005
0.919
+0.0011
0.536
+0.009
0.415
=0.020
0.173
+0.050
0.601
+0.013
0.589
+00006
0.745
#00.005
0.652
+0.009
0.566
0.321
#0.009
0.744
#0.013
0.793
#0.006
0.901
#00002
1
0.657
+0.021
0.586
20.017
0.322
#0.017
0.769
#0.014
0.819
+0.006
0.919
#0.002
edges
0.649 10.027
0.578
0.355
10.028
0.782
10. 007
0.827
10.006
0.924
1:0.002
0.549
0.453
20.032
0.203
+0.030
0.674
20.007
0.682
#0.003
0.815
#0.001
0.654
+0.020
0.598
20.021
0.351
#0.019
0.762
20.009
0.810
#0.003
0,913
200.022
Vd
2
0.660
10.015
0.600
20.008
0.367
10.015
0.775
0.827
0.925
20.002
0.666
+0.028
0.610
=000022 022
0.377
#0.021
0.786
+0.004
0.829
+0.004
0.928
000.002
0
0,615
10.115
0.285
10.150
0.014
10.015
0.662
20.073
1
0.665
10.147
0.671
20.018
0.049
#0.014
0.794
+0.057
2
0.746
+0.011
0.648
=0.074
0.060
+0.012
0.838 #0.016
0.768
+0.022
0.669
20.033
0.062
#0.005
0,853 +0.013
tags
0.604
10.134
0.464
20.115
0.046
#0.005
0.735 +0.066
0.734
0.702
00.031 0.058
+0.014
0,857
20.010
Va
0.724
+0.004
0.688
080
0.060
#0.007
0,864
20.018
0.762
10.017
0.690
=0. 080
0.057
10.012
0.850
10 +0.022 022
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction faithfully report the claims made in the rest of the paper, i.e. we show theoretically and empirically that normalizing biaffine scores for dependency parsing NLP provides higher parsing performance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In our paper, we discuss that we limit ourselves to the task of dependency parsing NLP. In addition, in the conclusions we mention that our approach does not use k-hop parsers, for example with GNNs triaffine transformations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
25

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: In Section B we provide lengthy introduction to Result I with additional information right after it, which provides more context.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All of the hyperparameters and methodology provided to the reader and the experiments can be easily reproduced through the provided GitHub repository.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
26

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code is provided. The repository includes README md file which guides the reader through its usage. The data for ADE, CoNLLO4, and SciERC can be
downloaded from the link provided in Section 4.1 As regards ERFGC, [46] provide access to the corpus upon request. UD 2.2 enEWT and SciDTB are openly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
guuides/CodeSubiisss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we report information about the data we use in Section 4.1
Training and
evaluation information is provided in Sections 4.2 4.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, all of our tables and diagrams provide standard deviations and error bars, calculated from model training runs carried out with five different seeds.
Guidelines:
27

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix provides all relevant information with relation the hardware used, its characteristics, and the time necessary to reproduce the experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 'tttt make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:////....e.pp..ccpc//aaaaaaa
Answer: [Yes]
Justification: Our work does not raise any ethics concerns with regard to any of the guidelines listed within the NeurIPS Code of Ethics. We have no human participants, we use publicly available data, and we believe our work does not have any broader potentially unintended societal impact.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
28

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: As mentioned at point #9, we do not believe that the Code of Ethics relevant for our work, reason for which we do not discuss any societal impacts in relation with our study.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do not release any new data and/or pre- gre-triined language models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
29

---

Justification:
We explicitly cite all of the used datasets and models and discuss their availability,
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, ppppesswittteeee thcode com dantsatsssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
We provide well-documented GitHub repo, which is linked to in the abstract.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve any human subjects in any manner.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
30

---

Answer: [NA]
Justification: Our work does not involve the participation of any human subjects in any manner.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We only use LLMs for the construction of pandas tables and matplotlib visualizations.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
31


Reviews:
[{"review_id": "G2PX0Y58R7", "reviewer": "Reviewer_BR5E", "summary": "It seems that normalizing biaffine scores for BiLSTMs gives significant improvement in terms of performance and parameter efficiency. This claim is supported by running dependency parsing task on multiple datasets, with several ablation studies. A possible reason for this is also discussed in section 3.", "questions": "- We see in some cases, e.g. in Figure 5 that the normalized version seems to drop in performance compared to the raw version as number of layers is increased. Do you have any guesses of why that might be?", "limitations": "We don't see a separate section for limitations, but in my opinion limitations are adequately and honestly discussed. Some with other text (e.g. size of dataset and kind of dataset when introduced), some in conclusions (last paragraph), and some in appendix.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- one possible cause of this result discussed sufficiently. Claim: maybe more layers cause variance to be in check and normalization is not much needed as small singular values decay.\n- many datasets utilized that have dependency parsing of different kinds (recipes, news, drug adverse effects, etc.)\n- normalizing shown to help with thorough experiments, with significance of outcomes measures satisfactorily\n- parameter efficiency for one particular case shown to be up to 85% more with normalization where with normalization only one layer is enough to match the performance of three without normalization\n\n\n**Weaknesses**\n- for now only seen for dependency parsing task\n- while a possible reason for raw version performing okay with higher number of layers is discussed, it is not known why in some cases the performance of normalized version drops and exhibits high variance when number of layers is increased (e.g. Appendix, line 489) [See Question related to this as well]\n- the gain for tagging tasks not that significant, but this adequately discussed", "was_revised": true, "final_justification": "I thank authors for additional information provided.\n\nI have read other reviews and I see the main concerns, and here are my thoughts about them:\n\n- Results on other languages e.g. low resource ones\n\nAlthough authors provide some results on more languages, I believe that is a separate question. The paper is good enough contribution only for high-resource languages\n\n- The contribution being simple\n\nYes perhaps, it is quite simple. I believe that is not grounds for rejection. And simple solutions should be published to benefit the wider ML / NLP community.\n\n- The contribution being too niche and perhaps better suited for more niche venues\n\nMaybe. I am not aware if it's kind of outside the area of NeurIPS.", "rebuttal": "We thank the reviewer for their insightful review.\n\nQ.1: for now only seen for dependency parsing task\n\nA.1: We acknowledge that our work targets semantic and syntactic dependency parsing tasks only. This was purposeful because this variant (Dozat and Manning) of biaffine scoring is widespread on these tasks in particular. While we plan to extend our normalization analysis to graph inference broadly (e.g. using QM9 for small molecules or citation networks), we believe that the current work presents a solid and focused contribution using one of the most common and SOTA architectures. Moreover, extending our analysis to non-linguistic tasks would involve many architectural variants (e.g. GNN types) that we feel are necessary to cover in a separate work built on the contribution of the current work. As an initial step towards using Graph Attention Networks (GAT), we have repeated the experiment by adding a stack of biaffine/GAT pairs to encode higher-order dependencies before the final biaffine layer. We achieve consistent results as shown in the table below. The performance increases for $L_\\phi \\in \\{1, 2\\}$, compared to $L_\\phi = 0$, but drops again at $L_\\phi = 3$:\n\n|   norm |   $L_\\phi$ | CoNLL04           | ADE               | SciERC            | enEWT             | SciDTB             | ERFGC             |\n|-------:|----:|:------------------|:------------------|:------------------|:------------------|:------------------|:------------------|\n|      n |   0 | 0.526 $\\pm 0.046$ | 0.517 $\\pm 0.038$ | 0.123 $\\pm 0.050$ | 0.610 $\\pm 0.009$ | 0.727 $\\pm 0.006$ | 0.536 $\\pm 0.013$ |\n|      n |   1 | 0.493 $\\pm 0.037$ | 0.509 $\\pm 0.022$ | 0.039 $\\pm 0.020$ | 0.583 $\\pm 0.011$ | 0.731 $\\pm 0.009$ | 0.599 $\\pm 0.008$ |\n|      n |   2 | 0.485 $\\pm 0.033$ | 0.509 $\\pm 0.025$ | 0.029 $\\pm 0.041$ | 0.540 $\\pm 0.012$ | 0.710 $\\pm 0.013$ | 0.611 $\\pm 0.015$ |\n|      n |   3 | 0.481 $\\pm 0.037$ | 0.487 $\\pm 0.040$ | 0.000 $\\pm 0.000$ | 0.511 $\\pm 0.015$ | 0.700 $\\pm 0.010$ | 0.585 $\\pm 0.009$ |\n|      y |   0 | 0.574 $\\pm 0.028$ | 0.556 $\\pm 0.037$ | 0.156 $\\pm 0.031$ | 0.671 $\\pm 0.007$ | 0.778 $\\pm 0.006$ | 0.607 $\\pm 0.009$ |\n|      y |   1 | 0.550 $\\pm 0.036$ | 0.587 $\\pm 0.015$ | 0.148 $\\pm 0.028$ | 0.696 $\\pm 0.005$ | 0.809 $\\pm 0.008$ | 0.639 $\\pm 0.007$ |\n|      y |   2 | 0.563 $\\pm 0.029$ | 0.534 $\\pm 0.023$ | 0.128 $\\pm 0.029$ | 0.657 $\\pm 0.007$ | 0.795 $\\pm 0.008$ | 0.637 $\\pm 0.007$ |\n|      y |   3 | 0.514 $\\pm 0.026$ | 0.543 $\\pm 0.024$ | 0.071 $\\pm 0.021$ | 0.596 $\\pm 0.010$ | 0.770 $\\pm 0.007$ | 0.616 $\\pm 0.006$ |\n\n**The normalization improves the scores in the case of GATs as well, showing that our method generalizes to other architectures, besides Transformers and BiLSTMS. We have included these results to the paper.**\n\nQ.2: while a possible reason for raw version performing okay with higher number of layers is discussed, it is not known why in some cases the performance of normalized version drops and exhibits high variance when number of layers is increased\n\nA.2: We have repeated the experiment with $L_\\psi > 3$ over three seeds and increased the amount of training steps to 20,000. All of the models made it to at least 6k steps (but not all of them made it to 8k or further, due to early stopping). We show the results at 2k, 4k, and 6k steps below (**rows with no high variances removed due to character limit**):\n\n2000 steps:\n\n| norm   | $L_\\psi$   | CoNLL04           | ADE               | SciERC            | enEWT             | SciDTB             | ERFGC             |\n|---:|---:|:------------------|:------------------|:------------------|:------------------|:------------------|:------------------|\n|  n |  7 | 0.606 $\\pm 0.012$ | 0.731 $\\pm 0.004$ | 0.260 $\\pm 0.026$ | 0.811 $\\pm 0.007$ | 0.898 $\\pm 0.006$ | 0.698 $\\pm 0.006$ |\n|  n |  8 | 0.594 $\\pm 0.012$ | 0.712 $\\pm 0.012$ | 0.248 $\\pm 0.002$ | 0.783 $\\pm 0.003$ | 0.882 $\\pm 0.007$ | 0.692 $\\pm 0.010$ |\n|  n | 10 | 0.565 $\\pm 0.012$ | 0.701 $\\pm 0.018$ | 0.233 $\\pm 0.022$ | 0.718 $\\pm 0.049$ | 0.839 $\\pm 0.034$ | 0.441 $\\pm 0.312$ |\n|  y |  9 | 0.215 $\\pm 0.289$ | 0.229 $\\pm 0.324$ | 0.260 $\\pm 0.039$ | 0.791 $\\pm 0.006$ | 0.883 $\\pm 0.007$ | 0.450 $\\pm 0.311$ |\n|  y | 10 | 0.386 $\\pm 0.261$ | 0.622 $\\pm 0.018$ | 0.168 $\\pm 0.120$ | 0.769 $\\pm 0.016$ | 0.879 $\\pm 0.009$ | 0.648 $\\pm 0.006$ |\n\n4000 steps:\n\n| norm   | $L_\\psi$   | CoNLL04           | ADE               | SciERC            | enEWT             | SciDTB             | ERFGC             |\n|---:|---:|:------------------|:------------------|:------------------|:------------------|:------------------|:------------------|\n|  n |  7 | 0.551 $\\pm 0.150$ | 0.719 $\\pm 0.017$ | 0.284 $\\pm 0.032$ | 0.833 $\\pm 0.022$ | 0.907 $\\pm 0.010$ | 0.700 $\\pm 0.005$ |\n|  n |  8 | 0.507 $\\pm 0.227$ | 0.711 $\\pm 0.008$ | 0.271 $\\pm 0.023$ | 0.813 $\\pm 0.030$ | 0.895 $\\pm 0.014$ | 0.694 $\\pm 0.008$ |\n|  n | 10 | 0.588 $\\pm 0.036$ | 0.701 $\\pm 0.033$ | 0.262 $\\pm 0.035$ | 0.755 $\\pm 0.059$ | 0.865 $\\pm 0.037$ | 0.442 $\\pm 0.313$ |\n|  y |  9 | 0.279 $\\pm 0.270$ | 0.244 $\\pm 0.322$ | 0.289 $\\pm 0.041$ | 0.813 $\\pm 0.022$ | 0.894 $\\pm 0.012$ | 0.457 $\\pm 0.311$ |\n|  y | 10 | 0.399 $\\pm 0.278$ | 0.651 $\\pm 0.032$ | 0.165 $\\pm 0.123$ | 0.797 $\\pm 0.030$ | 0.890 $\\pm 0.014$ | 0.656 $\\pm 0.009$ |\n\n6000 steps:\n\n| norm   | $L_\\psi$   | CoNLL04           | ADE               | SciERC            | enEWT             | SciDTB             | ERFGC             |\n|---:|---:|:------------------|:------------------|:------------------|:------------------|:------------------|:------------------|\n|  n |  7 | 0.581 $\\pm 0.130$ | 0.712 $\\pm 0.021$ | 0.307 $\\pm 0.041$ | 0.846 $\\pm 0.026$ | 0.910 $\\pm 0.010$ | 0.702 $\\pm 0.006$ |\n|  n |  8 | 0.466 $\\pm 0.254$ | 0.708 $\\pm 0.009$ | 0.292 $\\pm 0.039$ | 0.829 $\\pm 0.033$ | 0.901 $\\pm 0.015$ | 0.695 $\\pm 0.011$ |\n|  n | 10 | 0.592 $\\pm 0.036$ | 0.549 $\\pm 0.295$ | 0.288 $\\pm 0.047$ | 0.779 $\\pm 0.062$ | 0.877 $\\pm 0.035$ | 0.448 $\\pm 0.317$ |\n|  y |  9 | 0.329 $\\pm 0.282$ | 0.242 $\\pm 0.328$ | 0.310 $\\pm 0.047$ | 0.827 $\\pm 0.027$ | 0.899 $\\pm 0.012$ | 0.455 $\\pm 0.311$ |\n|  y | 10 | 0.410 $\\pm 0.280$ | 0.669 $\\pm 0.037$ | 0.187 $\\pm 0.141$ | 0.813 $\\pm 0.034$ | 0.896 $\\pm 0.014$ | 0.658 $\\pm 0.010$ |\n\nThe results show that once a model fails to find a good trajectory at the start, it likely remains on a poor trajectory. For example, the model with $L_\\psi \\in \\{7, 8\\}$ and norm = n does fine on CoNNL04 at 2,000 steps; however, one of the seeds then fails randomly at 4,000 steps and the performance remains poor past that point. For ADE, we can see that model performance with $L_\\psi = 10 $ and norm = n is fine at 2k and 4k steps, but suddenly drops with higher variance at 6k steps. Once again, this is because the model suddenly breaks and then remains broken for a specific seed. The table below does not show some of the poor performances because it reports the results of each seed at the best validation performance, up to 20k steps:\n\n| norm   | $L_\\psi$   | CoNLL04           | ADE               | SciERC            | enEWT             | SciDTB             | ERFGC             |\n|---:|---:|:------------------|:------------------|:------------------|:------------------|:------------------|:------------------|\n|  n |  7 | 0.668 $\\pm 0.015$ | 0.722 $\\pm 0.016$ | 0.380 $\\pm 0.032$ | 0.894 $\\pm 0.002$ | 0.929 $\\pm 0.002$ | 0.706 $\\pm 0.003$ |\n|  n |  8 | 0.625 $\\pm 0.028$ | 0.707 $\\pm 0.025$ | 0.392 $\\pm 0.016$ | 0.893 $\\pm 0.002$ | 0.926 $\\pm 0.002$ | 0.699 $\\pm 0.007$ |\n|  n | 10 | 0.635 $\\pm 0.038$ | 0.720 $\\pm 0.011$ | 0.373 $\\pm 0.016$ | 0.879 $\\pm 0.007$ | 0.917 $\\pm 0.002$ | 0.454 $\\pm 0.321$ |\n|  y |  9 | 0.454 $\\pm 0.274$ | 0.511 $\\pm 0.313$ | 0.384 $\\pm 0.024$ | 0.880 $\\pm 0.001$ | 0.916 $\\pm 0.001$ | 0.457 $\\pm 0.306$ |\n|  y | 10 | 0.478 $\\pm 0.283$ | 0.695 $\\pm 0.022$ | 0.237 $\\pm 0.169$ | 0.881 $\\pm 0.001$ | 0.911 $\\pm 0.002$ | 0.662 $\\pm 0.015$ |\n\nDespite the random crashes, increasing the number of steps increased the performance and reduced the Std for deeper stacks, compared to the original manuscript. While we still see instability for CoNLL04 and SciERC, it is reduced compared to the results shown in the original manuscript in Figure 5. Furthermore, the high variance almost disappears for ADE and ERFGC when training for a higher number of steps. Even more importantly, the results are identical for enEWT and SciDTB, with and without normalization, hinting at the fact that this is not inherently a problem of the model, but has to do with the nature of the datasets.\nMore specifically, we also see some instability for ERFGC **without** score normalization at $L_\\psi = 10$, and at $L_\\psi = 9$ with score normalization. Similarly high variance can be seen for ADE with normalization at $L_\\psi = 9$ (but not 10). Given that the high variances are produced by a few zero or near-zero scores, the instability seems to mostly be a product of the higher number of layers.\n\nWe have also analyzed the behavior of the gradients w.r.t. the depth of the layers. For SciERC (highest Std at $L_\\psi = 10$ in the original manuscript), the table below shows the head and tail for 14k steps of norm of the gradient for the 10 layers:\n\n|  step   |     0 |     1 |     2 |     3 |     4 |     5 |     6 |     7 |     8 |     9 |\n|------:|------:|------:|------:|------:|------:|------:|------:|------:|------:|------:|\n|     0 | 0.002 | 0.002 | 0.004 | 0.008 | 0.020 | 0.048 | 0.115 | 0.279 | 0.638 | 1.607 |\n|  ...  |       |       |       |       |       |       |       |       |       |       |\n| 13999 | 0.070 | 0.039 | 0.018 | 0.014 | 0.013 | 0.010 | 0.012 | 0.013 | 0.036 | 0.072 |\n\nSince the gradient evens out, we reckon it not to be the issue at hand here. It is likely that the higher parameter count of 10 layers simply necessitates more training steps.\n\nFormatting:\n\nWe commit to fix the issues raised by the reviewer w.r.t. line 38, line 208, and commit to improving the readability of Figs. 6 and 7."}, {"review_id": "dQnODJPxaO", "reviewer": "Reviewer_MP7r", "summary": "This work presents theoretical evidence and empirical results for the influence of scaling the scores produced by biaffine transformations in dependency parsing tasks. The authors find that the score variance produced by a lack of score scaling hurts model performance\nwhen predicting edges and relations. Experimental results demonstrate that dependency parsing models equipped with normalization can obtain better performance with substantially fewer trained parameters.", "questions": "See the weaknesses", "limitations": "yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:     \n1. Biaffine score function is a common and default component in dependency parsing tasks. The authors are very careful and aware of the potential problem in it. They analyze the relation between normalization and parameter quantity from theoretical and experimental perspectives.     \n2. This paper conduct extensive experiments to verify their claim.   \n\nWeaknesses:     \n1. I wonder whether this finding is still effective in other languages, especially low-resource languages.     \n2. This research question is too minor to have a significant impact on the broad research community. I think the main research content of this paper is not suitable for publication in NeurIPS. I recommend that the author pay attention to more targeted venues, so that researchers in specific sub-community can pay attention to this work.     \n3. Please change eh, ed, rh, and rd in figure 2 to the format of mathematical vector (i.e., $e^h$, $e^d$, $r^h$ and $r^d$), so they are consistent with the text and equation.", "was_revised": true, "final_justification": "The work lags behind the current state of the art in dependency parsing.", "rebuttal": "Q.1: I wonder whether this finding is still effective in other languages, especially low-resource languages.\n\nA.1: We thank the reviewer for suggesting multilingual settings. We have run the model on six non-English UD datasets (UD_Arabic-PADT, UD_Chinese-GSD, UD_Italian-ISDT, UD_Japanese-GSD, UD_Spanish-AnCora, UD_Wolof-WTB) and verified that our findings are consistent. In the following we summarized the multilingual results. We have added them to the paper. \n\n| score norm | $L_\\psi$ | UD_Arabic-PADT    | UD_Chinese-GSD    | UD_Italian-ISDT   | UD_Japanese-GSD   | UD_Spanish-AnCora   | UD_Wolof-WTB      |\n|:---|:----|:------------------|:------------------|:------------------|:------------------|:--------------------|:------------------|\n| $\\times$ | 0 | 0.538 $\\pm\\scriptstyle 0.005$ | 0.395 $\\pm\\scriptstyle 0.007$ | 0.563 $\\pm\\scriptstyle 0.006$ | 0.493 $\\pm\\scriptstyle 0.010$ | 0.554 $\\pm\\scriptstyle 0.004$   | 0.252 $\\pm\\scriptstyle 0.007$ |\n| $\\times$ | 1 | 0.723 $\\pm\\scriptstyle 0.008$ | 0.653 $\\pm\\scriptstyle 0.007$ | 0.792 $\\pm\\scriptstyle 0.003$ | 0.812 $\\pm\\scriptstyle 0.003$ | 0.775 $\\pm\\scriptstyle 0.003$   | 0.525 $\\pm\\scriptstyle 0.006$ |\n| $\\times$ | 2 | 0.745 $\\pm\\scriptstyle 0.004$ | 0.710 $\\pm\\scriptstyle 0.005$ | 0.826 $\\pm\\scriptstyle 0.003$ | 0.844 $\\pm\\scriptstyle 0.005$ | 0.807 $\\pm\\scriptstyle 0.002$   | 0.587 $\\pm\\scriptstyle 0.007$ |\n| $\\times$ | 3 | 0.748 $\\pm\\scriptstyle 0.004$ | 0.717 $\\pm\\scriptstyle 0.007$ | 0.832 $\\pm\\scriptstyle 0.002$ | 0.849 $\\pm\\scriptstyle 0.003$ | 0.808 $\\pm\\scriptstyle 0.002$   | 0.614 $\\pm\\scriptstyle 0.005$ |\n| $\\checkmark$ | 0 | 0.609 $\\pm\\scriptstyle 0.003$ | 0.479 $\\pm\\scriptstyle 0.007$ | 0.633 $\\pm\\scriptstyle 0.002$ | 0.585 $\\pm\\scriptstyle 0.005$ | 0.620 $\\pm\\scriptstyle 0.002$   | 0.305 $\\pm\\scriptstyle 0.005$ |\n| $\\checkmark$ | 1 | 0.737 $\\pm\\scriptstyle 0.007$ | 0.693 $\\pm\\scriptstyle 0.003$ | 0.820 $\\pm\\scriptstyle 0.004$ | 0.838 $\\pm\\scriptstyle 0.004$ | 0.801 $\\pm\\scriptstyle 0.002$   | 0.556 $\\pm\\scriptstyle 0.003$ |\n| $\\checkmark$ | 2 | 0.758 $\\pm\\scriptstyle 0.003$ | 0.736 $\\pm\\scriptstyle 0.005$ | 0.842 $\\pm\\scriptstyle 0.004$ | 0.859 $\\pm\\scriptstyle 0.004$ | 0.822 $\\pm\\scriptstyle 0.001$   | 0.613 $\\pm\\scriptstyle 0.008$ |\n| $\\checkmark$ | 3 | 0.759 $\\pm\\scriptstyle 0.005$ | 0.742 $\\pm\\scriptstyle 0.006$ | 0.845 $\\pm\\scriptstyle 0.002$ | 0.859 $\\pm\\scriptstyle 0.003$ | 0.823 $\\pm\\scriptstyle 0.002$   | 0.633 $\\pm\\scriptstyle 0.005$ |\n\nQ.2: This research question is too minor to have a significant impact on the broad research community. I think the main research content of this paper is not suitable for publication in NeurIPS. I recommend that the author pay attention to more targeted venues, so that researchers in specific sub-community can pay attention to this work.\n\nA.2: While it is true that semantic and syntactic dependency parsing are specific to communities in NLP, our paper contributes towards analyzing normalization phenomena in architectures used broadly in graph inference tasks (modeling edges as fully connected weighted graphs via attention mechanisms such as biaffine transformations, see [1,2,3] for reference). To further demonstrate this, we have conducted an experiment using Graph Attention Networks (GAT) by adding a stack of biaffine/GAT pairs. The purpose of this is to encode higher-order dependencies before the final biaffine layer. The results with this new architecture are consistent, as shown in the table below.\n\n|   norm |   $L_\\phi$ | CoNLL04           | ADE               | SciERC            | enEWT             | SciDTB             | ERFGC             |\n|-------:|----:|:------------------|:------------------|:------------------|:------------------|:------------------|:------------------|\n|      $\\times$ |   0 | 0.526 $\\pm 0.046$ | 0.517 $\\pm 0.038$ | 0.123 $\\pm 0.050$ | 0.610 $\\pm 0.009$ | 0.727 $\\pm 0.006$ | 0.536 $\\pm 0.013$ |\n|      $\\times$ |   1 | 0.493 $\\pm 0.037$ | 0.509 $\\pm 0.022$ | 0.039 $\\pm 0.020$ | 0.583 $\\pm 0.011$ | 0.731 $\\pm 0.009$ | 0.599 $\\pm 0.008$ |\n|      $\\times$ |   2 | 0.485 $\\pm 0.033$ | 0.509 $\\pm 0.025$ | 0.029 $\\pm 0.041$ | 0.540 $\\pm 0.012$ | 0.710 $\\pm 0.013$ | 0.611 $\\pm 0.015$ |\n|      $\\times$ |   3 | 0.481 $\\pm 0.037$ | 0.487 $\\pm 0.040$ | 0.000 $\\pm 0.000$ | 0.511 $\\pm 0.015$ | 0.700 $\\pm 0.010$ | 0.585 $\\pm 0.009$ |\n|      $\\checkmark$ |   0 | 0.574 $\\pm 0.028$ | 0.556 $\\pm 0.037$ | 0.156 $\\pm 0.031$ | 0.671 $\\pm 0.007$ | 0.778 $\\pm 0.006$ | 0.607 $\\pm 0.009$ |\n|      $\\checkmark$ |   1 | 0.550 $\\pm 0.036$ | 0.587 $\\pm 0.015$ | 0.148 $\\pm 0.028$ | 0.696 $\\pm 0.005$ | 0.809 $\\pm 0.008$ | 0.639 $\\pm 0.007$ |\n|      $\\checkmark$ |   2 | 0.563 $\\pm 0.029$ | 0.534 $\\pm 0.023$ | 0.128 $\\pm 0.029$ | 0.657 $\\pm 0.007$ | 0.795 $\\pm 0.008$ | 0.637 $\\pm 0.007$ |\n|      $\\checkmark$ |   3 | 0.514 $\\pm 0.026$ | 0.543 $\\pm 0.024$ | 0.071 $\\pm 0.021$ | 0.596 $\\pm 0.010$ | 0.770 $\\pm 0.007$ | 0.616 $\\pm 0.006$ |\n\nSimilar to BiLSTMs, the normalization improves the scores in the case of GATs as well, showing that our method generalizes to other architectures, besides Transformers and BiLSTMS. We have included these results to the paper.\n\nIn addition to our technical contribution, we believe that our work contributes to the broad NeurIPS community, by suggesting the importance of peculiar settings such as normalization in the training of neural network models. Moreover, we would like to respectfully point out that the NeurIPS guideline of acceptance suggests a work with \"high impact on at least one sub-area of AI\" (per grade 5), meaning that broad impact is not strictly required for acceptance. We appreciate the reviewer's consideration in this regard.\n\n- [1] Kazi et al. \"Differentiable Graph Module (DGM) for Graph Convolutional Networks\". 2023.\n\n- [2] P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, \"Graph attention networks\". 2017.\n\n- [3] J. Zhang, X. Shi, J. Xie, H. Ma, I. King, and D.-Y. Yeung, \"GAAN: Gated attention networks for learning on large and spatiotemporal graphs\". 2018.\n\nQ.3:  Please change eh, ed, rh, and rd in figure 2 to the format of mathematical vector (i.e., $e^h$, $e^d$, $r^h$, and $r^d$), so they are consistent with the text and equation.\n\nA.3: We have amended the formatting oversight as requested."}, {"review_id": "BwMfZShD9Z", "reviewer": "Reviewer_A3rQ", "summary": "This paper investigates the impact of score normalization in biaffine dependency parsing models. This work argued that the current dependency parsers using biaffine scoring are over parameterized because they do NOT the score normalization which is typically used in Transformer in attention mechanisms. The paper provided theoretical justification through implicit regularization theory. It shows that deeper networks reduce weight matrix rank and consequently score variance. It demonstrated that the normalizing biaffine scores can achieve comparable or even better performance with up to 85% fewer parameters.", "questions": "No.", "limitations": "The paper includes very limited discussions on limitations. I would suggest add some like the potential negative impact on datasets where current deep architectures are well-tuned, etc.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The connection between Transformer attention normalization and biaffine scoring is well-motivated. \n2. The paper includes experiments on 6 diverse datasets, covering both semantic (ADE, CoNLL04, SciERC, ERFGC) and syntactic (enEWT, SciDTB) dependency parsing. It achieves similar performance with 85% parameter reduction. It includes systematic exploration on different layer depths, hyperparameters, architectural components, etc. \n\nWeaknesses:\n1. While Claim 1 provides intuition, the proof is somewhat informal. The connection between rank reduction and variance reduction could be more rigorously established. \n2. The normalization benefits vary significantly across datasets. In addition, the main focus of the paper on BiLSTM-based architectures may limit applicability to modern Transformer-based parsers.", "was_revised": false, "final_justification": "", "rebuttal": "Q.1: While Claim 1 provides intuition, the proof is somewhat informal. The connection between rank reduction and variance reduction could be more rigorously established.\n\nA.1: We thank the reviewer for pointing out that our proof (and claim) could be made more rigorous. We have restated the claim and proof formally with a more standard analysis of covariance using the trace. Please see below:\n\n*Claim 1* (Monotonic Increase of Output Variance with Rank)\n\nLet $X \\in \\mathbb{R}^n$ be a random vector with covariance matrix $\\mathbf{K_{xx}} := \\text{Cov}(X) \\in \\mathbb{R}^{n \\times n}$, and let $\\mathbf{A} \\in \\mathbb{R}^{m \\times n}$ be a fixed matrix with singular value decomposition (SVD):\n\n$$\\mathbf{A} = \\sum_{i=1}^{min(m, n)} \\sigma_i \\mathbf{u}_i \\mathbf{v}_t^{\\top}$$\n\nwhere $\\sigma_1 \\geq \\sigma_2 \\geq ... \\geq 0$. Define the best rank-$r$ approximation of $\\mathbf{A}$ by truncated SVD as:\n\n$$\\mathbf{A_r} := \\sum_{i=1}^r \\sigma_i \\mathbf{u}_i \\mathbf{v}_t^{\\top}$$\n\nLet $Y_r = \\mathbf{A}_rX \\in \\mathbb{R}^m$ denote the image of $X$ under the rank-$r$ linear map. Then, the total variance of $Y_r$, as measured by the trace of its covariance matrix, increases monotonically with $r$:\n\n$$\\text{tr}(\\text{Cov}(Y_r)) \\leq \\text{tr}(\\text{Cov}(Y_{r + 1}))$$\n\n*Proof.* Let $\\mathbf{K_{xx}} := \\text{Cov}(X) \\in \\mathbb{R}^{n \\times n}$ denote the covariance matrix of the random vector $X$. Since covariance matrices are symmetric and positive semi-definite (PSD), we have $\\mathbf{K_{xx}} \\succeq 0$. Let $\\mathbf{A_r} := \\sum_{i=1}^r \\sigma_i \\mathbf{u}_i \\mathbf{v}_t^{\\top}$ be the rank-$r$ truncated SVD of $\\mathbf{A}$. Then the covariance matrix of $Y_r := \\mathbf{A_r} X$ is:\n\n$$\\text{Cov}(Y_r) = \\mathbf{A_r} \\mathbf{K_{xx}} \\mathbf{A_r}^{\\top}.$$\n\nTo evaluate the total variance we compute the trace:\n\n$$\\text{tr}\\left(\\text{Cov}(Y_r)\\right) = \\text{tr}\\left(\\mathbf{A_r}\\mathbf{K_{xx}} \\mathbf{A_r}^{\\top}\\right).$$\n\nUsing the cyclic property of the trace $\\left(\\text{tr}(ABC) = \\text{tr}(BCA)\\right)$ and symmetry of $\\mathbf{K_{xx}}$ we get:\n\n$$\\text{tr}\\left(\\mathbf{A_r}\\mathbf{K_{xx}} \\mathbf{A_r}^{\\top}\\right) = \\text{tr}\\left(\\mathbf{K_{xx}} \\mathbf{A_r}^{\\top} \\mathbf{A_r} \\right).$$\n\nNow define the matrix $\\mathbf{M_r} := \\mathbf{A_r}^{\\top} \\mathbf{A} \\in \\mathbb{R}^{n \\times n}$, which is PSD. As $r$ increases, $\\mathbf{A_r}$ includes more terms in its truncated SVD, so we have:\n\n$$\\mathbf{M_{r}} = \\sum_{i=1}^r \\sigma^2_i \\mathbf{v_i} \\mathbf{v_i}^{\\top}, \\quad \\mathbf{M_{r+1}} = \\mathbf{M_r} + \\sigma^2_{r+1} \\mathbf{v_{r+1}} \\mathbf{v_{r+1}}^{\\top}.$$\n\nThus:\n\n$$\\mathbf{M}_{r+1} \\succeq \\mathbf{M}_r.$$\n\nSince $\\mathbf{K_{xx}} \\succeq 0$, and since the trace of a product of PSD matrices respects Loewner order (i.e., if $A \\preceq B$, then $\\text{tr}(CA) \\leq \\text{tr}(CB)$ for all $C \\succeq 0$), we conclude:\n\n$$\\text{tr}(\\mathbf{K_{xx}} \\mathbf{M_r}) \\leq \\text{tr}(\\mathbf{K_{xx}}\\mathbf{M_{r+1}}),$$\n\nwhich implies:\n\n$$\\text{tr}\\left(\\text{Cov}(Y_r)\\right) \\leq \\text{tr}\\left(\\text{Cov}(Y_{r+1})\\right).$$\n\nTherefore, as the rank $r$ increases, the total variance $Y_r$ increases. $\\square$\n\nQ.2.1: The normalization benefits vary significantly across datasets.\n\nA.2.1: Thank you for raising this point. In order to show the generalizability of our results, we have carried out six additional experiments on multilingual syntactic dependency parsing, showing  consistent results for a wide variety of languages. \n\n| score norm | $L_\\psi$ | UD_Arabic-PADT    | UD_Chinese-GSD    | UD_Italian-ISDT   | UD_Japanese-GSD   | UD_Spanish-AnCora   | UD_Wolof-WTB      |\n|:---|:----|:------------------|:------------------|:------------------|:------------------|:--------------------|:------------------|\n| $\\times$ | 0 | 0.538 $\\pm\\scriptstyle 0.005$ | 0.395 $\\pm\\scriptstyle 0.007$ | 0.563 $\\pm\\scriptstyle 0.006$ | 0.493 $\\pm\\scriptstyle 0.010$ | 0.554 $\\pm\\scriptstyle 0.004$   | 0.252 $\\pm\\scriptstyle 0.007$ |\n| $\\times$ | 1 | 0.723 $\\pm\\scriptstyle 0.008$ | 0.653 $\\pm\\scriptstyle 0.007$ | 0.792 $\\pm\\scriptstyle 0.003$ | 0.812 $\\pm\\scriptstyle 0.003$ | 0.775 $\\pm\\scriptstyle 0.003$   | 0.525 $\\pm\\scriptstyle 0.006$ |\n| $\\times$ | 2 | 0.745 $\\pm\\scriptstyle 0.004$ | 0.710 $\\pm\\scriptstyle 0.005$ | 0.826 $\\pm\\scriptstyle 0.003$ | 0.844 $\\pm\\scriptstyle 0.005$ | 0.807 $\\pm\\scriptstyle 0.002$   | 0.587 $\\pm\\scriptstyle 0.007$ |\n| $\\times$ | 3 | 0.748 $\\pm\\scriptstyle 0.004$ | 0.717 $\\pm\\scriptstyle 0.007$ | 0.832 $\\pm\\scriptstyle 0.002$ | 0.849 $\\pm\\scriptstyle 0.003$ | 0.808 $\\pm\\scriptstyle 0.002$   | 0.614 $\\pm\\scriptstyle 0.005$ |\n| $\\checkmark$ | 0 | 0.609 $\\pm\\scriptstyle 0.003$ | 0.479 $\\pm\\scriptstyle 0.007$ | 0.633 $\\pm\\scriptstyle 0.002$ | 0.585 $\\pm\\scriptstyle 0.005$ | 0.620 $\\pm\\scriptstyle 0.002$   | 0.305 $\\pm\\scriptstyle 0.005$ |\n| $\\checkmark$ | 1 | 0.737 $\\pm\\scriptstyle 0.007$ | 0.693 $\\pm\\scriptstyle 0.003$ | 0.820 $\\pm\\scriptstyle 0.004$ | 0.838 $\\pm\\scriptstyle 0.004$ | 0.801 $\\pm\\scriptstyle 0.002$   | 0.556 $\\pm\\scriptstyle 0.003$ |\n| $\\checkmark$ | 2 | 0.758 $\\pm\\scriptstyle 0.003$ | 0.736 $\\pm\\scriptstyle 0.005$ | 0.842 $\\pm\\scriptstyle 0.004$ | 0.859 $\\pm\\scriptstyle 0.004$ | 0.822 $\\pm\\scriptstyle 0.001$   | 0.613 $\\pm\\scriptstyle 0.008$ |\n| $\\checkmark$ | 3 | 0.759 $\\pm\\scriptstyle 0.005$ | 0.742 $\\pm\\scriptstyle 0.006$ | 0.845 $\\pm\\scriptstyle 0.002$ | 0.859 $\\pm\\scriptstyle 0.003$ | 0.823 $\\pm\\scriptstyle 0.002$   | 0.633 $\\pm\\scriptstyle 0.005$ |\n\nFurthermore, we believe it is important to keep into account that we found the difference in performance to be statistically significant on all datasets, as mentioned in Figure 3.\n\nQ.2.2: In addition, the main focus of the paper on BiLSTM-based architectures may limit applicability to modern Transformer-based parsers.\n\nA.2.2: We would like to point out that the parser with which we experiment is Transformer-based, with the Dozat and Manning biaffine layer still being the standard for dependency parsing. Even without using BiLSTM layers and fully fine-tuning the Transformer, we have found that normalization provides statistically significant benefits, as shown in Figure 4 for the SciERC dataset. In Appendix B.6, we also show this more generally on all four SemDP datasets with BERT-base, BERT-large, DeBERTa-base, and DeBERTa-large. The effect is particularly pronounced for DeBERTa-base."}, {"review_id": "3zc2Wo3U1Y", "reviewer": "Reviewer_mCBU", "summary": "This work studies the effect of normalization on the scores produced by biaffine functions in dependency parsing tasks. The authors demonstrate, both theoretically and empirically, a lack of normalization necessarily results in overparameterized parser models, where the extra parameters compensate for the sharp softmax outputs produced by high variance inputs to the biaffine scoring function. Experimental results show that similar or better performance can be obtained by reducing the amount of trained BiLSTM parameters by as much as 85%.", "questions": "See the weaknesses.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:     \n1. This paper explores a problem that is overlooked. Biaffine function is a standard implement, and few people pay attention to it and its relationship with parameter quantity.     \n2. Comprehensive experiments.     \n\nWeaknesses:    \n1. Although this research question is very detailed and easily overlooked, I don't think this contribution meets the acceptance bar of NeurIPS.     \n2. Are there consistent experimental results on other language datasets? Experimental conclusion should be verified on more languages.     \n3. What are experimental results if add a Normal Layer?", "was_revised": true, "final_justification": "First, I do not think the contribution well-suitable for NeurIPS. Second,  even in the community of *ACL, the contribution is not strong enough. Might be better for a workshop oriented to syntax parsing.", "rebuttal": "Q.1: Although this research question is very detailed and easily overlooked, I don't think this contribution meets the acceptance bar of NeurIPS.\n\nA.1: We agree with the reviewer that the topic of our contribution is indeed overlooked. Thus, our work is novel in terms of improving dependency parsing as an application. Furthermore, our contribution involves analyzing normalization phenomena in architectures used broadly in graph inference tasks (i.e., modeling edges as fully connected weighted graphs via attention mechanisms, such as biaffine transformations, see [1,2,3] for reference). To show this, we have repeated the experiment using Graph Attention Networks (GAT), using a stack of biaffine/GAT pairs to encode higher-order dependencies before the final biaffine layer. The results with this new architecture are consistent, as shown in the table below. Moreover, we have expanded the experiments to six multilingual settings as reported in our answer to the reviewer's next comment.\n\n|   norm |   $L_\\phi$ | CoNLL04           | ADE               | SciERC            | enEWT             | SciDTB             | ERFGC             |\n|-------:|----:|:------------------|:------------------|:------------------|:------------------|:------------------|:------------------|\n|      0 |   0 | 0.526 $\\pm 0.046$ | 0.517 $\\pm 0.038$ | 0.123 $\\pm 0.050$ | 0.610 $\\pm 0.009$ | 0.727 $\\pm 0.006$ | 0.536 $\\pm 0.013$ |\n|      0 |   1 | 0.493 $\\pm 0.037$ | 0.509 $\\pm 0.022$ | 0.039 $\\pm 0.020$ | 0.583 $\\pm 0.011$ | 0.731 $\\pm 0.009$ | 0.599 $\\pm 0.008$ |\n|      0 |   2 | 0.485 $\\pm 0.033$ | 0.509 $\\pm 0.025$ | 0.029 $\\pm 0.041$ | 0.540 $\\pm 0.012$ | 0.710 $\\pm 0.013$ | 0.611 $\\pm 0.015$ |\n|      0 |   3 | 0.481 $\\pm 0.037$ | 0.487 $\\pm 0.040$ | 0.000 $\\pm 0.000$ | 0.511 $\\pm 0.015$ | 0.700 $\\pm 0.010$ | 0.585 $\\pm 0.009$ |\n|      1 |   0 | 0.574 $\\pm 0.028$ | 0.556 $\\pm 0.037$ | 0.156 $\\pm 0.031$ | 0.671 $\\pm 0.007$ | 0.778 $\\pm 0.006$ | 0.607 $\\pm 0.009$ |\n|      1 |   1 | 0.550 $\\pm 0.036$ | 0.587 $\\pm 0.015$ | 0.148 $\\pm 0.028$ | 0.696 $\\pm 0.005$ | 0.809 $\\pm 0.008$ | 0.639 $\\pm 0.007$ |\n|      1 |   2 | 0.563 $\\pm 0.029$ | 0.534 $\\pm 0.023$ | 0.128 $\\pm 0.029$ | 0.657 $\\pm 0.007$ | 0.795 $\\pm 0.008$ | 0.637 $\\pm 0.007$ |\n|      1 |   3 | 0.514 $\\pm 0.026$ | 0.543 $\\pm 0.024$ | 0.071 $\\pm 0.021$ | 0.596 $\\pm 0.010$ | 0.770 $\\pm 0.007$ | 0.616 $\\pm 0.006$ |\n\nScore normalization improves performance also when using GATs, showing that our method generalizes to other architectures, besides Transformers and BiLSTMS. We have included these results to the paper.\n\nIn addition to our technical contribution, we believe that our work contributes to the broad NeurIPS community, by suggesting the importance of peculiar settings such as normalization in the training of neural network models. Moreover, we would like to respectfully point out that the NeurIPS guideline of acceptance suggests a work with \"high impact on at least one sub-area of AI\" (per grade 5), meaning that broad impact is not strictly required for acceptance. We appreciate the reviewer's consideration in this regard.\n\n*[1] Kazi et al. \"Differentiable Graph Module (DGM) for Graph Convolutional Networks\". 2023.\n\n*[2] P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, \"Graph attention networks\". 2017.\n\n*[3] J. Zhang, X. Shi, J. Xie, H. Ma, I. King, and D.-Y. Yeung, \"GAAN: Gated attention networks for learning on large and spatiotemporal graphs\". 2018.\n\nQ.2: Are there consistent experimental results on other language datasets? Experimental conclusion should be verified on more languages.\n\nA.2: We thank the reviewer for pointing this out. We performed multilingual experiments using six non-English UD datasets (UD_Arabic-PADT, UD_Chinese-GSD, UD_Italian-ISDT, UD_Japanese-GSD, UD_Spanish-AnCora, UD_Wolof-WTB) and verified that our finding is consistent, as shown in the table below.\n\n| score norm | $L_\\psi$ | UD_Arabic-PADT    | UD_Chinese-GSD    | UD_Italian-ISDT   | UD_Japanese-GSD   | UD_Spanish-AnCora   | UD_Wolof-WTB      |\n|:---|:----|:------------------|:------------------|:------------------|:------------------|:--------------------|:------------------|\n| $\\times$ | 0 | 0.538 $\\pm\\scriptstyle 0.005$ | 0.395 $\\pm\\scriptstyle 0.007$ | 0.563 $\\pm\\scriptstyle 0.006$ | 0.493 $\\pm\\scriptstyle 0.010$ | 0.554 $\\pm\\scriptstyle 0.004$   | 0.252 $\\pm\\scriptstyle 0.007$ |\n| $\\times$ | 1 | 0.723 $\\pm\\scriptstyle 0.008$ | 0.653 $\\pm\\scriptstyle 0.007$ | 0.792 $\\pm\\scriptstyle 0.003$ | 0.812 $\\pm\\scriptstyle 0.003$ | 0.775 $\\pm\\scriptstyle 0.003$   | 0.525 $\\pm\\scriptstyle 0.006$ |\n| $\\times$ | 2 | 0.745 $\\pm\\scriptstyle 0.004$ | 0.710 $\\pm\\scriptstyle 0.005$ | 0.826 $\\pm\\scriptstyle 0.003$ | 0.844 $\\pm\\scriptstyle 0.005$ | 0.807 $\\pm\\scriptstyle 0.002$   | 0.587 $\\pm\\scriptstyle 0.007$ |\n| $\\times$ | 3 | 0.748 $\\pm\\scriptstyle 0.004$ | 0.717 $\\pm\\scriptstyle 0.007$ | 0.832 $\\pm\\scriptstyle 0.002$ | 0.849 $\\pm\\scriptstyle 0.003$ | 0.808 $\\pm\\scriptstyle 0.002$   | 0.614 $\\pm\\scriptstyle 0.005$ |\n| $\\checkmark$ | 0 | 0.609 $\\pm\\scriptstyle 0.003$ | 0.479 $\\pm\\scriptstyle 0.007$ | 0.633 $\\pm\\scriptstyle 0.002$ | 0.585 $\\pm\\scriptstyle 0.005$ | 0.620 $\\pm\\scriptstyle 0.002$   | 0.305 $\\pm\\scriptstyle 0.005$ |\n| $\\checkmark$ | 1 | 0.737 $\\pm\\scriptstyle 0.007$ | 0.693 $\\pm\\scriptstyle 0.003$ | 0.820 $\\pm\\scriptstyle 0.004$ | 0.838 $\\pm\\scriptstyle 0.004$ | 0.801 $\\pm\\scriptstyle 0.002$   | 0.556 $\\pm\\scriptstyle 0.003$ |\n| $\\checkmark$ | 2 | 0.758 $\\pm\\scriptstyle 0.003$ | 0.736 $\\pm\\scriptstyle 0.005$ | 0.842 $\\pm\\scriptstyle 0.004$ | 0.859 $\\pm\\scriptstyle 0.004$ | 0.822 $\\pm\\scriptstyle 0.001$   | 0.613 $\\pm\\scriptstyle 0.008$ |\n| $\\checkmark$ | 3 | 0.759 $\\pm\\scriptstyle 0.005$ | 0.742 $\\pm\\scriptstyle 0.006$ | 0.845 $\\pm\\scriptstyle 0.002$ | 0.859 $\\pm\\scriptstyle 0.003$ | 0.823 $\\pm\\scriptstyle 0.002$   | 0.633 $\\pm\\scriptstyle 0.005$ |\n\nQ.3: What are experimental results if add a Normal Layer?\n\nA.3: As regards the addition of LayerNorm, experimental results on the effect of adding normalization layers were present in the original manuscript. We refer the reviewer to Appendix B.5 for our analysis. In Table 8, we show that the addition of LayerNorm layers in between the BiLSTM layers is detrimental on average, over all datasets."}]

Author rebuttal:

Meta-review: The paper presents results of link prediction, in particular for dependency parsing, after implementation of score normalisation.  The authors motivate the normalisation theoretically based on architecture and exchange out a corresponding module and its required parameters, arguing that lack of normalisation resulted in overparamterisation.  In the rebuttal, the authors also showed that their trade-off is also valid for other (non-linguistic) tasks.  While the paper is heavily motivated with the dependency parsing task, it is clear that the observations are about network topology and not limited to this task.  While the paper aligns with well-established principles in deep learning, the reviewers do not point out any redundancy in this particular finding.  The task itself, however, dependency parsing, is a central task to NLP.  For researchers without access to compute resources beyond a laptop, an 85% reduction in parameters is of central importance.
Decision: Accept (poster)