Title: Dependency Parsing is More Parameter-Efficient with Normalization
Venue: NeurIPS 2025
Accepted: yes
Keywords: semantic dependency parsing

normalization

biaffine attention
Abstract: Dependency parsing is the task of inferring natural language structure, often approached by modeling word interactions via attention through biaffine scoring. 
This mechanism works like self-attention in Transformers, where scores are calculated for every pair of words in a sentence. However, unlike Transformer attention, biaffine scoring does not use normalization prior to taking the softmax of the scores. In this paper, we provide theoretical evidence and empirical results revealing that a lack of normalization necessarily results in overparameterized parser models, where the extra parameters compensate for the sharp softmax outputs produced by high variance inputs to the biaffine scoring function. We argue that biaffine scoring can be made substantially more efficient by performing score normalization.
We conduct experiments on semantic and syntactic dependency parsing in multiple languages, along with latent graph inference on non-linguistic data, using various settings of a $k$-hop parser.
We train $N$-layer stacked BiLSTMs and  evaluate the parser's performance with and without normalizing biaffine scores.
Normalizing allows us to achieve state-of-the-art performance with fewer samples and trainable parameters.
Code: https://github.com/paolo-gajo/EfficientSDP


Peer reviews:
comments: SUMMARY:
This work studies the effect of normalization on the scores produced by biaffine functions in dependency parsing tasks. The authors demonstrate, both theoretically and empirically, a lack of normalization necessarily results in overparameterized parser models, where the extra parameters compensate for the sharp softmax outputs produced by high variance inputs to the biaffine scoring function. Experimental results show that similar or better performance can be obtained by reducing the amount of trained BiLSTM parameters by as much as 85%.

STRENGTHS AND WEAKNESSES:
Strengths:     
1. This paper explores a problem that is overlooked. Biaffine function is a standard implement, and few people pay attention to it and its relationship with parameter quantity.     
2. Comprehensive experiments.     

Weaknesses:    
1. Although this research question is very detailed and easily overlooked, I don't think this contribution meets the acceptance bar of NeurIPS.     
2. Are there consistent experimental results on other language datasets? Experimental conclusion should be verified on more languages.     
3. What are experimental results if add a Normal Layer?

QUESTIONS:
See the weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-21

comments: SUMMARY:
This paper investigates the impact of score normalization in biaffine dependency parsing models. This work argued that the current dependency parsers using biaffine scoring are over parameterized because they do NOT the score normalization which is typically used in Transformer in attention mechanisms. The paper provided theoretical justification through implicit regularization theory. It shows that deeper networks reduce weight matrix rank and consequently score variance. It demonstrated that the normalizing biaffine scores can achieve comparable or even better performance with up to 85% fewer parameters.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The connection between Transformer attention normalization and biaffine scoring is well-motivated. 
2. The paper includes experiments on 6 diverse datasets, covering both semantic (ADE, CoNLL04, SciERC, ERFGC) and syntactic (enEWT, SciDTB) dependency parsing. It achieves similar performance with 85% parameter reduction. It includes systematic exploration on different layer depths, hyperparameters, architectural components, etc. 

Weaknesses:
1. While Claim 1 provides intuition, the proof is somewhat informal. The connection between rank reduction and variance reduction could be more rigorously established. 
2. The normalization benefits vary significantly across datasets. In addition, the main focus of the paper on BiLSTM-based architectures may limit applicability to modern Transformer-based parsers.

QUESTIONS:
No.

LIMITATIONS:
The paper includes very limited discussions on limitations. I would suggest add some like the potential negative impact on datasets where current deep architectures are well-tuned, etc.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
This work presents theoretical evidence and empirical results for the influence of scaling the scores produced by biaffine transformations in dependency parsing tasks. The authors find that the score variance produced by a lack of score scaling hurts model performance
when predicting edges and relations. Experimental results demonstrate that dependency parsing models equipped with normalization can obtain better performance with substantially fewer trained parameters.

STRENGTHS AND WEAKNESSES:
Strengths:     
1. Biaffine score function is a common and default component in dependency parsing tasks. The authors are very careful and aware of the potential problem in it. They analyze the relation between normalization and parameter quantity from theoretical and experimental perspectives.     
2. This paper conduct extensive experiments to verify their claim.   

Weaknesses:     
1. I wonder whether this finding is still effective in other languages, especially low-resource languages.     
2. This research question is too minor to have a significant impact on the broad research community. I think the main research content of this paper is not suitable for publication in NeurIPS. I recommend that the author pay attention to more targeted venues, so that researchers in specific sub-community can pay attention to this work.     
3. Please change eh, ed, rh, and rd in figure 2 to the format of mathematical vector (i.e., $e^h$, $e^d$, $r^h$ and $r^d$), so they are consistent with the text and equation.

QUESTIONS:
See the weaknesses

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIt seems that normalizing biaffine scores for BiLSTMs gives significant improvement in terms of performance and parameter efficiency. This claim is supported by running dependency parsing task on multiple datasets, with several ablation studies. A possible reason for this is also discussed in section 3.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- one possible cause of this result discussed sufficiently. Claim: maybe more layers cause variance to be in check and normalization is not much needed as small singular values decay.\n- many datasets utilized that have dependency parsing of different kinds (recipes, news, drug adverse effects, etc.)\n- normalizing shown to help with thorough experiments, with significance of outcomes measures satisfactorily\n- parameter efficiency for one particular case shown to be up to 85% more with normalization where with normalization only one layer is enough to match the performance of three without normalization\n\n\n**Weaknesses**\n- for now only seen for dependency parsing task\n- while a possible reason for raw version performing okay with higher number of layers is discussed, it is not known why in some cases the performance of normalized version drops and exhibits high variance when number of layers is increased (e.g. Appendix, line 489) [See Question related to this as well]\n- the gain for tagging tasks not that significant, but this adequately discussed\n\nQUESTIONS:\n- We see in some cases, e.g. in Figure 5 that the normalized version seems to drop in performance compared to the raw version as number of layers is increased. Do you have any guesses of why that might be?\n\nLIMITATIONS:\nWe don't see a separate section for limitations, but in my opinion limitations are adequately and honestly discussed. Some with other text (e.g. size of dataset and kind of dataset when introduced), some in conclusions (last paragraph), and some in appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}