Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning
Authors: Sarkar Snigdha Sarathi Das
Haoran Ranran Zhang
Peng Shi
Wenpeng Yin
Rui Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: Sequence Labeling
Low Resource Learning
Sparse Finetuning

Abstract: Unified Sequence Labeling that articulates different sequence labeling problems such as Named Entity Recognition, Relation Extraction, Semantic Role Labeling, etc. in a generalized sequence-to-sequence format opens up the opportunity to make the maximum utilization of large language model knowledge toward structured prediction. Unfortunately, this requires formatting them into specialized augmented format unknown to the base pretrained language model (PLMs) necessitating finetuning to the target format. This significantly bounds its usefulness in data-limited settings where finetuning large models cannot properly generalize to the target format. To address this challenge and leverage PLM knowledge effectively, we propose FISH-DIP, a sample-aware dynamic sparse finetuning strategy that selectively focuses on a fraction of parameters, informed by feedback from highly regressing examples, during the fine-tuning process. By leveraging the dynamism of sparsity, our approach mitigates the impact of well-learned samples and prioritizes underperforming instances for improvement in generalization. Across five tasks of sequence labeling, we demonstrate that FISH-DIP can smoothly optimize the model in low resource settings offering upto 40% performance improvements over full fine-tuning depending on target evaluation settings. Also, compared to in-context learning and other parameter-efficient fine-tuning approaches, FISH-DIP performs comparably or better, notably in extreme low-resource settings. The source code of FISH-DIP will be available at [this URL](https://github.com/psunlpgroup/FISH-DIP)

Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning
Sarkar Snigdha Sarathi Das', Ranran Haoran Zhang', Peng Shi', Wenpeng Yin',
Rui Zhang'
Pennsylvania State University, UUniversity of Waterloo
{ ss fdd525, haoranz6, wenpeng, rmz5227) @psu. edu; peng.shi shi wehhiiwwwerroo...omm
Abstract
Unified Sequence Labeling articulates differ- ent sequence labeling tasks such as Named Entity Recognition, Relation Extraction, Se- mantic Role Labeling, etc. in a generalized sequence-to-sequence format. Unfortunately, this requires formatting different tasks into spe- cialized augmented formats which are unfa- miliar to the base pretrained language model (PLMs). This necessitates model fine-tuning and significantly bounds its usefulness data- limited settings where fine-tuning large mod- els cannot properly generalize to the target for- mat. To address this challenge and leverage PLM knowledge effectively, we propose FISH- DIP, sample-aware dynamic sparse finetun- ing strategy. It selectively finetunes fraction of parameters informed by highly regressing examples during the fine-tuning process. By leveraging the dynamism of sparsity, ap- proach mitigates the impact of well-learned samples and prioritizes underperforming in- stances for improvement in generalization. Across five tasks of sequence labeling, we demonstrate that FISH-DIP can smoothly opti- mize the model in low-resource settings, offer- ing up to 40% performance improvements over full fine-tuning depending on target evaluation settings, Also, compared to in-context learn- ing and other parameter-effiee effficiett fine-tuning (PEFT) approaches. FISH-DIP DIP performs com- parably or better, notably in extreme low- resource settings. The source code of FISH- DIP will he available at: https:/ / thub. http:///wmmpproop/r//sss----
upe semplo
PLM
Periodic Update
Training
Sample
Sample
parametee sparsity
Figure 1: Overview of FISH-DIP for Unified Low- Resource Sequence Labeling. All augmented training samples (Table I) are first channeled through PLMs (e.g- T5-large) to calculate the training loss. Top-n most re- gressing samples are chosen for calculating parameter importance, for k% sparsity mask. In backpropagation. this sparsity mask is applied for sparse finetuning. Pa- rameter importance updated dynamically using the feedback from underperforming samples.
2023). reasonable way to address this gap is to linearize these structured prediction tasks into an augmented lanugage representation (Paolini et al., 2021; Liu et al., 2022b) that frames all sequence labeling tasks in unified sequence-tor suueence-o-sequencee format (See examples in Table 1). This allows to capture the structured information while align- ing the prediction and training objectives of large language models. Significantly, when paired with this unified format, the pretrained language models (PLMs) exhibit an impressive ability to effectively generalize to different tasks, capitalizing on the inherent potential for knowledge transfer.
1 Introduction
Sequence Labeling tasks such as Named Entity Recognition, Relation Extraction, Semantic Role Labeling, etc. aim assign target labels to the components of a sequence from of categor- ical classes. Accurate sequence labeling requires learning complex structural dependency, where even state-of-the-as pretrained LLMs like 5PT-3.5- turbo fail to show acceptable performance (Li et al.,
Despite these advantages, fine-tuning PLMs accommodate such an "augmented" format for se- quence labeling in data-limited low-resource envi- ronments is significantly challenging, which has

---

Tisk T
Input.X X
Transfomaaiin 2'y that produces Augmonted NL Target X
[ Over 900 million dollars Imonctary of foreign cupital was
Over 900 million US dollars fareign capiial was actually utilized, inereasing nearly 40% compared with the period previons year.
NER
octually utilized, increasing nearly 40 % percent) ] compared the same period previons year date) )-
the Middle Adriatic Basin, there evidence of Permian volcanismn abserved on the Vis island and as volcanic islands of Jahuka and Brussik,
relationship between | Jabuika J and Adriatic Basin located or nosr to body of water
Relation Extraction
1831, the President of United States Location [ James Garfield I person I liver in = Orange, Ohio I. was born [ Orange. Ohiol locution )-
1831, the 20th Presideni of the United States, James Garfield, was born Orunge. Ohio.
Joint ER extraction
Useerj: bi, can help me find a place [Ageat]: I have 13 hotels on the north side of town, do you have any preferences [User]: yes, there any expensive ones? I also would like free parking as well.
the north side!;
I beliet hotel area north, hotel book day not given, hotel book people not given, hotel book stay not given, hotel internet not given, hotel name given, hotel parking yes, hotel price yange expensive, hotel star not given, hotel type hotel [ belief)) ]
DST
The lawyers have renewed their arguments Texas eight other states where the defense is! permitted under state
lawyers renewed their arguments | TTTeass eight other states IAM-LOC Il where! R-AM-LOC) ] [ the defense |AA permitte under state law AMM-LOC
SRL
Table 1: We employ augmented language to unified sequence labeling tasks. Example transformations for Named Entity Recognition (NER), Relation Extraction, Joint Entity Relation extraction, Dialogue State Tracking (DST), and Semantic Role Labeling (SRL).
not been thoroughly investigated previously. For instance, most statc-of-the-arl Parameter Efficient Fine-Tuning (PEFT) methods, such as LoRA (Hu et al., 2021), Adapter (Houlsby et al., 2019), Com- pactor (Karimi Mahabadi al., 2021), (T4)² (Liu et al., 2022a), etc., introduce new trainable param- eters for altering the output. Although effective in sentence classification or generation tasks, we find these methods often underperform in data-limited low-resource sequence labeling tasks as shown in our results, primarily due to the insufficient avail- able data for training the added parameters to ac- commodate the new augmented language format.
ples are channeled through PLMs (e.g. T5-large,, from which we can calculate losses for each of the samples. Then under-performing samples are cho- sen for calculating parameter importance, which in turn is leveraged to determine sparsity mask for sparse finetuming. As training progresses, the importance of parameters is updated to accom- modate the evolving learning requirements of both the model parameters and training samples. Conse- quently, FISH-DIP ensures that the fraction of the model parameters chosen is primarily influenced by the most pertinent samples, effectively helping in smoother optimization (Figure 4) and generaliza- tion while sustaining the enhancement in few-shot performance.
In the context of data-limited low-resource uni- versal sequence labeling tasks, we identified fixed Fisher Information sparsity training (Sung et al., 2021) as a promising direction. Instead of intro- ducing new parameters to train from scratch, it selects and finetunes the most influential parame- ters from the existing well-trained PLMs, yielding results on par with full fine-tuning (fine-tuning all the parameters of the model). However, our in- vestigations reveal that a sample-awar aware dynamic selection of parameter importance can lead to a sig- nificant performance boost in few-sample settings. Thus, we propose leveraging FISHer information for Sparse Fine-tuning with Dynamic Importance of Parameters (FISH-DIP), a method of sparse fine- tuning that dynamically chooses and adjusts a small fraction of the most relevant parameters based on the feedback from the available samples with the highest loss, prioritizing those that exhibit highest deviation from the desired outcome. As shown in Figure! 1, augmented natural language training sam-
We conduct comprehensive testing of FiSH-DIP across a range of tasks, including Named En tity Recognition (Sang and De Meulder, 2003; Weischedel et al., 2013), Relation Extraction (Han et al., 2018), Joint Entity-Relation Extraction (Walker et al., 2006), Semantic Role Labeling (Carrers and Marquez, 2005), and Dialogue State Tracking (Eric et al., 2019). Our findings reveal that actoss diverse sew-shot settings, FISH-DIP sig- nificantly enhances the few-shot performance of universal sequence labeling tasks. Besides, unlike other methods (Lu et al., 2022) that require large- scale data and pretraining, FiSH-DIP only relies on prudent optimization strategy to augment perfor- mance. This unique feature makes it particularly valuable for future applications in low-resource lan- guage contexts, where external resources may be extremely scarce. Furthermore, FISH-DIP consis- tently outperforms other popular Parameter Effi- cient Finetuning (PEFT) methods, demonstrating

---

the efficacy of dynamic parameter selection in low resource settings. Our contributions are below:
After well-trained model generates an output sequence. specialized decoding schemes are uti- lized to properly parse the output. Specifically, all special tokens are removed, and class-type informa- tion extracted to produce cleaned output. Then, at the token level, Dynamic Frogramming-basee Needleman-Wunsee alignment algorithm (Needle- man and Wunsch, 1970) is used for robust align- ment of inputs and outputs, even in the case of noisy generation. Output classes that do not match any target types are discarded in the process.
We broadly explore the field of data-limiten low-resource sequence labeling from uni- fied perspective, domain that has predomi- nantly remained underexplored in spite of the inherent challenges it presents in capturing structural information.
We propose sample-aware dynamic sparse fine-tuning scheme, FISH-DIP, that dynami- cally updates the parameter importance with the feedback from the most regressing sam- ples, subsiding the influence of already trained samples. This helps in better generalization, particularly in low-resouree settings.
2.2 Fisher Information and Sparsity Mask
Fisher information indicates the importance of parameter for a particular task. Prior works in ma- chine learning have leveraged this for pruning and model compression (Theis et al., 2018; Singh and Alistarh, 2020). In this regard, Fisher informa- tion matrix Fø E [ needs be calculated. However, with pretrained language models, is computationally intractable because of the | x [@] scale of computation where is the number parameters. To get around this computational com- plexity. Sung et al. (2021) proposed using average of the squared gradient of the model's output as ap- proximate Fisher information. Although it requires taking expectation over ouput distribution, in su- pervised setting due to the availability of label yi- this "Empirical Fisher" can simply calculated for "heuristic" paramter importance as below:
We rigorously test our proposed method across five tasks in seven datasets a wide range of low-resource settings. In most cases, FISH-DIP outperforms other ap- proaches while finetuning only fraction of the parameters of the model.
2 FISH-DIP for Unified Low-Resource Sequence Labeling
In this section, we first discuss how different se- quence labeling problems can be transformed into a unified "augmented" format (Section 2.1). Next, we discuss how Fisher information is leveraged to calculate parameter importance (Section 2.2). Fi- nally, we discuss how FisH-DIP leverages sample- aware dynamism to boost sample performance in sequence labeling (Section 2.3).
N
1 Fo R
N Σ Vo log (v)(aa)))
where po(y) denotes the output probability y: with parameters 0 given input x;, N is the number of samples, and Fg C TRo Using the calculated empirical Fisher information of the parameters, top k% parameters mask for sparse fine-tuning, while keeping the rest of the network fixed. Sung et al. (2021) found that calculating this FISH mask at the start of training with a small number of randomly selected train- ing samples (e.g. N R 1024) and keeping the importance mask fixed during training can provide similar performance as full finetuning.
2.1 Unified Sequence Labeling by Augmented Language
be kept for creating FISH
Given a sequence labeling task T. and few-shot training samples (o,, 344 E T where x; denotes the input sentences, and yı denotes the associated ground truth predictions (e.g., target entities, rela- tions, etc.), we can apply transformation function FT such that F Vi, S) results in augmented language text Z3 using special tokens S where S [. This essentially converts stan- dard sequence prediction tasks into sequence-to- sequence format, and the output text can encode all the structured information required for sequence la- beling. Some examples of such transformation. FF used in Paolini et al. (2021) are given in Table
2.3 Dynamic Sample Aware Sparse Tuning: FISH-DIP
In low-resource settings, employing fixed sparsity can quickly fit the limited number of training sam- ples. However, this accelerated learning process

---

can result in a rapid change of parameter impor- lance, thereby necessitating the update of the pa- rameter importance. With fixed sparsity mask, de- creasing losses for some samples may cause in- creased losses for others, resulting in an overall irregular update trend.
Algorithm 1: FISH-DIP
Require: Low-resource training Data Xtr.
Train loss Ly,, Augmentation
function Fy. special tokens S,
Pretrained Language Model PLM, sparsity k%, update steps regressing examples total training steps T Initialize sparse mask M E TR/ zeros for in range(O. T) do
To alleviate these issues, we propose dynamic sample aware sparse fine-tuning, where we period- ically update parameter importance based on the feedback from most regressing training samples. Since samples with lower training errors do not take part in parameter selection, their associated pa- rameters remain unperturbed. This approach helps mitigate overfitting and fosters smoother training progression (Figure 4). More specifically,
Sample minibatch (X,Y) X
X Fr(X, Y, /lc.g.. Table X = PLM(X) losses = Ler (X',
if mod m = 0 then
// /eealcculaee parameter importance and sparsity mask
Fo
Σ [[ Etop,)
(oologp(()lzz)) (2)
n
Calculate parameter importance Fo using Eq. 2 with topn most regressing examples
where top, indicates the losses of top-n most re- gressing training examples. Algorithm demon- strates how FISH-DIP can be used finetuning the model low-resource sequence labeling.
sort all the parameters using Fe and select top k% parameters
Reset M and set the selected parameters
In short, this dynamic calculation of parameter
end
importance helps in two fronts: (i) We recalcu- late the parameter importance that parame- ters that are chosen for training focus on regressing examples. Similar boosting algorithms, they try to optimize the less performant examples to en- sure overall smooth optimization across samples (Figure 4). (ii) Samples that have already been op- timized do not take part in feedback process for parameter importance calculation. As result, they cannot overfit the model resulting in improved low- resource performance from finetuning.
Backprop and Apply sparsity mask M Update Parameters
end
use joint accuracy evaluation metric. Ta- ble 2 summarizes different datasets and evaluation metrics used for each of these tasks.
Low-resource Settings We compare the perfor- mance of FISH-DIP across different tasks and datasets in a wide range of few-sample settings. In CoNLL'03, CoNLL'O4, and OntoNotes we eval- uate the performance with 1,, 5%, and 10% the full training dataset. For large datasets e.g. CoNLL'OS SRL and MultiWoz 2.1 dataset, we com- pare the performance in 0.5%, 1%, and 5% data settings. Finally, in FewRel, we evaluate the per- formance in 5-way 1-shot, 5-way 5-shot, 10-way 1-shot, and 10-way 5-shot settings as standard.
3 Experiment Setups
We describe our tasks and datasets, evaluation set- tings, and different types of baselines.
Tasks and Datsets To evaluate the efficacy of FISH-DIP, we cover five different tasks across seven datasets in Named Entity Recognition (Sang and De Meulder, 2003; Weischedel et al., 2013), Relation Extraction (Han et al., 201 8), Joint Entity Relation Extraction (Roth and Yih, 2004; Walker al., 2006), Dialogue State Tracking (Eric al., 2019), and Semantic Role Labeling (Carreras and Marquez, 2005) tasks. For evaluation of NER and SRL, we use entity Fl-score, while in joint ER we use entity and relation Fl scores. Finally, in DST,
Baselines We compare the efficacy of FiSH-Do -IP against (i) full finetuning based baselines that in- clude task-specific state-of-the-art methods. TANL (Paolini et al., 2021), and constrained generation- based method ASP (T5-large) (Liu et al., 2022b) (ii) sparse Finetuning baseline FISH mask (Sung

---

Task
Dataset
Low-resouree Setting
Metric
NER
CoNLL'U3, OntoNotes
1%, 5%, 10%
FI FI
Relation Extraction
FewRel 1.0 CoNLL '04, ACE2005
5/10 way, 1/5 shol
Joint ER
1%, 5%, 10%
Entity FI, Relation FI
Dialogue State Tracking
MultiWoz
0.5%, 1%, 5%
Joint Accuracy F1
Semantic Role Labeling
CoNLL'05 SRL WSJ and Brown 0.5%, 1%, 5%
Table Our experiments cover diverse and comprehensive sequence labeling tasks, datasets, and evaluation settings.
et al., 2021) in 1% and 5% sparsity, and (iii) in- context baseline when applicable. For all methods, we use Ta-large as the backbone PLM. Table 3-7 lists these comparison results in detail across different tasks, datasets, and settings. We discuss these tasks, datasets, and baselines in more detail Appendix B. Finally, we also test the efficacy of FISH-DIP against different parameter-efficient finetuning approaches, such as Adapter (Houlsby et al., 2019), Compacter (Karimi Mahabadi et al., 2021), LoRA (Hu al., 2021), (IA)² (Liu et al., 2022a), and prefix tuning (Li and Liang, 2021) as shown in Figure 2.
sity) (Sung et al., 2021) often loses performance against full finetuning counterpart. This particu- larly demonstrates how importance of parameters changes over the course of training which can be extremely useful in improving performance in low- resource sequence labeling tasks. In named entity recognition (CoNLL'O3 and OntoNotes dataset), FiSH-DrP shows improved performance over all the other baselines while updating only a fraction of the parameters. On the other hand, in dedicated few-shot relation extraction dataset FewRel that traditional N-way K-shot settings, FISH-DIP shows substantial performance improvement over full-fnetuning TANL (Paolini et al.. 2021). Fixed FISH mask (Sung al., 2021) which showed sub- optimal performance low-resource NER, also shows comparable performance in this scenario.
4 Results and Analysis
Our results and analysis aim to answer the follow- ing research questions:
RQ I: What is the overall efficacy of FISH- DIP across low-resource sequence labeling tasks compared with baselines (Section 4.1)?
As we shift our focus towards more intricate tasks such as Joint Entity Relation extraction, FISH- DIP consistently exhibits superior performance compared to all baseline models. This task poses significant challenge to all baselines operating in low-resource settings. As we dig deeper, we find that while some baselines may achieve good entity-Fl score, they do with significant de- crease in relation-Fi score. In contrast, FISH-DIP consistently matches or exceeds the performance of all other baselines, regardless of low-resource settings. Interestingly, the baseline model ASP (T5- large) exhibits comparable performance to FISH- DIP in higher resource settings, despite struggling in very low resource settings (e.g. 1% data). This can potentially be attributed to the task-specific pa- rameterization of the ASP model. Although FISH- DIP may lead to even greater performance improve- ments when applied on ASP, it will compromise the generalization and multitasking capabilities funda- mental to unified sequence labeling. Therefore, we only apply FISH-DIP for augmentation transforma- tions as outlined in (Paolini et al., 2021).
RQ 2: How does FISH-DIP perform when com- pared with other PEFT approaches (Section 4.2)?
RQ 3: How does FISH-DIP compare with in- context learning using LLMs (Section 4.3)?
RQ 4: What is the effect of dynamically choos- ing regressing samples for calculating parameter importance for sparsity mask (Section 4.4)?
RQ 5: How is performance of FISH-DIP in- fluenced by the percentage of sparsity (Section 4.5)?
4.1 Overall Results
Table 3 - demonstrates that across different low- resource settings, FISH-DIP finetuning offers sub- stantial performance improvement over all other basclines. In particular, for very low resource set- tings (e.g. 1% training data) FISH-DIP does only offer significant performance uplift but also shows lower performance variance demonstrating its robustness. We also find that, while trained in identical settings, fixed sparse masks (1 %%%5% % spar-
Finally, in dialogue state tracking (Table 6) and semantic role labeling (Table 7), we observe a

---

FT Mask
CoNLL'03
OntoNotes
Method
Sparsity
1%
5%
10%
1%
5%
10%
TANL
100%
81.31pg
90.26g.4
90.68p.a
77.84pg
83.78g.
85.61p.
ASP (T5-large)
100%
49. 70.002
89.130.3
89.24u.4
Fixed FISH
1%
78. 78..4
86.981.8
87. 87.40p.4
75.001.1
82.75g.2
83.30
Fixed FISH
5%
81.011.0
87.08o.s
88. 88.24n.6
76.21o.4
83.0 83.08g.4
84.88n.2
FISH-DIP
1%
86.39..4
91.500.6
91.70p.6
78.69p.4
84.43g.3
86. p6.00p.4
Table 3: Results for Named Entity Recognition. We report Fl scores in CoNLL'03 and OntoNotes datasets.
FewRel
Mask Sparsity
[dy Prefic Tiring Compasiat Adapttr
Method
5-way 1-shot
5-way shot
10-way 1-shat
10-way 5-shot
LoMA saTibl anams
BERT-EM EM
100%
88.9
82.8
MERT-FM--MTR
BERT Pair
100%
85.7
89.5
76.8
81.8
TANL
100%
87.27
91.85.6
83. 6o2
90.8sa
Fixed FISH
1%
9
96.843
87.6ks
92.613
Fixed FISH
94.8
98.3k.
80.2g0
94.2g4
92.9p.
98.43.
89. 89.86.o
94.22.1
20
10-
Table 4: Results for Relation Extraction. We report F1 scores in FewRel dataset.
Relanonn %%
Relationgss
Entlly Entlyy@1 Relationg100
%
Entity@5%
Entiyyatos
Evaluation Setting
similar trend where FiSH-DIP outperforms other baselines, particularly in extremely low-resource settings while matching or exceeding the perfor- mance of baselines more resource-rich environ- ments. In dialogue state tracking, we also evaluate the performance in more challenging 0.5% data, where TANL, Fixed FISH(I %)), Fixed FISH (5%) and, FISH-DIP achieve joint accuracy scores of 34.17. 30.0, 30.9, and 35.87 respectively. These findings provide further evidence of the robustness of our FISH-DIP model irrespective of available resources.
Figure 2: Performance comparison of FISH-DIr against different PEFT techniques in CoNLL'04 Joint Entity Relation dataset. FISH-DIP matches or exceeds all other approaches demonstrating robustness all low-resource settings.
habadi al., 2021), and (IA)² (Liu al., 2022a). Moreover, we show the performance FiSH-DIP without
(parameter importance updates don't reflect regressing samples). All methodologies use identical hyperparameter config- urations. The comparative outcomes are illustrated in Figure 2, indicating that FISH-DIP demonstrates either comparable superior performance com- pared other baseline approaches. This shows its robustness across various scenarios. Notably, under more demanding circumstances with lim- ited data availability, other techniques experience performance degradation, whereas FISH-DIP con- sistently maintains optimal performance.
4.2 Comparison against other PEFT approaches
Because of the expanding size of language mod- els, over the past few years several techniques have been proposed for parameter-efficient fine- tuming (Houlsby et al., 2019; Karimi Mahabadi et al., 2021; Hu et al., 2021; Liu et al., 2022a; Li and Liang, 2021). Since FISH-DIP achieves no- table performance improvements by leveraging a prudent sparse update procedure, we want to com- pare the performance benefit against other PEFT approaches. To this end, we compare the perfor- mance in Joint Entity-Relation CoNLL'Os dataset. We use Huggingface's PEFT library for LoRA (Hu al., 2021) and Prefix Tuning (Li and Liang, 2021). We also also show the performance of Adapter (Houlsby et al., 2019), Compacter (Karimi Ma-
4.3 Comparison against In-Context Learning approaches with GPT-3.5-turbo
Recently, GPT-3.5-turbo gained attention for its high-quality text generation, supposed reasoning abilities, and impressive in-context learning (ICL.) with minimal few-shot samples. We evaluate the performance of ICL with OpenAT's GPT-3.5-turbo

---

FT Mask
CoNLL'04
ACE'0S
Method
Sparsity
1%
5%
10%
1%
10%
Entity
TANL
100%
46.0512.1
74.06p.1
79.1.o.s
72.52,.
81.35g
83,96y.8
DygiePP
100%
59. 10.64g..
74.191.0
77.82
ASP (T5-1)
100%
60.36p.2
75.02p.5
82.34p.6
61.551.3
79.58u.7
84.680.
Fixed FISH
1%
37.0815.6
71.273.0
77.272.5
70.561.8
79.57a.7
82.26y.7
Fixed FISH
5%
51.375.6
73.20 12
79.240.s
73.30
80.00g.s
82.20p.3
FISH-DIP
1%
56.882.
77.251
81.50n.
75.08
82.
84.64n
Relation
TANL
100%
14.13g.4
40.63g.6
14.37
15. 15.622.5
36.79
45.22,.1
DygiePP
100%
7.002.0
26.701.7
45.740.9
ASP (T5-I)
100%
13.80g.2
40.93 2
55.76p.
6.400.1
37.731.
85....
Fixed FISH
1%
12. 12.29
34.354.5
46 46.32
12,411.9
29.072.1
36,851.2
Fixed FISH
19.40
40.092.5
53.883.3
16.85g
33.841.6
42.312.
FISH-DIP
1%
21 21.63g.
49. 49.202.8
57. 07.082.4
19.012.
39.85p.
47.150.3
Table Results for Joint Entity Relation Extraction. We report Fl scores in CoNLL'O4 and ACE'OS datasets.
FT Mask
MultiWoz 2.1
FT Mask
CoNLL'05 WSJ
Method
Method
Sparsity
1%
5%
Sparsity
0.5%
1%
5%
TRADE
100%
12.58
31.17
TANL
100%
67.3415
74.50p.s
82.10o.3
SGPDST
100%
32.11
43.14
Fixed FISH
1%
61.31ss
65.60
78.35
-BT-BART BART
100%
28.55
37.71
Fixed FISH
5%
66.53
11p.5
79.5607
DS2-TS
100%
33.76
44.20
FISH-DIP
70.37
75. 75.490
80.9002
TANL
100%
34.96
47.37
Method
FT Mask
CoNLL'OS 05 Brown
IC-DST (GPT-Neo-2.7B)
(ICL)
16.70
26.90
Sparsity
0.5%
1%
5%
IC-DST (CodeGen-2.7B)
(ICL)
20.72
29.62
TANL
100%
58.711.2
66.53g.
73.69o.s
Fixed FISH
1%
34.17
41.70
Fixed FISH
1%
51.00..1
55.41e.9
68.362.4
Fixed FISH
37.50
45.70
Fixed FISH
5%
58.152.7
62.861.
70.31
FISH-DIP
42.33
47.24
10.4
08.077.5
72.42pu
Table 6: Results for Dialog State Tracking. We report Joint Accuracy in MultiWoz 2.1 dataset. Here (ICL.) denotes in-context learning methods where finetuning is not involved.
Table 7: Results for Semantic Role Labeling. We report Fl score in CONIL'05 WSJ and Brown datasets.
ture structural information for in-context learning.
4.4 Effect of Sample Aware Dynamic Sparse Tuning in Optimization
API in CoNLL'O4 and ACE2005 Joint Entity Re- lation datasets. The template used in prompting GPT-3.5-turbo is shown in Appendix C. We ran- domly select few-shot samples and demonstrate them as in-context pairs until reaching the token limit, leaving space only for the target input sen- tences. Finally, we give the test input sentence to generate output. The results are shown in Figure 3.
Figure 4 shows the trend of samplewise loss opti- mization for CoNLL'04 04 1% dataset. We find that across training steps, FISH-DIP focuses on opti- mizing samples with the highest losses because of selecting parameters associated with them. While doing that, parameters only relevant to the samples with lower losses are frozen, preventing irrelevant updates. In comparison, fixed sparsity mask results in samplewise irregular update trend, decreasing losses for some samples while increasing others. This illustrates the importance of dynamic parame- ter sparsity in few-sample optimization of PLMs.
We find that the ICL performance varies signifi- cantly based on the chosen evaluation setting and dataset. In CoNLL'04, the performance of ICL in 1% data outperforms all other baselines in this setting. However, in 5% and 10% data, the num- bers are suboptimal, demonstrating that it cannot take advantage of the extra lata.In ACE'OS 05 GPT- 3.5-turbo shows very poor performance, especially in relation extraction. The complexity of ACE'05 with more complex entities and relation targets may have made more difficult for the models to cap-
4.5 Effect of Sparsity Percentage, k
For uniformity across all our experiments, we have opted for a stable parameter sparsity of = 1% In Table 8, we analyze the influence various

---

Sourte Fish
CPT33 Soarto
Semple lasses using REW OFF
Samole lesses using 5% spariity mask
Rith DMP
2011
0
20
16-
Bostinght
Enteyorte
bomybio
350
305
305
400 300 203
300
600
A CEMLLSS
IN ACEE
Figure 4: Illustration of samplewise optimization loss between FISH-DIP and 5% Fixed FISH mask sparse fine-tuning in CoNLL'04 (1% data). Across the times- tamps, we see that FISH-DIP focuses on optimizing the samples having higher loss without compromising the learning of other samples. Compared to fixed sparse mask resulting in an overall smoother training trend.
Figure 3: Comparison against in-context learning us- ing GPT-3.5-turbo in different low resource setting in CoNLL'04 and ACE'05.
Mask Sparsity
CoNLL'04
1 1
5%
10 %
0.5%
56,451.1
76.37p.6
80.54g.
Fintity
1%
58. .8.552..4
78.05p.4
81.37o.s
5%
55.90s.9
76.67n.4
79.00o.1
Low-Resource Sequence Labeling Different learning approaches (Vinyals et al., 2016; Snell et al., 2017: Geng et al., 2019; Bao et al., 2019) have been leveraged in several NLP tasks to im- prove performance in few-shot settings. With the arrival of GPT-3 (Brown et al., 2020), in-context learning (Chen, 2022: Chen et al., 2023; Hu et al., 2022; Dong et al., 2022) started to become more commonplace for few-shot learning. However, in most previous works, few-shot/low-resource learning has been largely limited to sentence-level prediction, whereas sequence labeling tasks re- quire better-structured prediction capabilities for improved performance.
0.5%
22 2.227
49.22
56.032.3
Relation
1%
20.412.1
49. 19.78
57.032.7
5%
21.32s.1
46.41p.3
52.344.6
Table 8: Effect of different sparsity choice (0.5%, 1%. 5%) in FiSH-Dr DIP on CoNLL'04
sparsity levels on the performance of FISH-DIP in different low-resource settings.
is evident that the selection of parameter spar- sity, k has a potential impact on performance. While we have used 1% sparsity across all tests, tuning & based on the available data can potentially yield improved results. This highlights the impor- tance of carefully choosing the sparsity level in FISH-DIP. Hence, we recommend conducting ex- periments with different values of l for achieving the optimal performance with FISH-DIP.
Named Entity Recognition (NER) tasks are one of the first where different task-specific methods have been applied to improve few-shot NER perfor- mance (Yang and Katiyar, 2020; Das et al., 2021; Cui et al., 2021; Tong et al., 2021; Ma et al., 2022; Huang et al., 2022: Ming et al., 2022; Ji et al., 2022). In relation extraction, (Han et al., 2018) first introduced large-scale benehmark for evaluation in N-way K-shot settings. This benchmark is later used by following works to improve few-shot per- formance in a wide range of domain-centered meth- ods (Gao et al., 2019; Tong et al., 2021; Ma et al., 2022; Qu et al., 2020; Han et al., 2021). Other than that, (Hu et al., 2022; Wu et al., 2020; Lee et al., 2021; Shin et al., 2022) have applied in-context learning for dialogue state tracking, where they found that with in-context learning, gigantic lan- guage models can show noteworthy performance improvements over other models. These dedicated that few-shot techniques demonstrate improved per- formances but require extensive task-specific mod-
5 Related Work
Unified Sequence Labeling Paolini et al. (2021)
noted out the issues in discriminative structured pre- diction tasks and proposed for an augmented Natu- ral Lanugage format (Athiwaratkun et al., 2020) to complete all tasks in a seutence-to-sqqueee mat. Following their success, (Liu et al., 2022b) has attempted to further improve the performance of these approaches through constrained generation which requires explicit constraints tailored towards each of the tasks effectively taking away the advan- tage of structured prediction unification. Unfortu- nately, in low-resource settings, the usefulness of these techniques have not been explored leaving them largely suboptimal in those settings.
for-

---

Large language models are eling, limiting their applicability in a unified set- ting.
Acknowledgement
We would like to thank Cisco Systems for their gen- crous support through the Cisco Research Award for supporting this work. We would also like thank Nan Zhang, Yusen Zhang, and Vipul Gupta for their valuable feedback and suggestions.
Parameter Efficient Finetuning Different PEFT methods have been proposed for finetuning large language models efficiently. Small adapter layers that are added while keeping the rest of the net- work fixed was one of the first attempts at this efficient finetuning that proved to be quite use- ful (Houlsby et al., 2019). A follow-up approach (Karimi Mahabadi al., 2021) leverages matrix decomposition and low-rank parameterization for adapter weights. Later (Sung et al., 2021) has in- troduced FISH methods for selection of parameters to finetuned. On the other hand, recently (Hu et al., 2021) y) and (Liu et al., 2022a) introduced two other methods of PEFT where small number of new parameters are introduced for low-rank adap- tation and scaled activation respectively. This was recently made even more efficient by Dettmers et al. (2023) significantly reducing memory usage. An- other set of works have shown that is possible to get good performance by injecting prompts into in- put and optimizing the related layers (Li and Liang, 2021). While these techniques have demonstrated their utility in facilitating efficient fine-tuning, they might not possess the capability to modify the output format necessary accommodate the aug- mented language, especially the unified sequence labeling requirement in low-resource settings.
Limitations
While we primarily focus sequence labeling tasks, due the broad applicability it would be interesting to explore the efficacy of FISH-DIP in other NLU tasks, particularly in low-resource or few-shot scenarios. Notably our approach is par- ticularly effective extreme low-resource settings. However, it is worth noting that as the scale of available data increases, most methods show com- parable performance and room for improvement becomes narrower.
References
Ben Athiwaratkun, Cicero Nogueira dos Santos, Jason Krone, and Bing Xiang. 2020. Augmented natural language for generative sequence labeling. arXiv preprint arXiv:2009.1327
Yujia Bao, Menghua Wu. Shiyu Chang, and Regina Few-shot text classification with distributional signatures. arXiv preprint 1XXiv:908.00003
Barzilay. 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances neural information processing systems, 33: 877-19901
6
Conclusion
Unified sequence labeling poses significant dif- ficulties, particularly when dealing with limited resources necessitating the adoption of non- conventional augmented format. FISH-DIP tack- les this challenge through dynamic sparse finetun- ing. prioritizes underperforming instances and reduces the impact of well-learned samples, opti- mizing only a fraction of the parameters. Across five sequence labeling tasks, FISH-DIP demon- strated on on-par or consistent improvement over all other baselines, particularly in more challenging extremely low-resource scenarios. For future re- search, exploring the potential of FISH-DIP in other lbw-resourre/few--shot tasks (e.g. low-resource languages where external resources are extremely scarce) holds promise and warrants further investi- gation.
Xavier Carreras and Lluis Märquez. 2005. Introduction to the conll-2005 shared task: Semantic role labeling, In Proceedings of the ninth conference compu- rational natural language learning (CoNLI1-2005), pages 152-164
Jiawei Chen, Yaojic Lu, Hongyu Lin, Jie Lou, Wei Jia, Dai Dai, Hua Wu, Boxi Cao, Xianpei Han, and Le Sun. 2023. Learning in-context learn- ing for named entity recognition. arXiv preprint arXiv:2305.I 1038.
Wenhu Chen. 2022. few (1)-shot table reasoners. arXiv preprint arXiv.2210.06710.
Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang 2021. Template- named entity recognition us- ing bart. arXiv preprint arXiv:2106.07660

---

Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca J Passonneau, and Rui Zhang. 2021. Container: Few- shot named entity recognition via contrastive learn- ing. arXiv preprint arXiv:2109.07589
Bin Ji, Shasha Li. Shaoduo Gan, Jie Yu, Jun Ma, and Huijun Liu. 2022. Few-shot named entity recognition with entity-level prototypical network enhanced by dispersedly distributed prototypes. arXiv preprint arXiv:2208.08022
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023, Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305. 14314.
Rabech Karimi Mahabudi, James Henderson, and Se- bastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34; 1022--21.
Qingxiu Dong, Lei Li. Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint ar11::3311.02244
Chia- Hsuan Lee, Hao Cheng, and Mari Ostendorf. 2021. Dialogue state tracking with a language model using schema-driven prompting. arXiv preprint arXiv:2109.07500
Mihail Eric, Rahul Goel, Shachi Paul, Adarsh Kumar, Abhishek Sethi, Peter Ku, Anuj Kumar Goyal, San- chit Agarwal, Shuyang Gao, and Dilek Makkani-Turt 2019. Multiwoz 2.1: A consolidated multi- dialogue dataset with state corrections and state track- ing baselines. arxiv preprint arXiv: 1907.01669.
Bo Li, Gexiang Fang. Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. 2023. Evaluating chatgpt's 's information extraction capabilities: An sessment of performance, explainability, calibration, and faithfulness. arXiv preprint arXiv:2304. 1633.
Tianyu Gao. Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019. Fewrel 2.0: Towards more challenging few-shot relation classifi- cation. arXiv preprint arXiv:1910.07122
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00
Ruiying Geng, Binhua Li, Yongbin Li, Xiaodan Zhu, Ping Jian, and Jian Sun. 2019. Induction net- works for shot text classification. arXiv preprint arXiv:1902 10482.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mo- hta, Tenghao Huang. Mohit Bansal, and Colin A Raf- fel. 2022a. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Ad- vances Neural Information Processing Systems, 35:1950-1965
Jiale Han, Bo Cheng, and Wei Lu. 2021. Exploring task difficulty for few-shot relation extraction. arXiv preprint arXiv:210900577
Tianyu Liu, Yuchen Jiang, Nicholas Monath, Ryan Cot- terell, and Mrinmaya Sachan. 2022b. Autoregressive structured prediction with language models. arXiv preprint arXiv:2210 14698.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang. Yuan Yao,
Zhiyuan Liu, and Maosong Sun. 2018. Fewrel: A large-scale supervised few-shot relation classifica- tion dataset with state-of-the-art evaluation. arXiv preprint arxiv: 10147..
Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022. Uni- fied structure generation for universal information extraction. arXiv preprini mxXiv2203.122777
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficieet transfer learning for nlp. In In- ternational Conference on Machine Learning, pages 2790-2799. PMLR.
Jie Ma, Miguel Ballesteros, Srikanth Doss, Rishita Anubhai, Sunil Mallya, Yaser Al-Onaizan, and Dan Roth. 2022. Label semantics for few shot named entity recognition. arXiv preprint arXiv:2203.08955
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. arXiv preprint arXiv:2106.0 09685.
Hong Ming, Jiaoyun Yang, Lili Jiang, Yan Pan, and Ning An. 2022. Few-shot nested named entity recog- nition. arXiv preprint 22.:2222.00095
Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A Smith, and Mari Ostendorf. 2022. In-context learning for few-shot dialogue state tracking, arXiy preprint 2003::2003.08833 08568.
Saul B Needleman and Christian D Wunsch. 1970. A general method applicable to the search for simi- larities in the amino acid sequence of two proteins. Journal of molecular biology, 4533):44--453
Yucheng Huang, Kai He. Yige Wang. Xianli Zhang. Tieliang Gong, Rui Mao, and Chen Li. 2022. Copner: Contrastive learning with prompt guiding for few- shot named entity recognition. In Proceedings of the 29th International conference on computational linguistics, pages 2515-2527.
Giovanni Paolini, Ben Athiwaratkun, Jason Krone. Jie Ma, Alessandro Achille, Rishita Anubhai, Ci- cero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation be- tween augmented natural languages. arXiv preprint 11X.v:2101.05779.

---

Meng Qu Tianyu Gao, Louis-Pascaa Xhonneux, and Jian Tang. 2020. Few-shot relation extraction via bayesian meta-learning on relation graphs. In In- termational conference on machine learning, pages 7867-7876. PMLR.
Ralph Weischedel, Martha Palmer, Mitchell Mareus, Ed- uard Hovy, Sameer Pradhan, Lance Ramshaw, Nian- wen Xue, Ann Taylor, Jeff Kaufman, Michelle Fran- chini, al. 2013. Ontonotes release 5.0 Idc2013t19. Linguistic Data Consortium, Philadelphia, PA.
Dan Roth and Wen-tau Yih. 2004. linear program- ming formulation for global inference in natural language tasks. Technical report, Illinois Univ at Urbana-Champaign Dept of Computer Science.
Chien-Sheng Wu, Steven Hoi, and Caiming Xiong. 2020. Improving limited labeled dialogue state tracking with self-supervision. arXiv preprint arXiv:2010.1 13920.
Yi Yang and Arzoo Katiyar. 2020. Simple and effee- tive few-shot named entity recognition with struc- tured nearest neighbor learning. arXiv preprint a0X0v:2010.02405.
Erik F Sang and Fien De Meulder. 2003. Introduction the conll-2003 shared task: Language-indepeenden named entity recognition. arXiv preprins cs/0306050.
Jamin Shin, Hangyeol Yu, Hyeongdon Moon, Andrea Madotto, and Juneyoung Park. 2022. Dialogue summaries as dialogue states (ds2), template-guided summarization for few-shot dialogue state tracking. arXiv preprint arXiv:2203.01552.
Sidak Pal Singh and Dan Alistarh. 2020. Woodfisher: Efficient second-order approximation for neural net- work compression. Advances in Neural Information Processing Systems, 33:18098-188 18109.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. Ad- vances neural information processing systems, 30.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. arXiv preprint arXiv: AXXx:1906.03158
Yi-Lin Sung, Varun Nair, and Colin A Raffel. 2021. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193-24222 242055.
Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. 2018. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXiv:/801.05787.
Meihan Tong, Shuai Wang, Bin Xu, Yixin Cao, Minghui Liu, Lei Hou, and Juanzi Li. 2021. Learning from miscellancous other-class words for few-shot named entity recognition. arXiv preprint arXiv:2106. 15167.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016. Matching networks for shot learning. Advances in neural information processing systems, 29.
David Wadden, Ulme Wennberg, Yi Luan, and Han- naneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. arXiv preprint
Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilin- gual training corpus. Linguistic Data Consortium, Philadelphia, 57:45.

---

A Implementation Details
model with 64 relations. During few-shot setting. we do the testing on 20 disjoint test relations. For these target relations, each of them have N-way K- shot support sets with which models are finetuned. Finally, inference is done on query set to calculate the Few-Shot performance. Table 4 compares the performance of FISH-DIP with TANL full finetun- ing (Paolini et al., 2021). fixed sparse FT (Sung et al., 2021), and several dedicated relation extrac- tion methods for few shot relation extraction such as BERT-EM (+MTB) (Soares al., 2019), BERT- PAIR (Gao et al., 2019)
We'll primarily use Huggingface with PyTorch deep learning framework to implement and com- plete all our experiments. For the choice of PLM, we choose T5-large due to its better performance and moderate size (770 million). Our experiments are conducted on 8x NVIDIA A6000 48GB GPUs. For FISH-DIP, we choose sparsity k = 1%, steps, m = 100, regressing examples, n = 15 as the hyperparameters. Consequently, FISH mask is re- calculated after every A
m
epochs. In low
BIAL Jamples
botch_sze
resource settings, given limited number of training samples, this recalibration should only introduce a minimal increment in training. While for simplic- ity, we fixed m to 100, modulating m dynamically, updating it more frequently during initial training phases and less so as the training progresses can be an interesting future direction towards further improvement. For fine-tuning hyperparameters (learning rate and epochs), we followed settings suggested in (Paolini et al., 2021).
Joint Entity and Relation Extraction In this task, given a sentence, the target is to extract en- tities and a set of relations between pairs of enti- ties. We evaluate the performance of FISH-DIL DIP in Joint Entity and Relation extraction in CoNLL'O '04 and ACE2005 Joint ER dataset. While ACF2005 is moderately sized dataset having ~7500 train- ing samples, CoNLL' 04 is a small dataset having ~000 training samples. We show the performance comparison in CoNLL'04 and ACE2005 in Table 5. Like other experiments, we compare primar- ily against full finetuning TANL, fixed sparsity mask. Moreover, we also compare against the per- formance of popular method used in this domain DygiePP (Wadden al., 2019), and autoregres- sive method ASP (Liu al., 2022b), both of which require task-specific
For all other baselines, we used their respective implementations. For LoRA and Prefix Tuning, we used Huggingface PEFT with LoRA rank and prefix# in prefix tuming, chosen from 5, 10, 20, 30, and 40 (best performance).
B Task, Dataset, and Baseline Details
Named Entity Recognition In named entity recognition (NER), given sentence we need to ex- tract all the relevant entities. In CoNLL'03 dataset, there are four target classes. and training set size of 14k. On the other hand, OntoNotes have 18 classes with over 60k+ training samples. Ta- ble 3 shows the performance comparison in these two datasets respectively. We compare the perfor- mance against TANL full finetuning (Paolini et al., 2021), Fixed sparse mask finetuning (Sung et al., 2021). We also compare the performance against ASP (Liu et al., 2022b) which also models some structured prediction tasks into an autoregressive format, however, also requires designing a task- specific constrained set of actions which restricts itself from being used in a unified scheme.
Dialogue State Tracking In dialogue state track- ing, given a history of dialogue turns between a user and an agent, who is assisting the user, we have to find the dialogue state, i.c. get the value for each slot from predefined list. It has over 10k dialogue sets over seven distinct domains. We com- pare against TANL full fine-tuning (Paolini et al., 2021) and fixed sparsity finetuning (Sung et al., 2021). Moreover, we compared against dedicated DST methods (TRADE (Wu et al., 2020), SGPDST (Lee et al., 2021), DS2 - BART and DS2 T5 (Shin et al., 2022)) for few-shot performance evaluation. For in in-context learning bassidg-based method IC-DST (Hu et al., 2022) results are only reported with GPT- Neo 2.7B and CodeGen 2.7B. We don't include the results for Codex-Da Vinci (reportedly 250 times larger than T5- 175 billion parameters) since OpenAl has discontinued support for it, which was also used for achieving state-of-the ffthe---- art results in this task.
Relation Extraction Given sentence and head and tail entities, this task requires classifying the re- lation between them. In FewRel (Han et al., 2018) dataset, standard N-way K-shot settings is followed for few-shot performance evaluation. Following standard setting in this dataset, we meta-train the

---

Semantic Role Labeling Given an input sen- tence and predicate, this task aims predict list of arguments and types to get the predicate- argument structure sentence. More specifically, we need to find all the arguments corresponding to the given predicate in the input sentence. Table 7 compares the performance of FISH-DIP with TANL full fine-tuning (Paolini et al., 2021) and fixed spar- sity finetuning (Sung al., 2021) in CONLL'05 WSJ and Brown datasets.
In-Context Learning
For in-context learning with gpt-3.5-turbo (0301) for joint entity relation, randomly selected in-context demonstrations are drawn from the pool of samples within low-resource environment, ensuring their adherence the token limit of the model. In this regard, we use the following template:
# Instruction
Identify and mark all entities in the input sentence with square brackets and assign them from the following entity list: [list of entities]. If an entity has relation another entity, indicate it with a vertical bar and the name of the relation after the type. Relation type should be one from following list: [list of relations]. Some demonstrations are shown below:
# Demonstrations
Input: Tolkien wrote The Lord of the Rings.
Output: Tolkien person wrote The Lord of the Rings book author Tolkien
Input: (Test Sample) Output:


Reviews:
[{"review_id": "X4zLoPCamU", "reviewer": "Reviewer_YNan", "summary": "", "questions": "A. How much computational overhead does FISH-DIP introduce compared to the baselines? Does it significantly slow down the training? What are the effects of varying the hyper-parameter m (update steps)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the comprehensive review and insightful comments provided. Please find our responses to your comments below:\n\n**W1:** We acknowledge the oversight regarding the clear specification of the baseline model inside the original draft. In our revised manuscript, we will explicitly state that all models utilize *T5-large* as the baseline. This should enhance clarity and eliminate any potential ambiguities.\n\n**W2/Q1:** We are grateful for highlighting this aspect. Since our paper mainly revolves around very low resource sequence labeling in a unified context, its emphasis is on the efficacy of our approach relative to other methods. That being said, on the computational front, the efficiency of *FISH-DIP* can, at its best, match that of the original *FISH* mask, but this is contingent on the number of necessary training steps. The updating frequency of the *FISH* mask, set at $m$ steps, implies that the mask is recalculated after every $ \\approx m/ \\lceil {num\\\\_samples}/ {batch\\\\_size} \\rceil $ epochs. Given the limited training samples in our low-resource context, this recalibration should introduce only a minimal increment in training duration.\n\nFurthermore, regarding the hyperparameter $m$, we've standardized it to 100 training steps for simplicity. Nevertheless, an intriguing avenue for future work could be to modulate $m$ dynamically, updating it more frequently during initial training phases and less so as training progresses.\n\nTo reiterate, our central focus is on the performance outcomes of unified sequence labeling. Consequently, a deep dive into these computational intricacies somewhat extends beyond our paper's intended scope.\n\n**W3:** This is a very good point and we concur with your observation. The advantage of our proposed technique is its independence from additional datasets/resources and its reliance on a generalizable strategy to augment performance. This characteristic makes it especially potent for low-resource languages where external resources are extremely limited. As a natural extension, its application with multilingual-T5 variants to further enhance performance in similar low-resource language tasks might be very interesting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a simple and effective extension to Fisher-information-based fine-tuning methods with a fixed sparsity mask. The benefit of this extension is convincingly demonstrated within the low-resource sequence2sequence setting on multiple standard information extraction benchmarks. The positive per-sample effect of dynamic sparse tuning vs fixed sparse tuning is illustrated in by the smooth sample-wise optimization plot in Figure 4.", "reasons_to_reject": "- It is inferred but not spelled out that the baseline (T5 large) is the same for FISH-DIP vs Fixed FISH 1% and Fixed FISH 5%. This should be spelled out more clearly in the experiments section, not the appendix.\n- Time and memory efficiency of dynamically re-computing the parameter importance and sparsity matrices is not discussed.\n- How does this method extend to truly low-resource settings, such as information extraction for low-resource languages?", "questions_to_authors": "A. How much computational overhead does FISH-DIP introduce compared to the baselines? Does it significantly slow down the training? What are the effects of varying the hyper-parameter m (update steps)?", "paper_topic_and_main_contributions": "This paper proposes FISH-DIP, an improvement to the parameter-efficient fine-tuning that introduces Dynamic Importance of Parameters (DIP) to the Fisher information method for sparse fine-tuning. The method only uses top k% of samples that produce the highest loss (i.e. most regressing samples) to calculate an empirical approximation to the Fisher information for parameter importance, and produces a FISH mask for sparse parameter updates to the most important parameters. The approach is evaluated in a low-resources setting within a unified sequence labeling framework, taking the T5 large model as baseline, on a variety of information extraction datasets. In the low-resource setting, experimental results demonstrate FISH-DIP's competitive or superior performance to full fine-tuning baselines, fixed importance-based parameter update methods, or to in-context learning LLMs.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zpOQXSbt9S", "reviewer": "Reviewer_C1eS", "summary": "", "questions": "1. Please clear up my confusion in *Reasons To Reject 1.\n\n2. Why is FISH-DIP better than other parameter-efficient tuning methods (such as LoRA) in low-resource settings? I didn't figure out what advantages of FISH-DIP can definitely lead to better results.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1 / Q1:** We sincerely thank you for your feedback. The principal objective of our paper is to present a generalizable, robust, unified framework aimed at enhancing performance across various sequence labeling tasks in low-resource scenarios. This can be highly useful in a broad range of downstream applications from preserving indigenous languages to enhancing knowledge transfer across different domains and crisis communication. Despite its significance, this task remains largely unexplored within the research community. The inherent scarcity of annotated data in this setting requires a versatile approach to low-resource sequence labeling, which can also be extremely useful for a wide range of downstream applications.\n\nGiven the highly varied natures of different sequence labeling tasks, crafting a universally adaptable solution presents inherent challenges. Our introduced methodology, FISH-DIP, is tailored to address this very issue. By being ingrained at the PLM fine-tuning level, FISH-DIP demonstrates commendable adaptability, seamlessly integrating with simple sequence-to-sequence structured prediction methods, and without needing any supplementary data or structural modifications. We posit this flexibility as a primary feature of our proposition. Its adaptability not only enhances performance in low-resource scenarios but also does so across a wide range of structured prediction tasks that are very different from each other.\n\n**W2 / Q2:** Thanks for the insights regarding the motivation and applicable baselines. Our replies are given sequentially for each of them:\n\n(1) **Difference between FISH-DIP and weight amplification:** Emphasizing challenging samples during training is certainly a motivation for our approach. While enlarging the weight of more difficult samples is a possible approach to attain that effect, it risks distorting the data distribution and promoting overfitting to the limited available samples. Conversely, we do not alter any weight to the training losses to achieve maximum learning from the given samples. Instead, we only allow the training of the parameters pertinent to the difficult samples while keeping the rest of the parameters frozen. In this way, we improve low-resource performance while maximizing learning from the small number of samples.\n\n(2) **Motivation for Sequence Labeling and Related Methods:** As highlighted in W1/Q1, sequence labeling encompasses a diverse array of problems that hold substantial implications for various downstream applications, especially in resource-constrained contexts—a facet that remains inadequately explored within the research landscape. Consequently, we apply and evaluate our method in sequence labeling tasks that are quite varied from each other, to show the effectiveness of FISH-DIP in different scenarios.\n\nTurning to baseline methodologies, UIE [1] emerges as particularly relevant within the realm of sequence labeling. Methodologically, it expresses sequence labeling in a structured format and pretrains a T5 model using large-scale additional resources, resulting in a modified pretrained model called UIE. This model can then be fine-tuned for different Information extraction tasks.\n\nIn our paper, across all baselines, we have kept the pretrained language model fixed to T5-large (please refer to R1 W1), to focus on the sequence labeling methodology itself rather than the pretrained language model. Furthermore, pretraining a language model with large-scale additional resources might not be possible in lower-resource languages, which significantly limits its usefulness and defeats the purpose of generalization (multilingual models such as mT5 can be a good PLM in these scenarios to use FISH-DIP).\n\nNevertheless, we summarize the performance of FISH-DIP and UIE in the CoNLL’03 NER task in the table below, the only task-dataset that is common to both the papers. We also add the performance with T5-base applied with FISH-DIP for better comparison. The table demonstrates that without any large-scale pretraining structured schema of UIE may offer suboptimal performance. Still, FISH-DIP offers commendable performance regardless of the PLMs used, proving its robustness in sequence labeling. We will include this comparison in the final version of our manuscript as well.\n\n| Method                | Large Scale Pretraining | F1-Score @1% Data | F1-Score @5% Data | F1-Score @10% Data |\n|-----------------------|-------------------------|-------------------|------------------|-------------------|\n| UIE (T5-v1.1-base)    | No                      | 75.74             | 85.71            | 87.70             |\n| UIE (base)    | Yes                     | 82.84             | 88.34            | 89.63             |\n| FISH-DIP (T5-base)    | No                      | 82.38             | 91.24            | 91.83             |\n| FISH-DIP (T5-large)   | No                      | 85.83             | 91.50            | 91.70             |\n\n(3) **Comparison with other PEFT approaches in low-resource setttings** We acknowledge that it would have been ideal to compare FISH-DIP against all PEFT approaches in all settings. However, given the large number of experiments (including each experiment done at least three times to calculate variance), it is quite difficult to cover all combinations in all settings. For rebuttal purposes, we are including some additional experiment results done on ACE’05 Joint ER task in the 1% data setting for popular PEFT approaches - Adapter [3], LoRA [4], and (IA)^3 [5]. We will add more comparisons in our final version:\n\n| Method   | Entity F1 | Relation F1 |\n|----------|-----------|-------------|\n| Adapter  | 72.96     | 14.08       |\n| LoRA     | 71.87     | 13.25       |\n| (IA)^3   | 63.54     | 6.7         |\n| FISH-DIP | 75.11     | 19.26       |\n\nAs for the reason for the effectiveness of FISH-DIP compared to other PEFT approaches, learning a unified sequence labeling approach requires adapting base PLMs to an augmented natural language structure. Contemporary PEFT techniques introduce new trainable parameters during fine-tuning. However, in the realm of generative sequence labeling, it is very difficult to accommodate the augmented linguistic structure by training only these newly added parameters due to the scarcity of training data, as empirically demonstrated in Figure 2. To address this, we propose dynamic parameter selection for sparse finetuning, building on the FISHer information mask [2], enabling our method to maintain effectiveness under limited data availability.\n\nFinally, in Figure 2 of the original manuscript, we present the performance of the CoNLL'04 Joint Entity Relation Extraction task. Within each data setting (1%/5%/10%), we delve into both entity and relation extraction performance. A careful observation of this figure reveals a consistent trend: FISH-DIP consistently outperforms all other PEFT approaches. Particularly noteworthy is its performance in the most challenging scenario, where only 1% of the data is available. In this case, FISH-DIP maintains a high level of performance, whereas every other method experiences a significant performance decline in both entity and relation extraction tasks.\n\nThis observation serves as a compelling illustration of how FISH-DIP is better suited for low-resource sequence labeling tasks, as depicted in Figure 2 of the paper.\n\n\n[1] Lu, Yaojie, et al. \"Unified structure generation for universal information extraction.\" arXiv preprint arXiv:2203.12277 (2022).\n\n[2] Sung, Yi-Lin, Varun Nair, and Colin A. Raffel. \"Training neural networks with fixed sparse masks.\" Advances in Neural Information Processing Systems 34 (2021): 24193-24205.\n\n[3] Houlsby, Neil, et al. \"Parameter-efficient transfer learning for NLP.\" International Conference on Machine Learning. PMLR, 2019.\n\n[4] Hu, Edward J., et al. \"LoRA: Low-rank adaptation of large language models.\" arXiv preprint arXiv:2106.09685 (2021).\n\n[5] Liu, Haokun, et al. \"Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.\" Advances in Neural Information Processing Systems 35 (2022): 1950-1965.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of sample-aware parameter selection is interesting. And the experiments show it is also useful.\n\n2. The authors have done extensive experiments on 7 datasets. The results seem to be solid.", "reasons_to_reject": "1. The motivation of this paper is quite confusing and inconsistent. In fact, I didn't figure out what's the relationship between \"Sequence-labeling tasks\", \"low-resource settings\" and the FISH-DIP method. In my opinion, the method can be used in any other tasks that require fine-tuning from a PLM. Also, there's no point to show that FISH-DIP is definitely better for low-resource settings. At least the introduction part didn't persuade me. \n\n2. I think there should include more baselines: (1) The idea of the sample-based parameter selection of FISH-DIP is similar to re-weighting the training loss (enlarging the weight of the loss of the more difficult samples). Would this be a reason that FISH-DIP is better than other PEFT methods? I think this needs further discussion. (2) As said in 1, I still wonder why the authors only chose the sequence-labeling tasks since I don't think the claimed problem is unique in sequence labeling. But if the sequence labeling task is chosen, then more baselines such as UIE [1] (that also solve the problem in low-resource settings) need to be compared. (3) The FISH-DIP is still a PEFT method, so why  not compare the PEFT methods on all datasets? I still can't figure out why FISH-DIP is definitely better than other PEFT method in low-resource settings, and the results in Figure 2 is not so promising.\n\n[1] Lu, Yaojie, et al. \"Unified structure generation for universal information extraction.\" arXiv preprint arXiv:2203.12277 (2022).", "questions_to_authors": "1. Please clear up my confusion in *Reasons To Reject 1.\n\n2. Why is FISH-DIP better than other parameter-efficient tuning methods (such as LoRA) in low-resource settings? I didn't figure out what advantages of FISH-DIP can definitely lead to better results.", "paper_topic_and_main_contributions": "This paper proposes a FISH-DIP method for better fine-tuning of PLMs on sequence labeling tasks. The FISH-DIP strategy is a kind of parameter-efficient finetuning method and is based on Fisher Information sparsity training while realizing dynamic sparse training with parameter selection during training. The authors have done many experiments to show the effectiveness of FISH-DIP on seven datasets. They also have done some analytical experiments to further illustrate the method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "d8VUsUNyYy", "reviewer": "Reviewer_XQLo", "summary": "", "questions": "Please consider the **Reasons To Reject** questions.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:** We thank you for bringing up the matter of the \"Low Resource\" terminology. Our primary objective within this paper revolves around enhancing sequence labeling performance within the constraints of **limited data scenarios** (similar to UIE [1]), all the while adhering to a unified and versatile approach to sequence labeling tasks. While we indeed draw motivation from a related PEFT approach, namely the FISHer information mask [2], this choice serves to bolster the overarching generalizability of our methodology across an extensive array of sequence labeling tasks. Notably, our proposed method does not need any supplementary pretraining with external resources, thereby rendering it adaptable to languages with limited resources. This characteristic significantly amplifies the \"generalizability\" aspect within the scope of low-resource sequence labeling.\n\nIn summary, our methodological selection is driven by a commitment to a universally applicable strategy for enhancing sequence labeling performance. Regarding baselines, we believe we have compared against the most popular state-of-the-art techniques commonly used on each dataset, demonstrating consistent performance improvement across the board.\n\n**W2:** Thank you for your comment. Due to our adherence to a unified approach to sequence labeling, it is imperative to adapt base PLMs to an augmented natural language format. Most contemporary PEFT approaches ((IA)^3, Adapter, Compacter, etc.) introduce a small amount of trainable new parameters during finetuning while keeping the rest of the network fixed during the finetuning process. However, in the realm of generative sequence labeling, it is very difficult to accommodate the augmented linguistic structure by training only these newly added parameters due to the scarcity of training data, as empirically demonstrated in Figure 2 and the newly added results of ACE'05 Joint ER task in 1% data in R2 - W2/Q2 (added here for ease of referencing)\n\n| Method   | Entity F1 | Relation F1 |\n|----------|-----------|-------------|\n| Adapter  | 72.96     | 14.08       |\n| LoRA     | 71.87     | 13.25       |\n| (IA)^3   | 63.54     | 6.7         |\n| FISH-DIP | 75.11     | 19.26       |\n\nTo address this inherent challenge, we propose sparsity dynamism suited for few-sample sequence labeling scenarios based on the foundation of the FISHer information mask [2], an approach that selects parameters for sparse fine-tuning. This strategy ensures that our method retains its effectiveness even within the constraints of extremely limited training data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Novelty: Although the concrete techniques deployed in the framework (the unified sequence labeling augmentation format, the sparsity mask based on Fisher Information) are directly borrowed from previous work, their application to the sequence labeling setup seems new to me, which can be considered as the novelty of the work.\n* Clarity: Most parts of the paper are clearly presented.\n* Effectiveness: The FISH-DIP method is shown to outperform previous PEFT methods on a series of sequence labeling tasks under the few-shot setting. The empirical results seem comprehensive to me.", "reasons_to_reject": "* Lack of a clear definition of \"low-resource\": The term \"low-resource\" is a broad concept and could mean either \"few training data\" or \"few computational budget for updating model parameters\". The methodological focus of this paper seems to be on the latter (i.e., selecting a subset of model parameters to update), while the former is more reflected in the experiment setting and seems to be a side-effect of the method. It would be better if the authors explicitly explain what they intend to mean by \"low-resource\" -- that would make the goal and focus of the paper more clear. Without the clarification, one would assume a lot more few-shot learning methods could be relevant baselines (e.g., self-training, data augmentation) that are currently missing from comparisons.\n* Not sufficiently strong motivation: While the authors claim that previous PEFT methods do not perform satisfactorily on low-resource sequence labeling problems, there does not seem to be a very convincing argument as to why that could be the challenge, and why one should consider the sparsity parameter update methods in this paper. I'd like to see more conceptual/in-depth analyses/discussions in this regard (i.e., what makes FISH-DIP fundamentally different from other PEFT methods and how does it address their limitations?).", "questions_to_authors": "Please consider the **Reasons To Reject** questions.", "paper_topic_and_main_contributions": "The paper studies a unified sequence labeling problem under the few-shot setting. The authors claim that most previous Parameter Efficient Fine-Tuning (PEFT) methods do not work very well in such settings, and they propose a new method FISH-DIP, which aims at selecting a small fraction of parameters in the pretrained language model (PLM) to update on the few-shot training sets, achieved via a sample-aware dynamic sparsity selection metric (Fisher information) with the help of feedback from highly regressing examples. \n\nThe authors conduct experiments on a series of sequence labeling tasks such as NER, RE, and SRL. The results demonstrate that the proposed method is able to outperform full fine-tuning and existing PEFT methods.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 045 & Line 449 (both right column): lanugage -> language"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes FISH-DIP to use fisher information to highlight those task-specific parameters and then tune these highlighted parameters. In fact, using fisher information to highlight important parameters is widely used in continual learning, which is not a novel method. Considering that the paper is well organized and clearly presented, it is appropriate for this paper to be accepted to the main conference or findings
Decision: Accept-Main