Towards Cost Efficient Use of
Pre-trained Models
Alan Ritter

Joint Work With:

Fan BaiJunmo Kang

Dayne
Frietag

Peter Madrid

GabrielStanovsky

Wei XuNLP is Getting Expensive!
‣ Reserved AWS p4d.24xlarge costs $143,544/year! 💰NLP is Getting Expensive!
‣ Reserved AWS p4d.24xlarge costs $143,544/year! 💰‣ Company: hire an employee?
Or, purchase a GPU server?

NLP is Getting Expensive!
‣ Reserved AWS p4d.24xlarge costs $143,544/year! 💰• Negative Externalities:
4993.2 kg Co2 ~1 car driving for a year (13K mi)
Source: Machine Learning Emissions Calculator‣ Company: hire an employee?
Or, purchase a GPU server?

Energy and Policy Considerations for Deep Learning in NLP, Strubell et. al. 2019

NLP is Getting Expensive!
InstructGPT (Ouyang et. al. 2022)

NLP is Getting Expensive!
InstructGPT (Ouyang et. al. 2022)

NLP is Getting Expensive!
InstructGPT (Ouyang et. al. 2022)

GPT-3 was Estimated to cost $4 Million to
Pre-Train

💰NLP is Getting Expensive!
InstructGPT (Ouyang et. al. 2022)‣Computation vs. (Human) Annotation

GPT-3 was Estimated to cost $4 Million to
Pre-Train💰Tradeoff #1: Domain Adaptive Pre-Training (Gururangan et. al. 2020)‣Computation vs. (Human) Annotation
EMNLP 2021

‣Computation vs. (Human) Annotation
Tradeoff #1: Domain Adaptive Pre-Training (Gururangan et. al. 2020)

EMNLP 2021

‣Computation vs. (Human) Annotation
Tradeoff #1: Domain Adaptive Pre-Training (Gururangan et. al. 2020)

EMNLP 2021

Tradeoff #2: Knowledge Distillation (Sanh et. al. 2019, Zhou et. al. 2022)

ACL 2023

Computation vs. Annotation
Tradeoff #1: Pre-Train or Annotate

Domain #1
Existing NLP
Model/Dataset

Domain #2

Computation vs. Annotation
Tradeoff #1: Pre-Train or Annotate
Domain Adaptive Pre-Training(Han and Eisenstein, 2019)(Gururangan et. al. 2020)

Domain #1
Existing NLP
Model/Dataset

Domain #2

Hand-label data

Pre-train in-domain

Computation vs. Annotation
Tradeoff #1: Pre-Train or Annotate
Domain Adaptive Pre-Training(Han and Eisenstein, 2019)(Gururangan et. al. 2020)

Domain #1
Existing NLP
Model/Dataset

Domain #2

Hand-label data

Pre-train in-domain

Conventional Wisdom: data labeling is expensive.
Q: Given current GPU/TPU costs, when is in-domain pretraining economical?

💰Key Insight: view domain
adaptation through the lens ofconsumer theory

Optimal Choice
Number of
Annotated
Documents(In-domain)

F1 = 0.9
F1 = 0.8
Budget constraint

F1 = 0.7

Xa C a + Xp C p  B<latexit sha1_base64="LJl8z4Ok0O2l3nbJ/YtpoiGdinY=">AAACAXicbZDNSgMxFIUz9a/Wv1E3gptgEQShzBRRl6XduKxg24F2GDJppg3NZGKSEcpQN76KGxeKuPUt3Pk2pu0stPXAhY9z7yW5JxSMKu0431ZhZXVtfaO4Wdra3tnds/cP2ipJJSYtnLBEeiFShFFOWppqRjwhCYpDRjrhqDHtdx6IVDThd3osiB+jAacRxUgbK7CPvADBhqlz6AXCkIA9Ru5hPbDLTsWZCS6Dm0MZ5GoG9levn+A0JlxjhpTquo7QfoakppiRSamXKiIQHqEB6RrkKCbKz2YXTOCpcfowSqQpruHM/b2RoVipcRyayRjpoVrsTc3/et1UR9d+RrlINeF4/lCUMqgTOI0D9qkkWLOxAYQlNX+FeIgkwtqEVjIhuIsnL0O7WnEvK9Xbi3KtnsdRBMfgBJwBF1yBGrgBTdACGDyCZ/AK3qwn68V6tz7mowUr3zkEf2R9/gAacpTB</latexit>

Number of Pretraining Hours (TPUs)

Q: Given current GPU/TPU costs, when is in-domain pretraining economical?

Q: Given current GPU/TPU costs, when is in-domain pretraining economical?

Key Insight: view domain
adaptation through the lens ofconsumer theory

Optimal Choice
Number of
Annotated
Documents(In-domain)

F1 = 0.9
F1 = 0.8
Budget constraint

Xa C a + Xp C p  B

F1 = 0.7<latexit sha1_base64="LJl8z4Ok0O2l3nbJ/YtpoiGdinY=">AAACAXicbZDNSgMxFIUz9a/Wv1E3gptgEQShzBRRl6XduKxg24F2GDJppg3NZGKSEcpQN76KGxeKuPUt3Pk2pu0stPXAhY9z7yW5JxSMKu0431ZhZXVtfaO4Wdra3tnds/cP2ipJJSYtnLBEeiFShFFOWppqRjwhCYpDRjrhqDHtdx6IVDThd3osiB+jAacRxUgbK7CPvADBhqlz6AXCkIA9Ru5hPbDLTsWZCS6Dm0MZ5GoG9levn+A0JlxjhpTquo7QfoakppiRSamXKiIQHqEB6RrkKCbKz2YXTOCpcfowSqQpruHM/b2RoVipcRyayRjpoVrsTc3/et1UR9d+RrlINeF4/lCUMqgTOI0D9qkkWLOxAYQlNX+FeIgkwtqEVjIhuIsnL0O7WnEvK9Xbi3KtnsdRBMfgBJwBF1yBGrgBTdACGDyCZ/AK3qwn68V6tz7mowUr3zkEf2R9/gAacpTB</latexit>

Number of Pretraining Hours (TPUs)

Q: Given current GPU/TPU costs, when is in-domain pretraining economical?

Key Insight: view domain
adaptation through the lens ofconsumer theory★

★

Xa annotated documents at a
cost of Ca each
Xp hours of pre-training withcost Cp

Optimal Choice
Number of
Annotated
Documents(In-domain)

F1 = 0.9
F1 = 0.8
Budget constraint

F1 = 0.7

Xa C a + Xp C p  B<latexit sha1_base64="LJl8z4Ok0O2l3nbJ/YtpoiGdinY=">AAACAXicbZDNSgMxFIUz9a/Wv1E3gptgEQShzBRRl6XduKxg24F2GDJppg3NZGKSEcpQN76KGxeKuPUt3Pk2pu0stPXAhY9z7yW5JxSMKu0431ZhZXVtfaO4Wdra3tnds/cP2ipJJSYtnLBEeiFShFFOWppqRjwhCYpDRjrhqDHtdx6IVDThd3osiB+jAacRxUgbK7CPvADBhqlz6AXCkIA9Ru5hPbDLTsWZCS6Dm0MZ5GoG9levn+A0JlxjhpTquo7QfoakppiRSamXKiIQHqEB6RrkKCbKz2YXTOCpcfowSqQpruHM/b2RoVipcRyayRjpoVrsTc3/et1UR9d+RrlINeF4/lCUMqgTOI0D9qkkWLOxAYQlNX+FeIgkwtqEVjIhuIsnL0O7WnEvK9Xbi3KtnsdRBMfgBJwBF1yBGrgBTdACGDyCZ/AK3qwn68V6tz7mowUr3zkEf2R9/gAacpTB</latexit>

Number of Pretraining Hours (TPUs)

Maximize utility, U(Xa, Xp), within the budget constraint XaCa + XpCp ≤ B.

Remaining Questions
1. Annotation cost Ca
2. Pre-training cost Cp

Optimal Choice

Number of
Annotated
Documents(In-domain)

F1 = 0.9

3. Experimental details

F1 = 0.8
Budget constraint

Xa C a + Xp C p  B

F1 = 0.7

4. Recommendation: Pre-train orNumber
Annotate?
of Pretraining Hours (TPUs)

Remaining Questions
1. Annotation cost Ca
2. Pre-training cost Cp

Optimal Choice
Number of
Annotated
Documents(In-domain)

F1 = 0.9
F1 = 0.8
Budget constraint

3. Experimental details

Xa C a + Xp C p  B

F1 = 0.7

Number of Pretraining Hours (TPUs)

4. Recommendation: Pre-train or Annotate?

Estimating Annotation Cost, Ca
Wet Lab Protocols
Tabassum et al.

Domain#Procedures#Sent.

Total Cost

Price/Sent.

protocols

726

17,658$7,820

$0.44

191

1,699

$1,730

$1.02

992

8,331

$5,000$0.60

PubMed Materials &journal articles
Methods(this work)

Synthetic
Procedureschemistry(this work)*Annotators were paid 13 USD/hour.

Estimating Annotation Cost, Ca
Wet Lab Protocols
Tabassum et al.

Domain#Procedures#Sent.

Total Cost

Price/Sent.

protocols

726

17,658$7,820

$0.44

191

1,699

$1,730

$1.02

992

8,331

$5,000$0.60

PubMed Materials &journal articles
Methods(this work)

Synthetic
Procedureschemistry(this work)*Annotators were paid 13 USD/hour.

Annotation cost Ca (Price / Sent)

Estimating Annotation Cost, Ca
Motivation: Automating Chemistry
Wet Lab Protocols
Tabassum et al.

Domain#Procedures#Sent.

Total Cost

Price/Sent.

protocols

726

17,658$7,820

$0.44

191

1,699

$1,730

$1.02

992

8,331

$5,000$0.60

PubMed Materials &journal articles
Methods(this work)

Synthetic
Procedureschemistry(this work)

Fully automated chemical synthesis: toward the universal synthesizer
Nathan Collins, David Stout, Jin-Ping Lim, Jeremiah P Malerich, Jason D White, Peter B Madrid, Mario Latendresse, David Krieger, Judy
Szeto, Vi-Anh Vu, Kristina Rucker, Michael Deleo, Yonael Gorfu, Markus Krummenacker, Leslie A Hokama, Peter Karp, Sahana Mallya
Organic Process Research & Development*Annotators were paid 13 USD/hour.

Annotation cost Ca (Price / Sent)

Measuring Utility, U(Xa, Xp)
pre-training

$800

Cp

Optimal Choice

Proc-RoBERTa

Number of
Annotated
Documents(In-domain)

F1 = 0.9
F1 = 0.8
Budget constraint

Xa C a + Xp C p  B

F1 = 0.7<latexit sha1_base64="LJl8z4Ok0O2l3nbJ/YtpoiGdinY=">AAACAXicbZDNSgMxFIUz9a/Wv1E3gptgEQShzBRRl6XduKxg24F2GDJppg3NZGKSEcpQN76KGxeKuPUt3Pk2pu0stPXAhY9z7yW5JxSMKu0431ZhZXVtfaO4Wdra3tnds/cP2ipJJSYtnLBEeiFShFFOWppqRjwhCYpDRjrhqDHtdx6IVDThd3osiB+jAacRxUgbK7CPvADBhqlz6AXCkIA9Ru5hPbDLTsWZCS6Dm0MZ5GoG9levn+A0JlxjhpTquo7QfoakppiRSamXKiIQHqEB6RrkKCbKz2YXTOCpcfowSqQpruHM/b2RoVipcRyayRjpoVrsTc3/et1UR9d+RrlINeF4/lCUMqgTOI0D9qkkWLOxAYQlNX+FeIgkwtqEVjIhuIsnL0O7WnEvK9Xbi3KtnsdRBMfgBJwBF1yBGrgBTdACGDyCZ/AK3qwn68V6tz7mowUr3zkEf2R9/gAacpTB</latexit>

Number of Pretraining Hours (TPUs)

Xp

$1500

annotation$700

Ca ($0.60/sent.)
1155 ChemSyn sent.

Xa

Measuring Utility, U(Xa, Xp)
pre-training

$800

Cp

Optimal Choice

Proc-RoBERTa

Number of
Annotated
Documents(In-domain)

F1 = 0.9
F1 = 0.8
Budget constraint

Xa C a + Xp C p  B

F1 = 0.7<latexit sha1_base64="LJl8z4Ok0O2l3nbJ/YtpoiGdinY=">AAACAXicbZDNSgMxFIUz9a/Wv1E3gptgEQShzBRRl6XduKxg24F2GDJppg3NZGKSEcpQN76KGxeKuPUt3Pk2pu0stPXAhY9z7yW5JxSMKu0431ZhZXVtfaO4Wdra3tnds/cP2ipJJSYtnLBEeiFShFFOWppqRjwhCYpDRjrhqDHtdx6IVDThd3osiB+jAacRxUgbK7CPvADBhqlz6AXCkIA9Ru5hPbDLTsWZCS6Dm0MZ5GoG9levn+A0JlxjhpTquo7QfoakppiRSamXKiIQHqEB6RrkKCbKz2YXTOCpcfowSqQpruHM/b2RoVipcRyayRjpoVrsTc3/et1UR9d+RrlINeF4/lCUMqgTOI0D9qkkWLOxAYQlNX+FeIgkwtqEVjIhuIsnL0O7WnEvK9Xbi3KtnsdRBMfgBJwBF1yBGrgBTdACGDyCZ/AK3qwn68V6tz7mowUr3zkEf2R9/gAacpTB</latexit>

Number of Pretraining Hours (TPUs)

Xp

$1500

annotation$700

Ca ($0.60/sent.)
1155 ChemSyn sent.

Xa

Measuring Utility, U(Xa, Xp)
pre-training$1500

annotation

$700

$800

Cp

Optimal Choice

Proc-RoBERTa

Number of
Annotated
Documents(In-domain)

F1 = 0.9

Xp

Ca ($0.60/sent.)
1155 ChemSyn sent.

Xa

F1 = 0.8
Budget constraint

Xa C a + Xp C p  B

F1 = 0.7<latexit sha1_base64="LJl8z4Ok0O2l3nbJ/YtpoiGdinY=">AAACAXicbZDNSgMxFIUz9a/Wv1E3gptgEQShzBRRl6XduKxg24F2GDJppg3NZGKSEcpQN76KGxeKuPUt3Pk2pu0stPXAhY9z7yW5JxSMKu0431ZhZXVtfaO4Wdra3tnds/cP2ipJJSYtnLBEeiFShFFOWppqRjwhCYpDRjrhqDHtdx6IVDThd3osiB+jAacRxUgbK7CPvADBhqlz6AXCkIA9Ru5hPbDLTsWZCS6Dm0MZ5GoG9levn+A0JlxjhpTquo7QfoakppiRSamXKiIQHqEB6RrkKCbKz2YXTOCpcfowSqQpruHM/b2RoVipcRyayRjpoVrsTc3/et1UR9d+RrlINeF4/lCUMqgTOI0D9qkkWLOxAYQlNX+FeIgkwtqEVjIhuIsnL0O7WnEvK9Xbi3KtnsdRBMfgBJwBF1yBGrgBTdACGDyCZ/AK3qwn68V6tz7mowUr3zkEf2R9/gAacpTB</latexit>

Fine-tune

Number of Pretraining Hours (TPUs)

EasyAdapt(Daumé III, 2007)

U(Xa, Xp)

Pre-train or Annotate?

dels: BERT
Allocatefull budget
to annotation($0)
ProcBERT($620)●●: ProcBERT
Pre-train in-domain($620) model, spend remaining $ on annotation($0)

Models
●

●

●
●

●

●

95
●

●

●

●

BERT ($0)

●

●●

●

●

●●

ProcBERT($620)
●
●
●
●

●

●

●

●

●

●

●

●

●

●

●

●

●●
●
●
● ● ●
●

●

●

●

●

●

●
●

●

●

●

●

●

●

●

●

●

Models●BERT ($0)

●
●

76

●

●

●

●
●

●

●

●

90

ProcBERT ($620)

74

96.0

●

●

●
●

●

●

●

●

●

●

●

●

●
●

●

●

●

●
●

●

●

●

●
●

●
●

95.5

●

85

96.7

95.0

94.0

96.4
●

0 ●
96.1 ●

●

●
●

●●

1000

●●

●●●
● ●

●

●
93.5
● ● 900

75

●

●●●

94.5

80

●

●
●●●●●

●

●●

●●

●●

●●

1900

●

●

●●●

F1

F1

●

●

72

●

●●

75.7

70

●

●

●

74.9
●

74.1

●

●

●
●2900

2000
3000
Budget● (total $)

●

3900

72.5

0

3000

●●

●

●
●

●

●

●●

4000

●

●●●●

●●

●

●

1600

2600

3600

2000
3000
4000
Budget (total $)Synthetic Procedures

95.5

2000

1000

●●

●

●●

●

●

●

600

●

4000

●

73.3

68

●●
95.8 PubMed
Synthetic Procedures
● ●
●

1000

76.5

●

4600

5600

5000

WLP

Pre-train or Annotate?

dels: BERT
Allocatefull budget
to annotation($0)
ProcBERT($620)●●: ProcBERT
Pre-train in-domain($620) model, spend remaining $ on annotation($0)

Models
●

●

●
●

●

●

95
●

●

●

●

●●

●

●

●●

ProcBERT($620)
●
●
●
●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●
●
●
● ● ●
●

●

●

●

●

●

BERT ($0)

●
●

●

●

●

●

●

●

●

●

●

Models●BERT ($0)

●
●

76

●

●

●

●
●

●

●

●

90

ProcBERT ($620)

74

96.0

●

●

●
●

●

●

●

●

●

●

●

●

●
●

●

●

●

●
●

●

●

●

●
●

●
●

95.5

●

●

85

96.7

95.0

●●●
●

94.5

80

●

●●●
● ●

94.0

96.4

●

●
93.5
● ● 90075●

0 ●96.1 ●

●
●
●

●●

●●

1000

●●●●●

●

●●

●●

●●

●●

1900

●

●

●●●

F1

F1

●

-

1000

72

●

●●

76.5

●

75.7

70

●

●

●

74.9
●

74.1

●

●

●
●2900

2000
3000
Budget● (total $)

●

3900

72.5

0

Crossover point

3000

●●

●

●
●

●

●

●●

●●

●

1000

1600

2600

3600

2000
3000
4000
Budget (total $)Synthetic Procedures

4000

●

●●●●

●

Large Budget => Annotation + Pre-training

2000

●●

●

●●

●

●

●

600

●

4000

●

73.3

68

●●
95.8 PubMedRecommendation:
Synthetic Procedures● ●● - Small Budget => Annotation

95.5

●

4600

5600

5000

WLP

Pre-train or Annotate?

dels: BERT
Allocatefull budget
to annotation($0)
ProcBERT($620)●●: ProcBERT
Pre-train in-domain($620) model, spend remaining $ on annotation($0)

Models
●

●

●
●

●

●

95
●

●

●

●

●●

●

●

●●

ProcBERT($620)
●
●
●
●

●

●

●

●

●

●

●

●

●

●

●●
●
●
● ● ●
●

●

●

●

●

●

BERT ($0)

●
●

●

●

●

●

●

●

●

●

●

●

●

●

Models●BERT ($0)

●
●

76

●

●

●

●
●

●

●

●

90

ProcBERT ($620)

74

96.0

●

●

●
●

●

●

●

●

●

●

●

●

●
●

●

●

●

●
●

●

●

●

●
●

●
●

95.5

●

●

85

96.7

95.0

●●●
●

94.5

80

●

●●●
● ●

94.0

96.4

●

●
93.5
● ● 90075●

0 ●96.1 ●

●
●
●

●●

●●

1000

●●●●●

●

●●

●●

●●

●●

1900

●

●

●●●

F1

F1

●

-

1000

-

72

●

●●

76.5

●

75.7

70

●

●

●

74.9
●

74.1

●

●

●
●2900

2000
3000
Budget● (total $)

●

3900

72.5

0

Crossover point

1000

●●

●●

●

●●
●

●
●

●

●

●

●
●●

●

●●●●

●●

●

●
●

600

●

4000

●

73.3

68

●●
95.8 PubMedRecommendation:
Synthetic Procedures● ●● - Small Budget => Annotation

95.5

●

1600

2600

3600

2000
3000
4000
Budget (total $)Synthetic Procedures

Large Budget => Annotation + Pre-training
Annotation
if you must choose
2000 is more
3000important,
4000

4600

5600

5000

WLP

Results are Similar Across Multiple Domains
Annotate

Pre-train + Annotate

76

75.78

75
F1

PubMedM&M
Wet Lab Protocols

75.25

74
73

74.26
73.46

73.18

72

72.91

700

1500

2300

Budget ($)
96
94.5
F1

WLP
Synthetic
Procedures

93

94.88

94.44

95.52

95.09

93.59

91.5
90

90.83

700

1500
Budget ($)

2300

Results are Similar Across Multiple Domains
Annotate

Pre-train + Annotate

76

75.78

75
F1

PubMedM&M
Wet Lab Protocols

75.25

74
73

74.26
73.46

73.18

72

72.91

700

1500

Small Budget => Annotation

2300

Large
Budget($) Budget => Annotation+ Pre-training

96

94.5
F1

WLP
Synthetic
Procedures

93

94.88

94.44

95.52

95.09

93.59

91.5
90

90.83

700

1500
Budget ($)

2300

Demo: Synthetic Protocol SearchDemo: Synthetic Protocol SearchDemo: Synthetic Protocol SearchDemo: Synthetic Protocol Search

EMNLP 2022 Demo TrackExample (SynKB)“What are the solvents used for reactions containing the reagent
triphosgene?”

EMNLP 2022 Demo TrackExample (SynKB)“What are the solvents used for reactions containing the reagent
triphosgene?”

EMNLP 2022 Demo TrackExample (SynKB)“What are the solvents used for reactions containing the reagent
triphosgene?”

EMNLP 2022 Demo Track

Computation vs. Annotation
Tradeoff #2: Distill or Annotate?

ffi

Distill or Annotate? Cost-E cient Fine-Tuning of Compact Models, Junmo Kang, Wei Xu, Alan Ritter, To Appear in ACL 2023

Computation vs. Annotation
Tradeoff #2: Distill or Annotate?

Goal: Fine-tune a
Compact Model(T5-Small)ffi

Distill or Annotate? Cost-E cient Fine-Tuning of Compact Models, Junmo Kang, Wei Xu, Alan Ritter, To Appear in ACL 2023

Computation vs. Annotation
Tradeoff #2: Distill or Annotate?
Option 1: Annotate, then ne-tune T5-Small

Goal: Fine-tune a
Compact Model(T5-Small)

fi

ffi

Distill or Annotate? Cost-E cient Fine-Tuning of Compact Models, Junmo Kang, Wei Xu, Alan Ritter, To Appear in ACL 2023

Computation vs. Annotation
Tradeoff #2: Distill or Annotate?
Option 1: Annotate, then ne-tune T5-Small

Goal: Fine-tune a
Compact Model(T5-Small)

Option 2: Fine tune T5-XXL (11B), then distill

fi

ffi

Distill or Annotate? Cost-E cient Fine-Tuning of Compact Models, Junmo Kang, Wei Xu, Alan Ritter, To Appear in ACL 2023

Pareto Curves

ffi

Distill or Annotate? Cost-E cient Fine-Tuning of Compact Models, Junmo Kang, Wei Xu, Alan Ritter, To Appear in ACL 2023

Pareto Curves

No Distillation

ffi

Distill or Annotate? Cost-E cient Fine-Tuning of Compact Models, Junmo Kang, Wei Xu, Alan Ritter, To Appear in ACL 2023

Pareto Curves

Distill T5-11B on 10K Unlabeled Examples
No Distillation

ffi

Distill or Annotate? Cost-E cient Fine-Tuning of Compact Models, Junmo Kang, Wei Xu, Alan Ritter, To Appear in ACL 2023

Pareto Curves
Distill T5-11B on 100K Unlabeled Examples

Distill T5-11B on 10K Unlabeled Examples
No Distillation

ffi

Distill or Annotate? Cost-E cient Fine-Tuning of Compact Models, Junmo Kang, Wei Xu, Alan Ritter, To Appear in ACL 2023

Pareto Curves

ffi

Distill or Annotate? Cost-E cient Fine-Tuning of Compact Models, Junmo Kang, Wei Xu, Alan Ritter, To Appear in ACL 2023

Distill or Annotate?
Small Initial Training Dataset

Distillation is usuallymore economical
than annotation.

Large Initial Training Dataset

Distill or Annotate? Cost-E cient Fine-Tuning of Compact Models, Junmo Kang, Wei Xu, Alan Ritter, To Appear in ACL 2023
ffi‣

GPT-3.5 as an Annotator?
Human Annotator

Distill or Annotate? Cost-E cient Fine-Tuning of Compact Models, Junmo Kang, Wei Xu, Alan Ritter, To Appear in ACL 2023

ffi

Related Work: Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce labeling cost? GPT-3 can help. EMNLP Findings 2021

GPT-3.5 as an Annotator?
GPT-3.5 Annotator
Human Annotator

Distill or Annotate? Cost-E cient Fine-Tuning of Compact Models, Junmo Kang, Wei Xu, Alan Ritter, To Appear in ACL 2023

ffi

Related Work: Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce labeling cost? GPT-3 can help. EMNLP Findings 2021

GPT-3.5 as an Annotator?
T5-XXL Distillation
GPT-3.5 Annotator
Human Annotator

Distill or Annotate? Cost-E cient Fine-Tuning of Compact Models, Junmo Kang, Wei Xu, Alan Ritter, To Appear in ACL 2023

ffi

Related Work: Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce labeling cost? GPT-3 can help. EMNLP Findings 2021

GPT-3.5 as an Annotator?
T5-XXL Distillation
GPT-3.5 Annotator
Human Annotator‣

Distilling ne-tuned T5-XXL is
more economicalDistill or Annotate? Cost-E cient Fine-Tuning of Compact Models, Junmo Kang, Wei Xu, Alan Ritter, To Appear in ACL 2023

ffi

fi

Related Work: Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce labeling cost? GPT-3 can help. EMNLP Findings 2021

Economical Use of
Pre-trained Models‣

So far: Computation vs. Annotation‣

Adapting to a new domain‣

Knowledge Distillation
Annotation

Computation

Economical Use of
Pre-trained Models‣

So far: Computation vs. Annotation‣

Adapting to a new domain‣

Knowledge Distillation
Annotation

Next: Cost-Ef cient Data Extraction from Tablesfi‣Computation

Extracting Data from Tables‣

Extracting Leaderboards from ML Papers, e.g. AxCell (Kadres et. al. 2020)‣

Materials Science, e.g. DiSCoMaT (Gupta et. al. 2023)‣

Chemistry Tables, e.g. ChemDataExtractor (Swain et. Al. 2016)‣

Webpages e.g. OpenCeres (Lockard et. al. 2019, 2020)

Sample ML Table(Viswanathan et. al. 2023)

Extracting Data from Tables‣

Extracting Leaderboards from ML Papers, e.g. AxCell (Kadres et. al. 2020)‣

Materials Science, e.g. DiSCoMaT (Gupta et. al. 2023)‣

Chemistry Tables, e.g. ChemDataExtractor (Swain et. Al. 2016)‣

Webpages e.g. OpenCeres (Lockard et. al. 2019, 2020)

AxCell (ML Leaderboards)
DiSCoMaT (Materials Science)Extracting Data from Tables‣

Extracting Leaderboards from ML Papers, e.g. AxCell (Kadres et. al. 2020)‣

Materials Science, e.g. DiSCoMaT (Gupta et. al. 2023)‣

Chemistry Tables, e.g. ChemDataExtractor (Swain et. Al. 2016)‣

Webpages e.g. OpenCeres (Lockard et. al. 2019, 2020)

AxCell (ML Leaderboards)
DiSCoMaT (Materials Science)

Q: Can Large Language Models Reduce the Cost ofExtracting Data from Tables?

Schema-Driven Information
Extraction‣

Only supervision is an Extraction Schema authored by domain expert.

Schema-Driven Information
Extraction‣

Only supervision is an Extraction Schema authored by domain expert.

Schema-Driven Information
Extraction‣

Only supervision is an Extraction Schema authored by domain expert.

Schema-Driven Information
Extraction‣

Only supervision is an Extraction Schema authored by domain expert.

Benchmarking Schema-Driven IE(Gupta et. al. 2023)(Hao et. al. 2011)

Benchmarking Schema-Driven IE
Machine Learning(LaTeX)(Gupta et. al. 2023)(Hao et. al. 2011)

Benchmarking Schema-Driven IE
Machine Learning(LaTeX)

Chemistry(PubMed XML)

(Gupta et. al. 2023)(Hao et. al. 2011)

Benchmarking Schema-Driven IE
Machine Learning(LaTeX)

Chemistry(PubMed XML)

Materials Science(CSV)(Gupta et. al. 2023)(Hao et. al. 2011)

Benchmarking Schema-Driven IE
Machine Learning(LaTeX)

Chemistry(PubMed XML)

Materials Science(CSV)(Gupta et. al. 2023)

Webpages(HTML)(Hao et. al. 2011)

Schema-Driven IE
Large
Language
ModelSchema-Driven IE
Large
Language
ModelSchema-Driven IE
Large
Language
ModelSchema-Driven IE
Large
Language
ModelSchema-Driven IE
Large
Language
ModelError Recovery‣

Challenge: Language models visit table cells in disorganizedmanner.

Error Recovery‣

Challenge: Language models visit table cells in disorganizedmanner.

Error Recovery‣

Challenge: Language models visit table cells in disorganizedmanner.

Results‣ Competitive with SOTA supervised models• No Labels or Custom Pipelines• Works on Diverse Data Formats and Domains• Only Human Supervision is Extraction Schema

Results
code-davinci-002‣ Competitive with SOTA supervised models• No Labels or Custom Pipelines• Works on Diverse Data Formats and Domains• Only Human Supervision is Extraction Schema

Distilling Table Extractors‣ Not dif cult to distill student table extraction models withfi

performance as good as the teacher.

Ablation StudyAblation Study‣ Schemas are crucial

Ablation Study‣ Schemas are crucial‣ Error recovery has strong performance while minimizing
API costs.

Downstream Task: ML Leaderboards(Kadras et. al. 2020)‣ Requires Additional Linking Steps

Downstream Task: ML Leaderboards(Kadras et. al. 2020)‣ Requires Additional Linking Steps‣ Performance comparable to custom
Pipeline w/ labeled examples

Takeaways(1) Schema-Driven Information
Extraction

Schema-Driven Information Extraction from Heterogeneous Tables
Fan Bai, Junmo Kang, Gabriel Stanovsky, Dayne Freitag, Alan Ritter
arXiv preprint (2023)• Key idea: Only supervision is extraction schema• Strong performance across multiple domains

Takeaways(1) Schema-Driven Information
Extraction

Schema-Driven Information Extraction from Heterogeneous Tables
Fan Bai, Junmo Kang, Gabriel Stanovsky, Dayne Freitag, Alan Ritter
arXiv preprint (2023)• Key idea: Only supervision is extraction schema• Strong performance across multiple domains
Distill or Annotate? Cost-E cient Fine-Tuning of Compact
ModelsJunmo Kang, Wei Xu and Alan Ritter
To Appear at ACL 2023

Pre-train or Annotate? Domain Adaptation with a Constrained
BudgetFan Bai, Alan Ritter and Wei Xu
Proceedings of EMNLP 2021

Optimal Choice(2) Computation
vs. Annotation

Number of
Annotated
Documents(In-domain)

F1 = 0.9
F1 = 0.8
Budget constraint

Xa C a + Xp C p  B

F1 = 0.7<latexit sha1_base64="LJl8z4Ok0O2l3nbJ/YtpoiGdinY=">AAACAXicbZDNSgMxFIUz9a/Wv1E3gptgEQShzBRRl6XduKxg24F2GDJppg3NZGKSEcpQN76KGxeKuPUt3Pk2pu0stPXAhY9z7yW5JxSMKu0431ZhZXVtfaO4Wdra3tnds/cP2ipJJSYtnLBEeiFShFFOWppqRjwhCYpDRjrhqDHtdx6IVDThd3osiB+jAacRxUgbK7CPvADBhqlz6AXCkIA9Ru5hPbDLTsWZCS6Dm0MZ5GoG9levn+A0JlxjhpTquo7QfoakppiRSamXKiIQHqEB6RrkKCbKz2YXTOCpcfowSqQpruHM/b2RoVipcRyayRjpoVrsTc3/et1UR9d+RrlINeF4/lCUMqgTOI0D9qkkWLOxAYQlNX+FeIgkwtqEVjIhuIsnL0O7WnEvK9Xbi3KtnsdRBMfgBJwBF1yBGrgBTdACGDyCZ/AK3qwn68V6tz7mowUr3zkEf2R9/gAacpTB</latexit>

ffi

Number of Pretraining Hours (TPUs)

Takeaways(1) Schema-Driven Information
Extraction

Schema-Driven Information Extraction from Heterogeneous Tables
Fan Bai, Junmo Kang, Gabriel Stanovsky, Dayne Freitag, Alan Ritter
arXiv preprint (2023)• Key idea: Only supervision is extraction schema• Strong performance across multiple domains
Distill or Annotate? Cost-E cient Fine-Tuning of Compact
ModelsJunmo Kang, Wei Xu and Alan Ritter
To Appear at ACL 2023

Pre-train or Annotate? Domain Adaptation with a Constrained
BudgetFan Bai, Alan Ritter and Wei Xu
Proceedings of EMNLP 2021

Optimal Choice(2) Computation
vs. Annotation

Number of
Annotated
Documents(In-domain)

F1 = 0.9
F1 = 0.8
Budget constraint

Thank you!

F1 = 0.7

Xa C a + Xp C p  B<latexit sha1_base64="LJl8z4Ok0O2l3nbJ/YtpoiGdinY=">AAACAXicbZDNSgMxFIUz9a/Wv1E3gptgEQShzBRRl6XduKxg24F2GDJppg3NZGKSEcpQN76KGxeKuPUt3Pk2pu0stPXAhY9z7yW5JxSMKu0431ZhZXVtfaO4Wdra3tnds/cP2ipJJSYtnLBEeiFShFFOWppqRjwhCYpDRjrhqDHtdx6IVDThd3osiB+jAacRxUgbK7CPvADBhqlz6AXCkIA9Ru5hPbDLTsWZCS6Dm0MZ5GoG9levn+A0JlxjhpTquo7QfoakppiRSamXKiIQHqEB6RrkKCbKz2YXTOCpcfowSqQpruHM/b2RoVipcRyayRjpoVrsTc3/et1UR9d+RrlINeF4/lCUMqgTOI0D9qkkWLOxAYQlNX+FeIgkwtqEVjIhuIsnL0O7WnEvK9Xbi3KtnsdRBMfgBJwBF1yBGrgBTdACGDyCZ/AK3qwn68V6tz7mowUr3zkEf2R9/gAacpTB</latexit>

Number of Pretraining Hours (TPUs)

Sponsors:

ffi

Fan BaiJunmo Kang