M2D2: A Massively Multi-Domain Language Modeling Dataset
Machel Reid1∗, Victor Zhong2 , Suchin Gururangan2 , Luke Zettlemoyer2
1
The University of Tokyo, 2 University of Washington
machelreid@google.com, {vzhong,sg01,lsz}@cs.washington.edu

Abstract
ct
pa
Im
of
y
ivit
n act
ma
hu

Natural and physical sciences

an
de
ve
nts

Hi
sto
ry

con

i

By

systems

E
Se xerc nce
i
lf
ca se
re

y

oph

By region

BeliefhNaessciencEarthture

Biology

ew
orld

Alla

H
He umaalt n me Nutrit
h s dic ion
cie ineilos

By period

nt

s
Toy

Logic

Gener
al

Th

Ma

tine

s and
me diaGa ss me

Re

th
al
he
ic
bl
ing
Pu
nk

Pe

s
ce
ien
sc

rts
la
ua arts
Visrforming

Fields of mathematics

Mathematics andlogicrefereMathematic
He ligion and
ce
s
Furth
belie
alt
Reference
er re
f sy
works
h
search
ste
ms
an
tools
and to
d
Majo
fitn
pi
cs
r be
liefs
es
of th
s

Ph

Currently at Google Research
We release our dataset publicly at https://github.
com/machelreid/m2d2.

n
atio
cre
Re
nd
a
orts
Sp

s
the art

ial
oc
ds
an
ty
cie
So

∗

1

and
CultureiencesPhysical sc

Even though they can contain a wide variety ofdifferent types of domains, the texts that make up
the corpora used to train and evaluate language
models (LMs) are often treated as if they are all
the same. This makes it challenging to characterize
LM performance under diverse data distributions
and understand how to effectively adapt LMs to
new ones. To address these challenges, we develop
M2D2, a Massively Multi-Domain Dataset, with
145 subdomains and a human-curated hierarchy forstudying fine-grained domain adaptation.

ng
Transport

Society

g
kin
hin
dt
an
hy
op
ilos
Ph

Introduction

Te
ch
no
log
ya
nd
ap
pli
ed
sc
ien
ce
s

Engin
eerin
Comp
g
uti

Human activites

1

Human activities

Agriculture

ies
Culture and Humanit

We present M2D2, a fine-grained, massively
multi-domain corpus for studying domain adaptation in language models (LMs). M2D2 consists of 8.5B tokens and spans 145 domains extracted from Wikipedia and Semantic Scholar.
Using ontologies derived from Wikipedia and
ArXiv categories, we organize the domains ineach data source into 22 groups. This twolevel hierarchy enables the study of relationships between domains and their effects on inand out-of-domain performance after adaptation. We also present a number of insights into
the nature of effective domain adaptation inLMs, as examples of the new types of studies M2D2 enables. To improve in-domain
performance, we show the benefits of adapting the LM along a domain hierarchy; adapting to smaller amounts of fine-grained domainspecific data can lead to larger in-domain performance gains than larger amounts of weakly
relevant data. We further demonstrate a tradeoff between in-domain specialization and outof-domain generalization within and across ontologies, as well as a strong correlation between
out-of-domain performance and lexical overlap
between domains.1

Figure 1: Visualization of the two-level fine-grained
domain hierarchy in the Wikipedia portion of M2D2.

Prior work on domain transfer focuses on a small
number of broad domains (typically 4-20; Gururangan et al., 2020; Gao et al., 2021; Gururangan
et al., 2021). In contrast, domains in M2D2 are
fine-grained and organized into a hierarchy derived
from human-curated ontologies in Wikipedia (Figure 1) and Semantic Scholar (Figure 2). Unlike
prior work, the fine granularity of M2D2 enables
the study of transfer to naturally occurring datascarce domains recognized by human curators (e.g.
Philosophy, Public Health, Transport). This hierarchy enables the study of domain transfer at varying levels of topic granularity. For instance, howshould we combine widely available internet text
(entire corpus), text on computer science (coarse
domain), and scarce corpus on machine learning(fine domain) to improve performance in the machine learning domain? To the best of our knowledge, M2D2 is the first dataset that combines finedomain granularity with human-curated domain
hierarchy in a massively multi-domain setting.

.IM

at
mat h.O
A

O

.C

h.G
A

2

tro

m
h.Q
a
m th.AAam th.C Tam th.R A
ath
A
m
.C
V

mat h.LOh.AC

mat
mat h.SG
h.M
G

ma
th.
SP
ma
th.

mat
CT
h.KT

math.
GN

cs.L
G

math

cs.CV

cs.IT

astro-phastro-ph

cs.AI
cs.DS

cs

cs.CR
cs.CL

Art

ph

cs.SY

cs.DM
cs.CC
cs.RO

cond-mat

cs.GT

cs.SE
cs.NE
cs.NA

cs.CY
cs.DB
cs.IR

PL
cs.

HC
cs.
CG
cs.

L
cs.F A
cs.M

E
cs.C

cs.PF
cs.MM

cs.GR

cs.ET

cs.SC

q-bio.CBq-bio.GNq-bio.MN

q-bio.BM

q-bio.QM

q-bio.NC

q-bio.PE

q-bio

ics

stat.

stat.A CO

M2D2 consists of a large quantity of fine-grain
domains. Unlike prior work that defines the domain of a corpus using its source (e.g. the web text
domain; Chronopoulou et al., 2021), we derive domains from a human-curated Wikipedia and arXiv
ontologies. In this section, we describe how M2D2
is collected and organized.

P
stat
.M

sta stat.THE
t.M
L
physi

physics.

phys cs.me

ed-phphys ics.a d-ph
ics.h cc-ph

phy physics sics. ist-p

phy phy .spa ao-p hsics sics ce-p h

ph
.geo
h
ysi .cla
ph
-ph
ph
ph
ys
ys ysi cs.ap ss-ph
ics. ics.d cs.gen p-ph
-ph
pl
at

2.1

Domain Organizationphph ph ysic asm a-an
ys ysic s.in -ph
ics
s.b s-dph phy

r

at.m

.soft
cond
-mat.supr-co
cond-mnat.dis-nncond-mat.quantgas
cond-mat.othe

cond
-mat

y
ph

Medicine

ph ysic sics.a .chem io-ph et
ys s.c tom -phoic
s.s mp -ph
oc -ph
-p
h

n
dy
uics
.ﬂ
pt
ics
ys
s.o
ph sic

trl-s
ci

c

at

ys

Medicine

Philosophy

sta
t

-m

co
nd
-m
co
at
nd
.st
-m
at
at.
con
me mec
d-m
h
s-h
at.s
con
all
t
r-e
d-m
l

d
on

cs.SD
cs.DL

Art

Philosophy

q-bio

cs.NI

cs.DC
cs.LO
cs.SI

M2D2

-p
h

as

roas
ph
t
.SR
as ro-pastro-phastr tro-ph.EP
o-ph
.HE
ast

math.COmath.MPmath.AG
R
math.P

.AP
math
.IT
ath

th
ma h.DS

m
at
h

m
G
th.D
ma .OCmGam th. A
a
.F
m ath

t
ma h.NTA
atm th.NTam th.R T

m

.G
a
m th
.S R
at
T

archy to improve domain transfer.

Figure 2: Visualization of the hierarchies contained
within the S2ORC portion of M2D2.

Using M2D2, we investigate the following questions, as examples of the broad classes of new questions that can be asked: (1) how well do coarse
and fine domains transfer to each other across thehierarchy? (2) which features and aspects of adomain are important for transfer? (3) how important is domain specificity versus breadth? We
perform preliminary experiments analyzing transfer between similar domains, disparate domains,and hierarchically related domains. Moreover, we
explore how to select source domains to improvetransfer performance.
We present baseline experiments using a
GPT2 (Radford et al., 2019) language model. We
find that (1) more specific data is often more important for performance than larger, less-specificdata, shown by our comparison of coarse-grained,
fine-grained and coarse-to-fine adaptation comparison (in which coarse-to-fine performed best) , (2)vocabulary overlap is a surprising good indicator
for transfer, and (3) data source provenance information is often a better predictor than ontology
when predicting transferability, perhaps indicating
a more multi-faceted definition of domain could be
developed in future work.
Given the importance of fine granularity domains in language modeling, we hope that M2D2
will encourage the community to further study domain transfer: how do we identify hierarchical finegrained domains in naturally occurring text, andhow do we leverage this fine-grained domain hier-

One of the unique properties of M2D2 is its hierarchical nature, enabling the study of transfer at
different levels of domain granularity. We assume aparticular corpus to have L0 , . . . , LK levels of hierarchy, where L0 refers to the lowest or most coarsegrained/broad level (i.e. the whole dataset), and LK
refers to the highest or most fine-grained/specificlevel. A given level of hierarchy Li contains Ni
i ,domains DNii
Li = [D0i , . . . , Dji , . . . , DN
]
i

(1)

Dji is composed of multiple subdomains
i+1{D0i+1 , . . . , DN}, which are represented in thei+1
next level of the hierarchy Li+1 . Similarly, we
assume that a given subdomain is contained withina larger domain.
For the rest of the paper, we use L1 and L2 torepresent the two levels of a K level hierarchy that
we consider in this paper.
2.2

Dataset Collection

We collect M2D2 from two resources, Wikipedia
and Semantic Scholar. This allows us to explore
domain adaptation in a massively multi-domain setting among domains of varying granularity, while
also allowing us to test whether our findings hold
across different data sources.
Semantic Scholar We use the S2ORC corpus(Lo et al., 2020), a large corpus of English academic papers annotated with extensive metadata.
Using this corpus, which is already categorized
into L1-domains representing broader fields of academic research (e.g. Computer Science, Physics),
we extract L2-domains by finding a given paper’s
respective arXiv2 category (e.g. “Computation andLanguage” ∈ Computer Science).
2

https://arxiv.org

L1 (Abbrv)

Size

#L2

#Tokens

Examples of L2 domains

Health and fitness (HEAL)
History and events (HIST)
Society and social sciences (SOCI)
Technology and applied sciences (TECH)
Culture and the arts (CULT)
Natural and physical sciences (NATU)
Human activites (HUMA)
Mathematics and logic (MATH)
General reference (GENE)
Religion and belief systems (RELI)
Philosophy and thinking (PHIL)

761.2MB
1.4GB
2.3GB
1.9GB
2.0GB
1.2GB
2.1GB
332.3MB385.3MB428.0MB
1.0GB

7
4
3
5
8
5
3
4
3
4
3

116M
226M
379M
297M
289M
189M
343M
52M
60M
64M
165M

Exercise, Health Science
Regions, Periods
Society, social sciences
Agriculture, Computing
Games and Toys, The arts and entertainment
Physical sciences, Earth sciences
Impact of human activity
Mathematics, Logic
Research tools and topics, Reference works
Major beliefs of the world, Belief systems
Philosophy, Thinking

Mathematics (math)
Quantitative Biology(q-bio)
Physics
Nonlinear Sciences (nlin)
Condensed Matter (cm)
Economics (econ)
Computer Science (cs)
Statistics (stat)
Astrophysics (astro-ph)Art†
Philosophy†(phil)

4.5GB
1.9GB
4.1GB
730MB
3.8GB
67MB
4.5GB
2.4GB
4.0GB
575MB
919MB

26
3
12
5
10
3
23
4
7
1
1

1.4B
336M
737M
134M
688M
11M
1.1B
450M
728M
98M
156M

Topology, Number Theory
Biomolecules, Cell Behavior
General Physics, Biological Physics
Self-Organizing Systems, Chaotic Dynamics
Materials Science, Quantum Gases
Econometrics, General Econometrics, Theory
Machine Learning, Databases, Graphics
Applications, Methodology
Earth/Planetary, Cosmology——

Average±s.d.
Total

1.9G±1.7G
41GB

6.6±6.2145

373M±347M
8.5B

—
—

Table 1: Dataset statistics for M2D2. We list L1 domains, with their corresponding sizes, number of L2 domains,
number of tokens, and examples of L2 domains. † These domains did not have any subdomains in the arXiv ontology.

Wikipedia We crawl the Wikipedia ontology,3which lists major categories contained within
Wikipedia. Within these major categories or L1domains, we then proceed to look up the category
pages within a given L1-domain, and gather respective L2-domains. This procedure yields a hierarchy of domains contained within Wikipedia. We
then download the Wikipedia data dump, which
we clean using wikiextractor4 and assign agiven page to its respective domain.
2.3

Unique Properties

M2D2 has the following major unique properties when compared to previous domain adaptation datasets. First, it is massively multi-domain:
we have 145 L2 domains grouped into 22 L1 domains, which allows us to test domain adaptation
for language modeling on a variety of axes (suchas hierarchy, subject matter, and ontology) thatwould be more difficult with more coarse-grained
datasets. Second, M2D2 is hierarchical: this allows us to also test the performance of domain
3
https://en.wikipedia.org/wiki/
Wikipedia:Contents/Categories
4
https://github.com/attardi/
wikiextractor

specificity versus domain breadth in more flexible
adaptation settings.
We describe dataset statistics in Table 1, including dataset size (measured in MB/GB), token count(measured by whitespace tokenization), and thenumber of L2 domains within each L1 domain.
M2D2 contains 8.5B tokens, with an average of
373 million tokens per L1 domain. Demonstrating the hierarchical nature of M2D2, we also list
examples of L2 domains contained within the L1
domains (e.g. Computing ∈ Technology and Applied Sciences, Topology ∈ Mathematics) which
are also graphically shown in Figures 1 and 2).
2.4

Dataset Splits

We split each domain into the respective train, validation, and test sets. To prevent data leakage between the domains when pages belong to two or
more domains, we construct validation and test setsfrom pages that are not contained within any otherdomains on the same level of hierarchy. For example, the page for “Biotechnology” overlaps indomain with both Biology ∈ Natural and Physical Sciences and Engineering ∈ Technology andApplied Sciences so this would not be included inany evaluation set due to the potential for direct

3.1

ADAPTATIONMETHODS

Language Model(GPT2)

L1
Adaptation

Technology
andApplied Sciences
ComputingL1-to-L2
Adaptation

Language Model(GPT2)

L2
Adaptation

Language Model(GPT2)

Computing

Test Data

ZERO
SHOT

Figure 3: The types of domain adaptation that we consider in this work: L1, L2, and L1-to-L2 adaptation.
Here, we use “Technology and Applied Sciences” toillustrate our L1 domain and “Computing” to illustrate
our L2 domain. Bold arrows refer to adaptation steps,
and dotted lines refer to an evaluation phase.

leakage. However, the page for “Computer” is onlyin Computing ∈ Technology and Applied Sciences
and therefore could be included in an evaluation
set. We include at least 1 million tokens in thevalidation and test sets, respectively. This enables
us to have a precise evaluation set of texts that only
belong to a single fine-grained domain.

3

Experiments

As examples of the types of new studies M2D2
enables, we explore a number of key questions
about the nature of effective domain adaptation inlanguage models. For example, how does one best
specialize a language model to a domain, given an
ontology? How well can adapted models be applied
out-of-domain, within and across ontologies? Whatfeatures of target domains are predictive of out-ofdomain transfer?
In this section, we present a set of experimentsthat begin to answer these questions. First, westudy the impact of adapting to the L1 and L2 domains of our dataset on in-domain (§3.2) and outof-domain (§3.3) language modeling performance.
Then, we perform an analysis of lexical featuresin domains that are predictive of out-of-domain
performance (§3.4).

Experimental setup

In all experiments, we use the 112M GPT2 model(Radford et al., 2019) as the baseline model. Our
implementation is based on HuggingFace Transformers (Wolf et al., 2020) and PyTorch (Paszke
et al., 2019). All adaptation techniques are performed using Adam (Kingma and Ba, 2015),
dropout value of 0.2 (Srivastava et al., 2014), using
a learning rate of 5e-5 and a batch size of 64000
tokens. We train all models for a maximum of 1
million iterations and perform early stopping overthe validation set. All experiments are run on 8
NVIDIA V100 GPUs.
When adapting our GPT2 model to domains in
M2D2, we use one of three settings:
L1 Adaptation We continue training on a given
L1 domain (e.g. Computer Science).
L2 Adaptation We continue training on a given
L2 domain (e.g. Machine Learning).
L1-to-L2 Adaptation Given a L2 domain(e.g. Machine Learning), we first perform L1 adaptation on its corresponding L1 domain (e.g. Computer Science), and then we further perform L2
adaptation. This setting similar to multi-stage adaptive pretraining approaches used for supervised
tasks (Gururangan et al., 2020).
For all techniques, we evaluate test perplexity on
L2 domains validation sets. Due to the large quantity of L2 domains, we aggregate L2 results by theircorresponding L1. For each ontology, we report
the average and standard deviation (averages.d. ) of
perplexities across L2 domains in each L1.
3.2

In-Domain Results

The first set of experiments in this study considers the impact of adapting the language model todifferent levels of the M2D2 ontologies. We only
consider in-domain perplexity, or the perplexity ofmodel on the domain it is adapted to.
Adaptation improves in-domain performancedespite pretraining. Table 2 shows test-set perplexities on L2 domains, averaged across each L1
domain, after performing each adaptation technique (see Appendix on full results). First, weobserve that all proposed adaptation techniques
improve performance over the base GPT-2 model.
This highlights the effectiveness of adaptation in
improving in-domain performance, even when considering domains that the language model has likely

Wiki

HEAL

HIST

SOCI

GPT2
L1
L2
L1-to-L2
S2ORC
GPT2
L1
L2
L1-to-L2

TECH

CULT

HUMA

MATH

GENE

23.1

27.5

24.5

18.12.5
17.52.7
16.82.7

20.90.5
17.81.9
16.72.1

19.70.8
17.50.7
15.40.5

Math

Econ

CS

26.12.8

28.22.7

26.82.99.23.48.03.2
7.53.215.92.2
13.42.1
12.52.2

15.44.0
15.16.7
14.05.9

12.51.0
12.01.3
11.51.0

RELI

PHIL

27.8

27.5

28.9

26.6

22.30.8
21.81.0
21.40.9

21.22.3
21.72.6
20.6 2.6

23.01.4
22.40.9
22.0 0.8

18.35.2
17.85.2
17.15.0

CM

PhysicsArtPhil

29.71.2

32.72.1

35.10.0

32.90.0

17.11.7
16.51.3
16.11.6

27.70.0
27.70.0
27.70.0

24.40.0
24.40.0
24.40.0

NATU

25.9

26.3

26.2

26.7

26.5

21.60.8
20.81.0
19.61.1

19.80.5
18.30.4
16.9 0.521.80.6
21.00.4
20.5 0.4

20.83.2
21.71.6
20.31.5

20.7
19.8
18.8

Stat

Q-Bio

Nlin

Astro-Ph

Avg

22.77.3

30.11.3

25.51.4

31.61.5

29.2

11.03.5
10.22.5
9.33.322.62.2
21.01.3
20.31.0

9.82.49.62.1
9.22.1

15.53.2
14.02.3
12.92.3

16.5
15.7
15.0

Avg

Table 2: In-domain test perplexities, aggregated to each L1 domain. We look at the impact of L1 vs L2 vs L1-to-L2
finetuning settings when compared to simply finetuning on L1. L2 Adaptation is usually more effective than
L1 Adaptation, emphasizing the importance of fine-grained domains, with a coarse-to-fine setup using L1-to-L2
Adaptation is most effective. This finding is statistically significant (p < 0.05; measured using the KolmogorovSmirnov test).
Wiki

HEAL

HIST

SOCI

TECH

CULT

HUMA

MATH

GENE

RELI

PHIL

NATU

Avg

L1
L2
L1-to-L2

23.63.5
26.13.8
25.53.8

23.22.0
26.13.9
25.93.8

22.42.2
25.72.7
25.22.6

22.42.3
26.13.5
26.03.3

22.32.2
27.03.7
27.03.7

22.72.0
25.63.6
25.13.6

25.13.4
28.96.9
28.57.0

24.22.3
25.12.4
24.52.4

24.72.8
26.32.9
26.22.9

23.62.7
24.12.6
23.22.6

23.33.2
26.33.7
25.23.7

23.3
26.1
25.7

S2ORC

Math

Econ

CS

CM

Physics

Art

Phil

Stat

Q-Bio

Nlin

Astro-Ph

Avg

L1
L2
L1-to-L2

32.017.2
36.021.9
36.824.8

28.810.9
33.411.1
31.912.6

23.110.1
32.118.7
31.022.0

24.914.0
32.717.3
30.218.2

22.810.6
25.412.1
24.211.4

26.83.3
26.83.3
26.83.3

25.73.9
25.73.9
25.73.9

23.411.5
32.724.7
30.423.0

23.211.3
33.219.6
32.123.4

23.812.9
34.822.4
36.530.8

26.212.8
27.211.4
27.515.1

25.5
30.9
30.3

Table 3: Out-of-domain test perplexities, aggregated to each L1 domain. We look at the impact of L1 vs L2 vsL1-to-L2 finetuning settings when compared to simply finetuning on L1. We can see that L2 Adaptation andL1-to-L2 Adaptation are generally less performant in out-of-domain settings that L1 Adapted models, given theirin-domain specification. The comparison between L1 versus L2 is statistically significant p < 0.01.

been exposed to during pretraining (as is the casewith Wikipedia; L1 adaptation results in a 5.8 decrease in perplexity). For domains which the language model is less likely to have been exposed toduring pretraining, this is more pronounced (as is
the case with S2ORC; L1 adaptation results in a
12.7 decease in perplexity).
Specificity and hierarchy is more important
than broad coverage in adaptation. Next, weobserve that in most cases, adapting to L2 domains is more beneficial to in-domain performance than adapting to L1 domains. Adaptation
to finer-grained domains better specializes a language model, even though these domains are muchsmaller than their L1 counterparts. Finally, weobserve that using L1-to-L2 adaptation further benefits in-domain performance over L2 adaptationin all cases. Our results suggest that adapting tosmaller amounts of domain-specific data leads tomore effective in-domain specialization than adapting to large quantities of data that may be more
weakly domain-relevant. Moreover, the best results

may be achieved by organizing the target domain
into subsets of broader and fine-grained data, andadapting along this hierarchy. However, this approach has increased memory and computational
requirements relative to solely relying on L1 Adaptation.
3.3

Out-of-Domain Results

We also study the effects of our adaptation techniques on out-of-domain performance, by performing zero-shot inference with adapted models on
domains (e.g. Art) other than the ones they are
adapted to (e.g. Machine Learning). We first transfer models between domains in the same ontology(e.g. Wikipedia → Wikipedia), and then across ontologies (e.g. Wikipedia → S2ORC).
L2 Adaptation decreases out-of-domain performance. We show out-of-domain performance foreach adaptation technique in Table 3. We show thatconversely to L2 and L1-to-L2 adaptation whichsignificantly improved in-domain performance, thiscomes with the tradeoff at less performance in both

Domain

NATU

TECH

SOCI

HEAL

HIST

RELI

CULT

GENE

MATH

HUMA

PHIL

Avg

NATU
TECH
SOCI
HEAL
HIST
RELI
CULT
GENE
MATH
HUMA
PHIL

—
23.6
23.8
24.3
24.7
26.3
23.7
25.4
26.3
23.9
25.1

25.5
—
24.2
25.6
25.3
28.2
24.3
26.4
26.7
24.0
25.7

22.1
20.8
—
21.6
20.7
21.9
20.6
21.8
23.7
20.1
21.8

20.0
19.4
19.8
—
21.8
22.8
20.1
22.1
23.1
20.7
21.3

24.5
23.1
22.3
24.4
—
24.0
23.0
24.2
26.4
22.0
24.4

23.5
22.7
21.7
23.7
21.4
—
22.1
23.2
25.0
21.3
22.9

25.7
23.5
23.4
25.2
24.2
25.8
—
25.4
27.1
24.1
24.7

23.7
22.6
22.0
24.0
22.8
24.4
22.5
—
25.2
22.3
23.5

21.0
21.7
22.6
25.0
24.0
26.0
22.8
24.5
—
23.2
20.9

25.6
24.5
24.1
26.2
23.9
26.3
24.4
26.2
28.3
—
26.0

23.2
22.4
22.0
23.9
22.6
24.0
22.2
23.3
24.4
22.3
—

23.3
22.5
22.4
23.9
23.0
24.5
22.5
24.1
25.0
22.5
23.5

Avg

24.4

25.3

21.3

20.8

23.6

22.5

24.6

23.1

22.7

25.3

22.9

23.3

Table 4: Out-of-domain transfer performance between all L1 domains (using abbreviations from Table 1) in the
Wikipedia portion of M2D2. For each domain, we use the first four letters to refer to itself. The x-axis shows
evaluation domains, and the y-axis shows training domains.
Domain

math

econ

cs

cm

physics

Art

Philosophy

stat

q-bio

nlin

astro-ph

Avg

math
econ
cs
cm
physics
Art
Philosophy
stat
q-bio
nlin
astro-ph

—
18.5
12.6
13.6
14.1
22.9
20.8
12.7
13.7
11.0
16.6

25.0
—
17.6
20.7
20.5
25.8
24.7
14.0
18.1
19.7
23.9

22.0
23.8
—
21.6
21.0
25.9
23.4
18.2
19.2
20.7
25.2

21.2
24.6
18.0
—
14.1
27.5
26.2
17.9
14.6
13.3
17.1

35.8
35.1
24.5
17.9
—
31.1
31.2
24.8
20.9
22.3
23.6

66.1
48.1
43.2
55.9
46.2
—
30.4
47.0
48.1
51.7
54.4

57.2
43.9
40.1
50.0
41.9
29.0
—
43.2
42.9
45.8
48.1

19.6
15.3
13.9
16.2
15.8
21.3
20.3
—
13.6
15.7
17.8

38.8
33.7
26.6
26.4
24.8
29.1
28.1
26.6
—
25.9
30.9

13.6
20.0
14.0
13.2
13.3
22.6
21.5
14.8
14.3
—
15.2

43.2
37.5
28.6
25.8
22.1
31.7
31.4
27.0
26.8
25.9
—

32.0
28.8
23.1
24.9
22.8
26.8
25.7
23.4
23.2
23.8
26.2

Avg

15.1

20.5

21.5

18.8

25.8

47.1

42.4

16.4

28.5

15.7

28.7

25.5

Table 5: Out-of-domain transfer performance between all L1 domains in the S2ORC portion of M2D2. “GPT2”
refers to the zero-shot performance of the LM on our dataset.

L2 and L1-to-L2 settings when compared to L1
Adaptation.
Specific adaptation transfers better to related
categories across ontology. Although the twodata sources in M2D2 differ considerably in styleand content, their ontological categories partiallyoverlap. For example, Mathematics and Art appear
in both Wikipedia and Semantic Scholar. Is it possible to transfer between corresponding categories
across ontologies?
To answer this question, we first manually alignL1 domains from Wikipedia and Semantic Scholarwith similar ontological categories (e.g., grouping Mathematics from Wikipedia and Mathematics
from S2ORC). We then apply a model adapted toan L1 domain in a source ontology onto its corresponding L1 domain in a target ontology. We
compare this cross-ontology performance with twobaselines: 1) the average out-of-domain performance of other L1 adapted models in the target
ontology and 2) the in-domain performance of amodel adapted to the target L1 domain.
Our results are displayed in Table 6. We observethat while L1 adapted models are effective at transferring to other domains within an ontology, they
are less effective at transferring to corresponding
domains outside an ontology. Surprisingly, in allcases, transferring outside an ontology performs
even worse than using the base GPT-2 model withno additional adaptation. Moreover, the averageout-of-domain performance of L1 adapted models
generally outperforms cross-ontology performance,
indicating properties shared within an ontology(e.g. style) could be transferred.
Summary Our investigations into the out-ofdomain performance of adapted language models

S2ORC

Mathematics

Computer Science

Art

Philosophy

Physics

S2ORC (in-domain)
Wiki (in-domain)
S2ORC (out-of-domain)

9.2
19.6
15.1

15.4
26.8
21.5

27.7
35.3
47.1

24.4
33.4
42.4

17.1
29.6
25.8

Wiki

MATH

TECH

CULT

PHIL

NATU

Wiki (in-domain)
S2ORC (in-domain)
Wiki (out-of-domain)

18.3
29.6
22.7

22.3
29.5
25.3

21.2
26.8
24.6

21.8
27.0
22.9

20.8
31.5
22.9

Micro averaged PPL over all L2 domains

Table 6: Transfer performance between corresponding domains(Math↔Mathematics and Logic(Math), Computer
Science↔Technology and Applied Sciences, Art↔Culture and the Arts, etc..) in both ontologies. It can be seenthat provenance is a stronger indicator of transfer performance on M2D2 than ontological correspondence.

Noun, Pearson (r = 0.79, p < 0.01)
Adj, Pearson (r = 0.77, p < 0.01)
Verb, Pearson (r = 0.85, p < 0.01)
Entities, Pearson (r = 0.78, p < 0.01)

36
34
32
30
28
26
24
22
20
0.2

0.3

0.4

0.5

0.6

Overlap by POS/Entity

0.7

0.8

0.9

Figure 4: The relationship between overlap and transfer
performance over all Wikipedia L2 domains. Entities,
verbs, nouns and adjectives are all strongly correlatedwith performance across domains.

reveals a tradeoff between specialization and generalization. The more fine-grained the specializationof the language model, the less one can expect it tobe applicable outside of the domain it was trainedon. This effect size increases as we move outside
the ontology: models trained on one ontology are
not useful in other ontologies, despite being trainedon similar categories of data. These findings lead us
to believe that domain adaptation should be studied
from a multi-faceted perspective to exploit specificaspects of domain (e.g. style, content). Future
work may look at reducing the tradeoff between
highly domain specialized models and out of domain performance, perhaps through ensembling orother approaches.
3.4

Lexical indicators of out-of-domain
performance

Looking closer at the out-of-domain performanceof L1 models, we see intuitive relationships be-

tween subject similarity and zero-shot out-ofdomain transfer performance (Table 4). For example, Society and Human Activities domains tend
to transfer well to each other, whereas Religion andMathematics do not transfer as well. These findings
suggest that out-of-domain transfer is correlatedwith content overlap. In this section, we present
some basic lexical indicators of out-of-domain performance which support this hypothesis.
Vocabulary overlap strongly correlates withtransfer regardless of part-of-speech. Figure 4
shows the correlation of vocabulary overlap a given
part-of-speech tag (VERB, NOUN, ADJ) or entities and average out-of-domain performance on
M2D2. We compute this by taking the top-k(k = 1000) most common words for a given domain which correspond to a given POS tag. For every given domain, we then calculate the intersectionof shared most common words corresponding to thepart-of-speech tag with the entirety of M2D2 andplot them against the L2-domain-averaged perplexity over the entire dataset. We use spacy (Honnibal and Montani, 2017) for both entity recognition
and POS tagging. We find that vocabulary overlap is a strong predictor of transfer performance
regardless of part-of-speech, perhaps indicating its
relevance in transfer between fine-grained domains.

Related domains mostly transfer domainspecific tokens. We analyse domain adaptation
at a token-level to characterize what different adaptation settings transfer. Specifically, we measure
which tokens are most impacted in terms of perword perplexity when we finetune on a domainspecific corpus. We do this by taking the difference
between the softmax-normalized probability of pre-Transfer

Domain-specific

GeneralExamplesDistant L1
Easy L1
Zero-shot
L1-to-L2

25.7%
12.3%
23.4%
31.6%

74.3%
87.7%
76.6%
68.4%

Blockchain, Alexathe, cacherenewals, Markov
lambda DCS, Tacotron

Table 7: Average percentage of tokens transferred indomain and out of domain. Examples are taken from
Philosophy→Computer Science, Statistics→Computer
Science, GPT2→Computer Science, and Computer
Science→Computation and Language.

dicting a given word in a given domain when comparing two models adapted to different corpora.
We compare S2ORC adapted models in four settings: two best-transferred domains (a proxy forsimilar domains; easy transfers), two worst transferred L1 domains (a proxy for distant domains;difficult transfers), L1-to-L2 Adaptation (hierarchical domain transfer), and no adaptation (zero-shot
performance of the base LM). We show the distribution between domain-specific (terms that appear
less than 0.00001% of the time in any other domain)and non-domain-specific terms in Table 7 that appear in the top 1000 most adapted words. Finally,
we show representative samples of tokens with thegreatest change after adaptation. We find that themost changed tokens between easy transfers (e.g.
Statistics and Computer Science) are non-domainspecific words (such as the) but harder transfers
include words that are more domain specific (suchas Blockchain).
Summary Our preliminary analyses suggest that
simple lexical characteristics of domains are strong
indicators of how well an adapted model may generalize. Developing computationally inexpensive
indicators of transfer (as lexical overlap is), is important for domain transfer to find the best out ofa large set of candidate corpora to perform adaptation to a target domain. This would allow one
to approximately find the best corpus, without thecomputational overhead of adapting to all candidate corpora.

4

Related Work

Domain Adaptation Techniques (Gururangan
et al., 2020) show that pretrained language models can be adapted to new domains by continued pre-training on domain-specific corpora.
Chronopoulou et al. (2021); Gururangan et al.
(2021) build upon this work by using hierarchically constructed domain specific adapters/experts(Houlsby et al., 2019). Another line of work indomain generalization is to simply scale the model
pre-training on a corpus containing different domains (e.g. GitHub, PubMed) such as done withGPT-J (Wang and Komatsuzaki, 2021) and the Pile
(Gao et al., 2021). Dery et al. (2021) also look tobridge these approaches by learning a task/domain
specific mixture of tasks. Overall, however, muchof this work (Daumé III, 2007; Ruder et al., 2017;
Ruder and Plank, 2018; Gururangan et al., 2020;
Ramponi and Plank, 2020; Gururangan et al., 2021;
Chronopoulou et al., 2021) fits in a paradigm inwhich a base model is trained further on domainspecific corpora and then testing on tasks withinthat domain (e.g. abstract sentence role classification (Bird et al., 2008) for the scientific domain).
M2D2 is complementary to these works in providing a testbed for fine-grained and hierarhicaladaptation across a large quantity of domains.
Domain Adaptation Datasets One approach toward improved pre-trained language models includes building large-scale pre-training datasetsthat contain a diverse set of domains, such as thePile (Gao et al., 2021). Overall, this emphasis haslead to improved performance in various domains,
especially with large-scale pre-trained language
models, such as GPT-J (Wang and Komatsuzaki,
2021). Another line of work has been in documenting large-scale web-crawled datasets, so practitioners and researchers can be more informed andmindful of the data used (Dodge et al., 2021). Our
work extends this thread with a massively multidomain corpus with a manually curated ontologythat can be used to study fine-grained and hierarchical domain transfer.

5Conclusion

We developed M2D2, a new massively multidomain language modeling dataset for studying
domain adaptation in language models. M2D2
consists of 145 fine-grained domains (curated from
Wikipedia and Semantic Scholar) that are hierarchically organized using domain-specific ontologies. Using M2D2, we find that domain precision
is more important than data quantity to improvein-domain performance, a tradeoff between specialization and out-of-domain generalization. We
release M2D2 publicly to spur further researchon building effective language models on highly
heterogeneous data.

6

Limitations

In this work, we only consider adaptation techniques that assume domains are monolithic andnon-overlapping. Future work may instead explore modeling the data as a mixture of domains,which may improve out-of-domain performance.
In addition, M2D2 only covers two data sources(Wikipedia and Semantic Scholar). Future work
could expand this corpus with ontologies fromother data sources, such as Reddit, which havea fine-grained and hierarchical domains. Moreover, data sourced from the web may contain hatespeech and other harmful content, which may be reproduced by language models adapted to such data.
The data sources we use adhere to research-friendly
data licenses, but training models on web-curated
data while maintaining the rights of authors as data
subjects and creators remains an open problem.

Acknowledgements
We thank Nikita Haduong, Jungo Kasai, Sophia
Serrano, Wenya Wang for their feedback and proofreading comments. We also thank Sebastian Ruder
for useful discussions. MR is grateful to the Masason Foundation for their support.

References
Steven Bird, Robert Dale, Bonnie Dorr, Bryan Gibson,
Mark Joseph, Min-Yen Kan, Dongwon Lee, BrettPowley, Dragomir Radev, and Yee Fan Tan. 2008.
The ACL Anthology reference corpus: A reference
dataset for bibliographic research in computational
linguistics. In Proceedings of the Sixth International
Conference on Language Resources and Evaluation(LREC’08), Marrakech, Morocco. European Language Resources Association (ELRA).
Alexandra Chronopoulou, Matthew E. Peters, and Jesse
Dodge. 2021. Efficient hierarchical domain adaptation for pretrained language models.
Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of
the Association of Computational Linguistics, pages
256–263, Prague, Czech Republic. Association for
Computational Linguistics.
Lucio M. Dery, Paul Michel, Ameet Talwalkar, and Graham Neubig. 2021. Should we be pre-training? anargument for end-task aware training as an alternative.
arXiv preprint arXiv: Arxiv-2109.07437.
Jesse Dodge, Maarten Sap, Ana Marasović, William
Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret
Mitchell, and Matt Gardner. 2021. Documenting

large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the2021 Conference on Empirical Methods in Natural
Language Processing, pages 1286–1305, Online andPunta Cana, Dominican Republic. Association for
Computational Linguistics.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang,
Horace He, Anish Thite, Noa Nabeshima, Shawn
Presser, and Connor Leahy. 2021. The pile: An
800gb dataset of diverse text for language modeling.
Suchin Gururangan, Mike Lewis, Ari Holtzman,
Noah A. Smith, and Luke Zettlemoyer. 2021. Demixlayers: Disentangling domains for modular language
modeling. arXiv preprint arXiv:2108.05036.
Suchin Gururangan, Ana Marasović, Swabha
Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey,
and Noah A. Smith. 2020. Don’t stop pretraining:
Adapt language models to domains and tasks. In
Proceedings of the 58th Annual Meeting of the
Association for Computational Linguistics, pages
8342–8360, Online. Association for Computational
Linguistics.
Matthew Honnibal and Ines Montani. 2017. spaCy 2:
Natural language understanding with Bloom embeddings, convolutional neural networks and incremental
parsing. To appear.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski,
Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.
Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long
Beach, California, USA, volume 97 of Proceedingsof Machine Learning Research, pages 2790–2799.
PMLR.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A
method for stochastic optimization. In 3rd International Conference on Learning Representations,
ICLR 2015, San Diego, CA, USA, May 7-9, 2015,
Conference Track Proceedings.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The semantic
scholar open research corpus. In Proceedings of the58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983, Online. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam
Lerer, James Bradbury, Gregory Chanan, Trevor
Killeen, Zeming Lin, Natalia Gimelshein, Luca
Antiga, Alban Desmaison, Andreas Köpf, Edward
Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang,
Junjie Bai, and Soumith Chintala. 2019. Pytorch: An
imperative style, high-performance deep learning library. In Advances in Neural Information Processing

Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages
8024–8035.
Alec Radford, Jeff Wu, Rewon Child, David Luan,
Dario Amodei, and Ilya Sutskever. 2019. Language
models are unsupervised multitask learners.
Alan Ramponi and Barbara Plank. 2020. Neural unsupervised domain adaptation in NLP—A survey. In
Proceedings of the 28th International Conference
on Computational Linguistics, pages 6838–6855,
Barcelona, Spain (Online). International Committee
on Computational Linguistics.
Sebastian Ruder, Parsa Ghaffari, and John G. Breslin.
2017. Knowledge adaptation: Teaching to adapt.
arXiv preprint arXiv: Arxiv-1702.02052.
Sebastian Ruder and Barbara Plank. 2018. Strong baselines for neural semi-supervised learning under domain shift. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics(Volume 1: Long Papers), pages 1044–1054, Melbourne, Australia. Association for Computational
Linguistics.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky,
Ilya Sutskever, and Ruslan Salakhutdinov. 2014.
Dropout: A simple way to prevent neural networks
from overfitting. Journal of Machine Learning Research, 15(56):1929–1958.
Ben Wang and Aran Komatsuzaki. 2021. GPTJ-6B: A 6 Billion Parameter Autoregressive
Language Model.
https://github.com/
kingoflolz/mesh-transformer-jax.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien
Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen,
Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu,
Teven Le Scao, Sylvain Gugger, Mariama Drame,
Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing.
In Proceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing: System
Demonstrations, pages 38–45, Online. Association
for Computational Linguistics.

A

Appendix

A.1

Hyperparameters
Computing Infrastructure

8 Volta 16GB GPUs

Hyperparameter

Assignment

architecture

GPT-2

tokens per sample

1024

batch size

64000

number of workers

8learning rate

5e–5

clip norm

0.1

number of steps

1,000,000

save interval updates

1,000

validation interval

1,000

number of warmup steps

10,000learning rate scheduler

polynomial decay

learning rate optimizer

Adam

Adam beta weights(0.9, 0.99)

Adam epsilon

1e-6

weight decay

0.1

Table 8: Hyperparameters for finetuning in all settings.

A.2

Licenses

Our data sources have open licenses. Wikipedia has a Creative Commons Attribution-ShareAlike 3.0
Unported License and a S2ORC has a Creative Commons Attribution-NonCommercial 4.0 International(CC BY-NC 4.0).
A.3

More examples of most transferred tokens

We give more examples of tokens transferred from the L1 S2ORC Computer Science (given its assumed
familiarity to our audience) domain in the following table:
Transfer

Example Tokens

Computer Science→Computation and Language

lambda DCS, perplexity, Artetxe, Tacotron,
Swayamdipta, Transformer, parallel, Socher, Gigaword, Lapata
Computer Science→Machine Learning
criterion, Ganchev, Ioffe, labeling, autoencoder, Hinton, hyperparameters
Computer Science→Art
Atheist, heroism, intellectuals, horrors, witchcraft,mourning, apostles
Computer Science→Technology and Applied Sci- Sunderland, accounting, inventory, Libyan, bishop,
ences
ravaged, traffic
Table 9: More examples of most transferred tokens

A.4

All domains

We list all domains contained within dataset in Table 10.

S2ORC
cs.CE, cs.IT, cs.CG, cs.SI, cond-mat.quant-gas, math.SG, cs.SC, cs.CY, econ.GN, math.CO, cs.AR,
cs.MS, cs.DC, q-bio.TO, cs.GR, physics.acc-ph, physics.geo-ph, math.RT, math.HO, cs.RO, q-bio.SC,
math.QA, cs.NI, math.CA, cs.DS, astro-ph.GA, physics.atom-ph, math.CT, cs.CV, cond-mat.mtrl-sci,
math.CV, math.AC, cond-mat.str-el, physics.comp-ph, cs.CC, math.FA, cond-mat.dis-nn, econ.TH,
physics.gen-ph, physics.data-an, astro-ph.IM, q-bio.CB, math.LO, physics.ins-det, q-bio.BM, cs.LO,
math.GR, physics.optics, cs.GT, math.AG, cs.NE, cs.SY, physics.bio-ph, physics.flu-dyn, cs.CL, math.MG,
cs.AI, math.OC, nlin.CG, math.IT, stat.OT, math.OA, cond-mat.soft, Art, cs.GL, cs.PF, math.ST,
physics.ao-ph, physics.plasm-ph, math.RA, physics.hist-ph, cs.PL, cs.MA, physics.chem-ph, physics.socph, physics.med-ph, physics.ed-ph, stat.AP, stat.CO, math.DS, cs.DB, nlin.SI, q-bio.GN, physics.atmclus, nlin.CD, astro-ph.CO, cs.CR, cond-mat.supr-con, cs.LG, math.KT, stat.ML, nlin.PS, q-bio.MN,
cs.IR, math.GT, cs.SD, math.NA, cond-mat.other, math.NT, cs.FL, physics.pop-ph, cond-mat.stat-mech,math.GN, cs.DL, astro-ph.EP, q-bio.QM, cs.ET, q-bio.PE, cs.OH, Philosophy, physics.space-ph, econ.EM,
physics.class-ph, cs.DM, cond-mat.mes-hall, stat.TH, cs.SE, astro-ph.HE, math.MP, nlin.AO, math.AP,
q-bio.NC, q-bio.OT, astro-ph.SR, math.DG, math.AT, cs.MM, stat.ME, cs.OS, math.SP, physics.app-ph,cs.NA, math.PR, math.GM, cs.HC
Wikipedia
Culture and Humanities, Games and Toys, Mass media, Performing arts, Sports and Recreation, The
arts and Entertainment, Visual arts, Further research tools and topics, Reference works, Exercise, Healthscience, Human medicine, Nutrition, Public health, Self care, By continent, By period, By region, Humanactivities, Impact of human activity, Fields of mathematics, Logic, Mathematics, Biology, Earth sciences,
Nature, Physical sciences, Philosophy, Thinking, Allah, Belief systems, Major beliefs of the world, Social
sciences, Society, Agriculture, Computing, Engineering, Transport
Table 10: All domains contained within M2D2