***FastDoc***: Domain-Specific Fast Continual Pre-training Technique using Document-Level Metadata and Taxonomy
Authors: Abhilash Nandy
Manav Nitin Kapadnis
Sohan Patnaik
Yash Parag Butala
Pawan Goyal
Niloy Ganguly
Venue: Accepted by TMLR ()

Abstract: In this paper, we propose FastDoc (Fast Continual Pre-training Technique using Document Level Metadata and Taxonomy), a novel, compute-efficient framework that utilizes Document metadata and Domain-Specific Taxonomy as supervision signals to continually pre-train transformer encoder on a domain-specific corpus. The main innovation is that during domain-specific pretraining, an open-domain encoder is continually pre-trained using sentence-level embeddings as inputs (to accommodate long documents), however, fine-tuning is done with token-level embeddings as inputs to this encoder. We perform such domain-specific pre-training on three different domains namely customer support, scientific, and legal domains, and compare performance on 6 different downstream tasks and 9 different datasets. The novel use of document-level supervision along with sentence-level embedding input for pre-training reduces pre-training compute by around 1,000, 4,500, and 500 times compared to MLM and/or NSP in Customer Support, Scientific, and Legal Domains, respectively. The reduced training time does not lead to a deterioration in performance. In fact we show that FastDoc either outperforms or performs on par with several competitive transformer-based baselines in terms of character-level F1 scores and other automated metrics in the Customer Support, Scientific, and Legal Domains. Moreover, reduced training aids in mitigating the risk of catastrophic forgetting. Thus, unlike baselines, FastDoc shows a negligible drop in performance on open domain.

Published in Transactions on Machine Learning Research (05/2024)
FastDoc: Domain-Specific Fast Continual Pre-training Technique using Document-Level Metadata and Taxonomy
Abhilash Nandy
mandyachilashhkggginnn..iigg
Deportment of Computer Science Indian Institute of Technology Kharugpur
Manav Nitin Kapadnis
maaapadniiSanddee..ccm cma,, edu
School Computer Science
Carnegie Mellon University
Sohan Patnaik
sokamppanncikk100000iia
Department of Mechanical Engineering Indian Institute of Technology Kharagpur
Yash Parag Butala
upb www. edu
School of Computer Science
Carnegic Mellon Universily
Pawan Goyal
pawang Sitttngg@eeeee.kgg.....
Department of Computer Science Indian Instttute of Technology Kharagpur
Niloy Ganguly
niloy Bose. Kilkyp. ac.in
Department of Computer Science
Indian Institute of Technology Kharagpur
Reviewed on OpenReview:
Abstract
In this paper, propose FastDoc (Fast Continual Pre-training Technique using Document Level Metadata and Taxonomy). novel, compute-efficient framework that uti- lizes Document metadata and Domain-Specific Taxonomy as supervision signals to continu- ally pre-train transformer encoder on a domain-specific corpus. The main innovation that during domain-specifie pretraining, an open-domain encoder continually pre-trained using sentence-level embeddings as inputs (to accommodate long documents). however, fine- tne-tuning is done with token-level embeddings inputs to this encoder. We perform such domain- specific pre-training three different domains namely customer support, scientific, and legal domains, and compare performance on 6 different downstream tasks and 9 different datasets. The novel use of document-level supervision along with sentence-level embedding input for pro-training roduces pro-training compute by around 1, 10000, 4, 500, and 500 times compared to MLM and/or NSP Customer Support, Scientific, and Legal Domains, re- spectively[] The reduced training time does not lead a deterioration in performance. In fact show that Fast.Doc cither outperforms performs on par with several competitive transformer-basee baselines in terms of character-level F1 scores and other automated met- rics in the Customer Support, Scientific, and Legal Domains. Moreover, reduced training aids in mitigating the risk of catastrophic forgetting. Thus, unlike baselines, FastDoc shows a negligible drop in performance on open domain.
Code and datasets available https:://gttbb.ccc///////o// http://www.../oaaaaaaa//aaaaaaaa Pre-trrinttnnn---- Technique/

---

Published in Transactions on Machine Learning Research (05/2024)
1 Introduction
In present times, continual pre-training (Arumae et al., 2020 Gururangan et al. 2020) on unlabelled, domain- specific text corpora (such as PubMed articles in medical domain, rescarch papers in Scientific Domain, E-Manuals in Customer Support Domain, elc.) has emerged as an important training strategy in NLP to enable open-domain transformer-baset language models perform various downstream NLP tasks such as Question Answering (QA), Named Entity Recognition (NER), Natural Language Inference (NLI), etc. on domain-specific datasets (Hendrycks et al., 2021 Beltagy et al. 2019 Nandy et al. 2021]. Most of the pre-
training strategies involve variants of Masked Language Modelling (MLM) (Liu et al., 2019). Next Sentence
Prediction (NSP) (Devlin al.) 2019), Sentence Order Prediction (SOP) (Lan al. 2019) etc. that use local sentence pantencce/span-level conlexts as supervision signals. However, such methods require lot of pre-training data and compute. For instance pre-training of BERTBASE architecture on 3.17 billion word corpus was performed on 8 GPUs for around 40 days to obtain SCIBERT (Beltagy et al. 2019).
MLM-style domain-specific pre-training makes an implicit assumption that the constituent documents are independent of each other, which may not be true always. Documents from a particular domain (e.g.. customer support, scientific papers, legal procoodings, etc.) may be categorized into different groups by experts in that area, each group containing similar documents. This information is generally stored either 'metadata' of the document. (Borchert. et al.
2022 Lipscomb, 2000), or in terms of a 'taxonomy' (Margiotta et al., 2022; Karamanolakis et al. 2020 of documents. For example, E-manuals of different versions of a cell phone series are very similar, scientific articles written on a particular topic (e.g., pre- training) follow a certain type of taxonomy, legal proceedings on related crimes are similar. While few modols such as LinkBERT (Yasunaga et al., 2022), MetricBERT (Malkiel et al., 2022). ; etc. have used document metadata as an additional signal, no work to the best of our knowledge has singularly leveraged taxonomy- information
2020:
Contrarily, in this paper, we completely replace the local context-based supervision (MLM, NSP, etc.) dur- ing pre-training with (a). document similarity learning task using the available domain-specific meladata (through a triplet network), and (b). hierarchical classification task that predicts the hierarchical categories corresponding to the domain-specific taxonomy in a supervised manner.
However, to leverage docurnent-level supervision, a robust encoding of documents is required. We use a hierarchical architecture (Zhang al., 2019) and propose various innovations (see Figure D (a). We initialize the lower-level encoder using a pre-trained sentence transformer (sBEER//SRoBLLI (Reimers & Gurevych 2019t) and freeze its weights. We then initialize the higher-level encoder using pre-trained HERT/RoBEREE encoder, which now operates with a sentence embedding input, received via the lower-level encoder. This design choice (inspired by works that initialize a larger encoder through a smaller pre-trained encoder - c.g., Bert2BERT (Chen et al., 2022)) helps us to directly work with sentence embeddings as inputs which in turn enables much larger contexts in a single input, and decreases the required pre-training compute by a huge margin. (b). After pre-training, we use only the higher-level encoder for downstream sentence and token-level tasks. As the higher-level encoder was originally pre-trained with token embedding inputs, it can still be tne-tuned with token embedding inputs. We conduct various experiments to analyze this very interesting and surprising aspect of interoperability of token and sentence embedding inputs.
Using these ideas, we propose FastDoc pre-training framework, and apply it to varied NLP tasks across three disparate domains - Customer Support, Scientificc Papers, and Legal Domain, to evaluate the generalizability of FastDoc across multiple domaing Customer Support requires answering consumer queries related to device maintenance, troubleshooting, etc., and hence, we apply FastDoc on two Question Answering lasks. In the domain of scientifie papers, we focus on tasks such as extracting important scientific
keywords Li et al. 2016 Kim et al. 2004 Dogan et al.) 2014), extracting the type of relation between such
keywords Kringelum et al. 2016 Luan et al. 2018 as well as classifying citation intents (Cohan et al.
DDetailed Prior Art is described in Section of Appendia. CContimally Pre-training single model across domains does give good performance in all domains. That is why there are works for developing models for particular domain, such as BioBERT Lee et (2020). ScIEERT Beltagy al. (019 F.aamaas BERT Nandy et al. (2021). I Legal-BERY Chalkidis al. (2020). FinBERT Huang al. 2023
2

---

Published in Transactions on Machine Learning Research (05/2024)
2019 In the legal domain, focus the task of automating contract review Henddyycks et al. 2021 which involves finding key clauses in legal contracts.
We show that FastDoc drastically reduces (order of 500x) pre-training compute across domains while still achieving comparable to modestly better performance in downstream tasks. We further show that the result holds even when we increase model size and consider situations where document metadata and laxonomy may not be explicitly available. We also show that the frugal pre-training helps FastDoc resist catastrophic forgetting very common when transformers undergo continual in-domain pre-training Gururangan et al.
2020
Arumae et al. 2020
Document Representation
Pre-trained
using FastDoc
Transformer Encoder
Transformer Encoder
Pre-trained Encoder
Tum
Tm
Tim
Fine-tune on Downstream Tasks
Token Embeddings
Sentence Embeddings
BERT/RoBERTa
Token Embeddings
Domain-Specific Pre-trained Model
Pre-trained Sentence st> Transformer (Frozen)
Hierarchical
Document
Encoder
COPY
Input Document
Figure End-to-end training pipeline using FastDoc
2 FastDoc Framework
The aim of FastDoc is to learn robust representations for documents (in specialized domains) using potent locument-level supervision signals. We treat a document as sequence of sentences and provide pre-trained sentence embeddings as imput. This, in turn, helps in accommodating documents that contain more than 512 tokens even using a standard BERTBASE /RoBERTagASE encoder (c.g. from Figure 3 in Section B of Appendia, we observe that using sentences as inputs enable coverage of around 50% more documents than when tokens are inputs). We train the network with two losses. (a). The first loss is contrastive or triplet loss based on the similarity or dissimilarity of document with pair of documents; (b). The sccond loss is supervised loss derived while classifying document to domain-specific taxonomy.
Figure D depicts the end-to-end training pipeline using the proposed FastDoc architecture (detailed pre- training architecture shown in Figure 4 in Section B of Appendix). Typically a hierarchical document encoder like HiBERT (Zhang et al.) 2019 would be suitable model for encoding documents. It has a lower-level encoder with token inputs and a higher-level encoder with sentence-level inputs. In general, during pre-training, both these encoders need to be tuned (which is computationally expensive) and only the lower-level encoder is utilized for downstream sentence and token-level tasks such QA, Relation Classification, NER, etc. However, we propose different, compute--fficient method. The steps in pipeline are (a). The pipeline starts using open-domain pre-trained transformer model (c.g. BERT (019t/RoBERe RRoBERTa (Liu et al. 2019h) for fast convergence in domain-specific scenarios. (b).
DDvlin et al. 2019) Its transformer layers excluding the input token embedding layer are used to initialize the higher-level encodor, while the lower-level encoder is a frozen sBERT/s this document encoder obtained by averaging the output context-aware sentence representations from the higher-level encoder. (c). The higher-level encoder is (further) pre-trained with document-level supervision
The Document representation from
3

---

Published in Transactions on Machine Learning Research (05/2024)
using the proposed FastDoc Framework on domain-specific documents, (d). Finally, only this higher-level encoder is fine-tuned on downstream tasks, with input token embeddings copied from the open-domain model.
Our specific design choices help in the following manner - (a) Freezing the sentence embeddings while training the encoder with document-level loss helps in achieving fast. pre-training. (b) While a hierarchical encoder could also have used the document-level loss, the lower-level encoder using token inputs would be directly used for fine-tuning, but this encoder would learn less robust pre-training task-specific, semantic features compared to the higher-level encoder (Tenney et al. 2019 van Aken et al. 2019h. Our design trains the higher-level encoder to make the best use of pre-training loss. Next, we describe the pre-training loss functions in great detail. The inter-operability of input token and sentence embeddings is reasoned via several experiments in Section 7.4 and Section G.4 of Appendix.
Contrastive Learning using document similarity labels.
We use a Triplet Network (Cohan et al. 2020), where three documents serve as input for three document encoders, the first (anchor) and socond (positive) documents being similar, and the first and third (negative) documents being dissimilar (based on metadala). The encoders have hard parameter sharing (Caruana 1993). The three encoded representations used to formulate triplet margin loss function, denoted by L. Mathematically,
LD D2, D2) = max(d(Di, D2) d(D1, D3) 1,03
(1)
where D1, D2, D3 refer to the document representations of documents, and d(...) represents the L2 norm distance. We use a unit margin in accordance with prior art using the same or similar contrastive loss functions (Oh Song el al., 2016 Weinberger & Saul 2009).
Note that we do not use NT-Xent (normalized temperature-scaled cross entropy) Loss Function (Chen al, 2020), which uses multiple negatives for given (anchor, positive) pair, using such large number of negatives would significantly increase the compute (corresponding to the augmentation, forward pass, and backpropagation for a large number of inputs), which defeats FastDoc's purpose.
Hierarchical Classification using Hierarchical Labels.
Here we try to formulate a Supervised Hierarchical Classification Task based on a domain-specific hierarchical taxonomy. Given a document, the task is to predict the hierarchical categorics present in the taxonomy.
In FastDDc, each document's representation is passed through H classification heads, H being the maximum number of hicrarchical levels present the taxonomy. may so happen that the hicrarchy for document has less than H levels. Hence, to bring uniformity, 'null' class is added to each remaining level. For Hierarchical Classification, Local Classifier per Level (LCL) (Silla & Freitas 2011) is used, where one multi- class classifier is trained for each level of hicrarchy. At each level, a classification head is an MLP layer (followed by Max). The hierarchical loss function Laier is the sum of the categorical cross-entropy loss (CF.oss)) over all the H classification heads, for all the N input, documents per training sample (in our case, N = 3). Mathematically,
N H
Chier ΣΣ CELoss(Zij...) 9iy).
(2)
Fij and Vij are predicted and target class distributions respectively, for the ish document, and jeh classifi- cation head.
The loss backpropagated during pre-training the sum of the triplet margin loss and the hierarchical loss functions.
4

---

Published in Transactions on Machine Learning Research (05/2024)
3 Pre-training Setup
We represent BERT-bbase and RoBERTTabased FostDoc as BaStDOCBERT and FostDoc respec- tively, along with abbreviation of the domain (Customer Support Cus., Scientific Domain - Sci., Legal Domain Leg.). The proposed models and domain-specific baselines are pre-trained (in-domain) for 1 epoch. We use a batch size of 32, and AdamW optimizer (Loshchilov & Hutter 2018 with an initial learning rate of 5 x 10-5, which linearly decays to 0.
We next outline the specifics of the dataset used, its associated taxonomy, metadata leveraged. Table shows examples of sample triplets and hicrarchies from each domain.
Domain, Source
Data
Example Triplet
Example Hierarchy
Customer Support (E-
stereo equalizer E-Mamual,
Stereo Equalizer
Manuals Corpus)
steroo equalizer E-Mamall (of a different brand),
Electronics + Audio + Audio Players
bu buerry player E-Manual
Recorders + Stereo Systems
Scientitic
Domain
Proximal Policy Optimization Algorithms
Generating Natural Adversarial Exam-
(ArXiv)
Generating Natural Adversarial Examples
ples
Autonomous Tracking of RF Source Using a UAV Sworm
Computer Science + Machine Learning "... importation of olive oil ...
Legal Domain (BURLEX57k)
".. ".. import licences ... dairy products"
market research measures ... milk and milk products"
Agriculture + Products subject to market or- ganisation + Oils and fats
importations of fishery and aquacualaree products"
Table 1: Examples of triplets and hicrarchies in the domains. (When representing triplcts, we specify domain-specific meladata- - product calegory in Customer Support, paper Litle in the Scientific Domain, and certain key phrases from each document in the Legal Domain) [2nd column] Underlined phrases denote "positive" documents; italicized phrases denote "negative" documents.
3.1 Pre-training in the Customer Support Domain
Dataset and Triplets Chosen. We pre-train FastDoc on a subset of the E-Manuals Corpus (Nandy et al., 2021)- we sample 2, 000 E-Manual triplets, such that, the anchor and positive E-Manuals belong the same product category and the anchor and negative E-Manuals belong to different product categories. The amount of data is a mere 3% of the entire E-Manuals Corpus.
Hierarchy considered. Google Product Taxonomy (GPrT) (5, 583 possible hierarchies across 7 levels of hicrarchy) is used to obtain hicrarchical classification labels using (a single) category of an E-Manual. This allows similar E-manuals (e.g. 'TV' and 'Monitor') to have more similar hierarchies compared to dissimilar F-Manuals (e.g. "TV" and 'Refrigerator". Details on mapping product category to hierarchy are mentioned in Section C.1 of Appendia.
3.2 Pre-training in the Scientific Domain
Datasct and Triplets Chosen. We pre-train FastDoc on subset of the ArXiv we sample 2, 0.00 triplets of scientific papers based on the "primary category" assigned the paper, such that, the anchor and positive papers belong to the same category, and the anchor and negative papers belong to different categories. For each such triplet, we add another triplet, where the positive and anchor samples are swapped. The amount of data used is negligible compared to the 1.14M Papers used by ScIBERT (Beltagy et al. 2019) during its pre-training. Note that several recent works have used citations as a similarity signal (Cohan et al. 2020 Ostendorff et al. 2022 Yasunaga et al. 2022). However, a paper might cite another paper that is not similar terms of the content. Instead, similarity basod on "primary category" would more intuitively lead to content-based similarity.
Hierarchy Considered. ArXiv Category Taxonomy (consisting of 155 possible hierarchies across 3 levels of hicrarchy) is used to obtain hicrarchical classification labels for each document, where each document is already mapped to its corresponding hierarchy via the laxonomy.
htttps//sspprr.ggggll..o///reeee https://aritvvorg//atgggrr taxonnn
5

---

Published in Transactions on Machine Learning Research (05/2024)
3.3 Pre-training in the Legal Domain
Triplets Chosen. We pre-train Fast.Doc subset of the EURLEX57K datasel (Chalkidis et al. 2019 of legislative documents we sample 2, 000 document triplets based on the list of annotated EUROVOC assigned to each document, such that, the anchor and positive documents have at least Concept in common, and the anchor and negative documents have no Concepts in common. We double the number of triplets a way similar to Scientific Domain. The amount of data used is negligible compared to the 8GB of legal contracts used for domain-specifee pre-training in Hendrrycs al. (2021).
Concept& Concept:@assigned
Hierarchy Considered. The hierarchical class assignments of the documents in the EUR-Lex Dataset [ozaa Mencia et al. 2010) (consisting of 343 possible hierarchies across 4 levels of hierarchy) are used as hierarchical classification labels, where each document is already mapped to its corresponding hierarchy.
4 Downstream Pitasets/Tasks
The efficacy of the pro-training framowork is tested through its performance downstream tasks. We describe those lasks and the corresponding datasets used (The names of all lasks, their corresponding datasets, and domains are listed in Table 15 of Section D of Appendiz).
4.1 Customer Support
We evaluate Question Answering Task on two datasets single span QA on TechQA Dataset and multi-span QA on S10 QA Dataset (described én Section D.1 of Appendia).
TechQA Dataset. TechQA (Castelli et al. 2020 is a span-based QA dataset with questions from a technical discussion forum and the answers annotated using IBM Technotes, which are documents released resolve spocific issues. The dataset has 600 training, 310 dev, and 490 evaluation QA pairs. Each QA pair provided with the document that contains the answer, along with 50 candidale Technotes retrieved using Elasticsearch
Fine-tuning Setup. The fine-tuning is carried out two stages first on the SQuAD 2.0 Dataset (inspired by Castelli et al. (2020)), and then on task-specific QA datasets, which is discussed in Section E.1 of Appendiz) Note that results without intermediate fine-tuning SQuAD 2.0 deteriorate, as shown in
Section E.1 of Appendir.
4.2 Scientific Domain
We multiple datascts from SciBERT Benchmark Datascts (mentioned in Beltagy et al.
2019 for
training and evaluation. The following downstream tasks and corresponding datasets are used for evaluation
(1) NER (Named Entity Recognition): We use the BC5CDR (Li et al. 2016 JNLPBA (Kim et al.
2004), and NCBI-Disease (Dogan et al. 2014) NER Datasets of the Biomedical Domain. (2) REL (Re-
lation Classification): This task predicts the type of relation between entities. The Chem Dataset
KKingelum et al.] 2016) from the Biomedical Domain and SciERC Dataset (Luan et al. 2018 from the
Computer Science Domain are used for evaluation. (3) CLS (Text Classification); SciCite Dataset
(Cohan
el al. 2019 gathered from Multiple Domains is used.
Fine-tuning Setup. We fine-tune and evaluate on the downstream tasks mentioned above. The hyperpa- rameters are the same as that in Beltagy et al. (2019).
4.3
Legal Domain
CUAD (Contract Understanding Atticus Dataset) (Hendrycks et al. 2021 is used, which is annotated by legal experts for the task of Legal Contract Review. It consists 13, 101 clauses across 41 types of clauses
http:///eeuroooeeurrpp...o httass//ww..laatti..////otttttt
6

---

Published in Transactions on Machine Learning Research (05/2024)
annotated from 510 contracts. Given a contract, for each type of clause, the task requires extracting relevant clauses as spans of text related to the clause type. Details of the dataset splits are given in Section D.3 of Appendiz.
Fine-tuning Setup. We fine-tune and evaluate on the Contract Review Task on CUAD. The hyperparam- eters are the same as that in Hendrycks et al. [2021).
5 Experiments and Results
To assess the performance of our proposed methods, we fine-tune and evaluate these methods and baselines the datasets described in Section 4 and draw inferences. Due to space constraints, the performance on the S10 QA Dataset is reported in Section E. of Appendiz. In all these experiments, we perform an ablation study by considering each of the two losses of FastDoc separately i.e. we use only Triplet Loss (triplet) and only Hierarchical Classification Loss (hier.). We perform several additional ablations (sce Section E of Appendiz)- (1) Pre-training both lower and higher-level encoders (entire hierarchical architecture), followed by fino-tuning the lower encoder worsens performance, suggesting higher-level encoder learns better task- with BERT/
specific features (2) replacing sBERT/sRoBBl
worsens performance, suggesting (3). used more fine-graud aue-grained document
sentence transformers provide effective sentence embeddings. similarity criterion (changed Eq. 1 and found the result to be inferior, and (4). compared Fast with the much larger GPT-3.5 model and found that GPT-3.55 models perform much inferior in 0 and 1-shot settings.
5.1 Customer Support Domain
We compare our pre-training approach to 3 types of pre-training baselines described below. For the sake of completeness, we also compare with baselines using span/se span/sentence-level supervision signals. Domain-speeific Continual Pre-training is carried out on the corpus of E-Manuals for all basclines (except BART RoBERTaBASE and Longformer).
Baselines:
(1) Pro-training using masked language modcling (MLM) and/or Next Sentence Prediction
(NSP): We use BERT (Devlin et al.] 2019 ROBERTABASE.
(Liu et al. 2019). LinkBERT BASE BASE 2020), EManualsgERT and EM EManualsOBERR. Nandy
Yasunaga et al. (2022), Longformer (Beltagy et al.
et al. 2021) (domain continual pre- treiiraining of BERT BASE
Devlin et al. 2019) and RoBERT&BASE (Liu et al.
2019), respectively, on the entire E-Manuals corpus). (2)
Using intra-document contrastive learning:
DeCLUTR Giorgi el al. 2021) and ConSERT (Yan et al.
2021) are the intra-docunent contrastive learning (3) Using inter-document contrastive learning: SPECTER (Cohan et al.) 20200) is the
methods.
inter-document contrastive learning bascline used. (Details tailoring SPECTER to Customer Support are given in Section E.1 of Appendix).
Performance on TechQA Dataset The answer-retrieval performance on the development set (as per Castelli et al. (2020)) is reported in Table 2 The model gives five candidate answers per question and corresponding confidence scores. Each answer is assigned an 'evaluation score' If the confidence score is below a threshold provided by the model, 'evaluation score' is 1 if the question actually unanswerable, and 0 otherwise. However, if the confidence score is above the threshold, the 'evaluation score' is character F1 between the predicted answer and ground truth and 0 if the question is actually unansweabbe.. The evaluation metrics used, as mentioned in Castelli et al. (020 are (a). F1 'evaluation score' for the predicted answer (with the highest confidence score) averaged across all questions. similar to F1, except that, the averaging is done on the answerable question set (160 out of 310 questions in the dev set are answerable). (c). HA F1@5- macro average of the 5 best candidate answers per question, averagod across the answerable question set.
(b). 1A@11@1
From the results in Table 2 we can infer (1). Among the baselines, (a) Longformer gives the best F1 and HA Fl@1 and the second- best HA F105. This is because of the long sequence length of 4, 096 compared to
"We do not use BEST_ F1, as a threshold tuned the dev. set using score, which realistic
7

---

Published in Transactions on Machine Learning Research (05/2024)
HA
FI01
HA
F105
BERT
13.67
26.49
36.14
RoBERTaBBSS
16.46
31.89
42.4
LinkBERT BASE
14.24
27.59
36.77
Longformer
16.57
32.1
42.66
EManualSBERT
13.41
25.98
36.69
EManualsRoBRRT.
16.04
31.08
44.71
DeCLUTR
15.11
29.28
38.93
ConSERT
11.12
21.51
30.37
SPECTER
12.92
25.03
34.74
Fast (CatD, (iBs))BERR (hier.)
14.19
27.49
36.62
DastDoc (Crd., BERT (triplet)
14.47
28.04
37.21
Dos BUER)BBRTT
14.56
28.2
35.54
Doc CuS.J aaaeDDDoenn. (hier.)
16.52
32.00
44.77
Do Cus.) (isss.)oBEnrr((riollee
16.39
31.76
46.59
Dost
17.52
33.94
44.96
Table 2: Results for the QA task on the TechQA Dataset.
512 of other m (b) SPECTER does not perform well, even though it uses document-level supervision, as it cannot accommodate the entire document. within 512 tokens, so only the first 512 tokens are used which does not help much in learming.
(c) ConSERT performs contrastive learning on sentence inputs, prohibiting it from learning context beyond a single sentence (unlike FastDoc that learns inter-sentence context during pre-training due its hierarchical architecture), thus reducing performance on QA tasks. (d) In general, contrastive learning baselines perform inferior to those using MLM/NSP. (2). FastDoc ( variants perform better than BERT-based baselines, and Fast.Dc CostDooCCus.. (Cus.) Ro variants than almost all RoBERTa- based baselines, suggesting that our proposed pre-training methods are better than that of baselines. (3) FastDoc (Cus. RoBERTa variants perform better than Dast
variants, as RoBERTa (Liu et al. 2019) performs better than BERT @Dvliin et al. 2019) in span-based QA tasks such as SQuAD [Rajpurkar et al. 2018 2016). (4) FastDoc Cus. RoB ETTa performs the best all models in and F185. Fast (CroocCCu...RBBBe
performs around 6% better than the best bascline Longformer both in terms F1 and HA _A_ F1@1 (even though Longformer has a long sequence length, it is not able to encode most documents properly).
FI@1 and the bectnd-best
Additionally, we perform qualitative analysis in Table 20 of Section E.1 of Appendiz by comparing the ground-truth answers and the answers predicted by FastDoc and well- bascline for 3 answerable questions in TechQA and S10 QA Datasels. This analysis suggests that Fast.Dc is comparatively better at extracting numerical entities, tackling multiple questions in a sample, and answering location-based questions.
5.2 Scientific Domain
Baselines: SCIBERT Beltagy et al. 2019 (pre-trained using MLM and NSP on a huge scientific corpus]
Performance on Different Datasets We fine-tune and evaluate on the datasets mentioned Section The results on the test set for each task are shown in Table 3 We see that Dost.Doc(Sc performs better than ScIBERT However, Fast.Doe(Stt B (hier.) performs the best on 3 datasets with NER and text classification tasks, as (1) fine-grained NER benefits from finc- anc-grrined hicrarchical information, and (2) text classification dataset has samples from multiple domains, where diversity in the hierarchical categories helps.
4 out of 6 datasets, and performs the best the Relation Classification Tasks.
For completenssss, have also continually pre-trained Longformer the data used by FastDoe, and shows inferior results Longformer 2/3 metrics due the data being insufficient to adapt Longformer. 1ovcabbuyryy used for ScIERT same that of BERT BASE for consistency among ScIBRRT and FostDoe variants. Spocif- ically, this model ScIBRRT https:://// scibert, scibert__aseeocab. www.bbaaae_eaaaaaeeaaeaaaaaeaeae
2.aaazonaaw aws. com/ai www.//ii----eeerrraaaaaao we-rssearrc//eeeee wws.erchh/scbbert//uggigggggaii _tcpttoorcccpcooccccooooo

---

Published in Transactions on Machine Learning Research (05/2024)
Fast.Doc
FastDoc (hier.)
Field
Task
Dataset
ScIBERT
FastDoc
(triplet)
BC5CDR
85.55
87.7
87.94
87.81
NER
JNLPBA
59.5
75.86
75.97
75.84
BIO
NCBI-D
91.03
84.15
87.81
84.33
REL
ChemProt
78.55
75.12
80.28
80.48
CS
REL
SciERC
74.3
75.4
75.62
78.95
Multi
CLS
SciCite
84.44
84.31
84.48
83.59
Table 3: FastDoc(Sci, (Sci.) BERT and its variants vs. SCIBERT in tasks presented in Beltagy et al. (2019)
Following Beltagy et al. (2019), we report macro F1 for NER (span-level), and for REL and CLS (sentence- level), except for ChemProt, where we report miero F1.
Since recent works have used citations as a similarity signal, we report the performance of FastDoc using citations as a similarity signal in Table 22 of Section E.2 of the Appendiz. This gives a satisfactory performance, showing that FastDoc works on different types metadata. However, on average, a system using citations does not perform as well as when using "primary category'.
5.3 Legal Domain
Baselines: We use baselines from Hendrycks et al. 2021 Contracts Pre-training (domain-speciff pre-training of RoBERTa- BASE on 8GB of unlabeled contracts collected from the EDGAR database). Also,
BERTBASE ROBERTABASE, RoBERTTpASE +
CDLM Caciularu al. (2021), L.GAL-BERT-FP
Chalkidis al. 2020), and LEGAL-ROBERTT--BE (Geng al. 2021 additional baselines.
Performance on CUAD Dataset
Precision@
Model
AUPR
80% Recall)
BERTRASE
32.4
8.2
LEGAL-BERI-FP
32.6
21.16
RoBERTABASE
42.6
31.1
L.GAAL-RBERRT-BAAT
42.9
31.7
RoBERTABASE Contracts Pre-training
45.2
31.1
CDLM
43.2
34.6
FastDoe Leg)))xxx((rrpple
32.5
8.3
Dost Leeeoeeeeegeggpeeeeeee (ezr((her))
32.8
9.4
DastDoc Leg-)BERT
32.6
9.4
FastDoel L RaBERre
42.4
32.7
FastDoc Leg.) RaBERTa(hier,
42
32.3
Fost Leg.)RoBERT..
44.8
34.6
Table 4: Foc(Doo(Leg.) and its variants baselines in the Contract Review task on CUAD (AUPR Area Under Precision-Recall Curve).
CUAD exhibits class imbalance, rendering AUPR (Area Under Precision-Recall) and Precision@80% Re- call as suitable metrics. Furthermore, AUPR effectively encapsulates model performance across various confidence thresholds. From Table 4 we infer (1) All FastDoe(Legy Legboe((eg)))R) variants perform better than BASE, and FastDoc(Leg. (eoo((Leg))nn RoBETT performs better than RoBERTABASE and Legal- Regal-RoBERTaa--- BASE (2) FastDoc([Leg [ ROBERTa performs better than CDLM, even though CDLM has a long sequence length of 4096 (3) Dast Leg.) RoBERTa gives the best Precision@80% Recall, and second-best AUPR, even though it uses negligible domain-specific pre-training data compared to baselines.
Summary of the Experiments and Results. Note that although FastDoe uses document information, it does not use the information derived from MLM during continual pre-training that many other conventional domain-specifie baselines use. Therefore, maintain that approach is both equitable and innovative

---

Published in Transactions on Machine Learning Research (05/2024)
when compared to the baselines. We observe that across various domains and different types of generalization, DastDoe typically outperforms (albeit modestly) the baselines, and in cases where it falls short, the difference is marginal. This happens despite using much less compute which is elaborated in Section
5.4 Utility of the pre-training losses:
Examples
We looked into the datasets and gauged the impact of the two pre-training losses in Fast.Doc's performance. Specifically, we took cases where FastDoc has produced better results than well-performing baseline and chose some representative examples to present. Table S shows examples from each domain where triplet and hierarchical losses are beneficial, along with probable reasons. We can see that domain-specific knowledge present in the metadata and taxonomy helps FastDoc in performing well on domain-specific downstream tasks.
Dataset
Triplet Loss is beneficial
Hierarchical Loss is beneficial
Cus.
$10
QA
Q. How I enable the accidental touch pro- toction ?
Q. I need the registered fingerprint list. Where I find this?
(QA)
Reasons
"socidental touch" benefits from triplets hav- ing anchor and positive Touch- based device F-Mamals,, and negative as E-Manual of
"fingerprint" benefits from multiple hierarchies with "Biometric Monitors".
device without touch screen.
Sci.
SciCite
A primary benefit of these models is the inclu- variahility in model parameters (Parnell 2010). Output "background"
The SVR be considered as a novel train- ing technique; the following section presents a concise introduction the SVR [33, 35, 38). Output "background"
(Text
Classifica-
tion)
Reasons
Using triplets with anchor and positive belong- ing to "Machine Learuing" help in classifying the text as "background', as "model param- eters' is a common term used in "Machine Learning' papers.
Although the first sentence could lead to the inference that "SVR" is a new method, other papers belonging to the hierarehy "Computer Science - Machine Learning" would suggest that "SVR' exists already.
Leg.
CUAD
..ta make or have made the Products anywhere in the world for import or sale the Field in
.such commercial crops will be interplanted as agriculture and forestry as well as medici- nal materials;..
(Clause
Extrac-
the Territory in each case,..
tion)
Reasons
The triplet on the concepts of "import policy' and "sale" in the anchor and positive is benefi-
the hierarchy " "Agriculture -> Products subject to market organisation' helps here.
cial.
Table 5: Samples from each domain, where Triplet Loss and Hierarchical Loss are beneficial. Note that we add outputs for the SciCite Text Classification Task for more clarity.
6 Pre-training Compute of FastDoc relative to the baselines
Compute (in GPU-hours)
Domain
Model
EManualsBERT
576
EManualsR.DERT.
980
DeCLUTR
370
Customer
ConSERT
40
Support
SPECTER
600
FastDDoc ShattoocChus.)gee (Cus.)
0.58
FastDoc (Croo)CCCSS))) ( RoBERTa
0.75
Scientific
SCIBERT
7680
Domain
Fast.Doc Fust.Doc(Sci)DFFT
1.7
RoBERTaRASE:
Legal
710
Contracts Pre-training
Domain
Fast LaatDDooTee..... Re..RR
1.49
Table 6: Pre-training Compute of FastDoc vs. baselines
10

---

Published in Transactions on Machine Learning Research (05/2024)
We compare the compute (in terms of GPU-hours GPUs needed multiplied by the number of hours) for pre-training Das with baselines. NVIDIA GeForce GTX 1080 Ti GPUs are used for pre-training.
Baselines in Customer Support and Legal Domains are continually pre-trained on the domain, while the ScIBERT baseline in Scientific Domain is pre-trained from scratch.
Customer Support: Table shows that FastDoe (aaDDDoCCus, (Cus.) and FastDoc (CDDDooCCu.))BBBBEE use roughly 1,000 time and 1,300 times less compute compared to EManualsEERT and EMamualsR respectively, and require significantly less compute than all the baselines. It actually takes less than 1 GPU-hour for pre-training Doc. ConSERT is the closest baseline in terms of compute time, as its inputs are a limited number of sentence pairs, unlike a huge number of spans in DeCLUTR, a large number of triplets in SPECTER, and several masked sentences in EManualSEERT and EMMmualSR.AERTT Legal Domain: FastDoc(Leg) Leg.) needs around 480 times less compute than continual pre-training of RoBERTa- BASE on contracts as in Henddyyck et al. (2021).
Scientific Domain: Fast.Doc(Sci.)) BERT needs around 4, 520 times less compute than ScIBERT The decrease in compute of FastDoc compared to the domsin-specific baselines is much more compared to Cus- tomer Support and Legal Domains, as SCIBERT is pre-trained from scratch. FastDoc(Sel) (Sci.))BRe) BERT continu- ally pre-trains BERT on Scientific Domain, and its downstream task performance and domain-specific compute compared to SCIBERT shows that domain-specific pre-training from scratch is not necessary.
Thus, these experiments demonstrate the remarkable efficiency of the proposed pre-training paradigm, as well as the choice of the pre-training architecture used in FastDoc.
7 Analysis and Ablations
In this section, we report the following analysis and ablations - (1) Catastrophic Forgetting when evaluating FastDoc in open-domain (2) absence of document supervision in the domain of interest (3) Effect of using larger backbone model for Dast (4) Reasons behind FastDoc working the way it does. Also, apply
Carameter-F.fiiient training on Fast.Do as an ablation in Section G.2 of Appendin, which gives very poor downstream task results and is not beneficial from a compute perspective as well.
TASK
CoLA
MRPC
QQP
MNLI
QNLI
RTE
METRIC
Matthews
CC
Fl-score
Pearson CC
Spearman CC
Fl-score
Acc.
RORERTARASE
63.71
94.15
92.71
80.71
90.91
90.66
89.1
91.84
87.24
92.26
80.14
62.57
94.27
93.1
90.44
90.98
90.66
80.08
91.84
87.22
92.62
79.06
FostDoc MaetDoe(Cus.))e (naaa)a
(-1.14)
(+0.12)
(40,39)
(+0.73)
(+-0.07
(0)
(-0.02)
(0)
(-0.02)
(+0.336)
(-1.88)
51.82
91.97
91.42
87.30
88.4
$5.30
36.85
91.56
86.16
91.34
EMunnnsq.nERT.
(-11.80)
(-2.18)
(-1.29)
(-1.22)
(-2.11)
(-2.3)
(-0.45)
(-0.29)
(-2.09)
(-0.92)
(-9.74)
Table 7: Dev. set results on GLUE Benchmark (CC Correlation Co-efficient, Acc. Accuracy)
7.1
Catastrophic Forgetting in open-domain
Recent works show that continual in-domain pre-training of transformers leads to significant. performance drop when fine-tuned on open-domain datasets (Arumae al. 2020 Gururangan et al. 2020 resulting in Catastrophic Forgetting (CF). Such works start with opeu-domain model (e.g. BERT (EIRTRRoBETaa) and perform open-domain benchmark tasks (e.g. GLUE). Then, they consider a model pre-trained continually on a specific domain (e.g. BioBERT) and re-assess performance on those tasks. Decrease in performance of domain-specific model compared to the open-domain model determines degree of catastrophic forgetting.
Similarly, we fine-tune RoBERTaEASE (pre-trained on open-domain corpora), FastDoc (iliDDoeCCu.))aae EMamaalSt.SERTT from customer support on the datasets of the (open-domain) GLUE Wang et al. 2018)
(Cs..) and
11 Pustt uses 33.3x less documents compared EMamualsERRT during pre-training. Additionally, FastDoc(Cou ) takes sentence embeddings as inputs, while EManualsEERT takes in token embeddings. There 37.3 tokens per sentence in the pro-training corpus, meaning that there 37.3x losser samples for FastDoe PustDoc(Cus.)pgn w.r,t EMamualsEEnT for the same text, reducing compute further from 33.3x 1000x (33.3 37 is z 1000)
12 According to Beltagy et (2019) it takes minimum of 40 days on 8 GPUs (elaborated Section of Appendinn)
11

---

Published in Transactions on Machine Learning Research (05/2024)
Figure 2: Relative change (in Logio Scale) in the Ll-norm of different types of parameters during pre- ave-training
via MLM vs. FastDoc.
0.0
Parameter L1 Norm Changes (Log Scale)
-1.0
-1.5
-2.-
query
key
-3.0
value
dense
all
-40
EManualsR.BERT
FastDoc
benchmark, and the results shown in Table 7 The hyperparameters used are mentioned in Section G.1 of Appendix.
performs better than RoBERTaBASE in 4 out of 8 tasks (although the improvement minor), even after contimual pro-training on E-Manuals, while the drop performance the other 4 lasks is negligible. The performance improvement in tasks that require predicting relation between sentence pairs, like STS, QNLI, MRPC, could be attributed to the Contrastive Learning Objective when pre-training FastDoc (b). EMMaualSRoBERTA performs considerably worse compared to RoBERTaBASE all tasks, suggesting that MLM is not robust against domain change. The possible reason is that it requires only small fraction of
We observe that (a). CrttDoocCCs.RReBBBB
behind the superior performance of Fast (Cus.) pre-training data compared to what is used by domain-specife baselines such as E'MamaalsoBBERT: hence making only small changes the parameter space that helps retain open-domain knowledge while learning essential domain-specific knowledge. We perform an experiment to test the proposition and plot the relative change L1-norm different types of paramcters such as attention query, key, value matricos, and dense MLP parameters (similar to Wu al. (2022)) during pre-training via MLM vs. Fast shown in Figure 2 We observe that the relative change of parameters in Dast is about 100 times less compared to MLM.
7.2 Absence of document level information
Model RoBERTaBASE
F1
HA FI@I
HA F1@5
16.46
31.89
42.4
17.52
33.94
44.96
PastDoc(Cus.))y ROBERT RRR.EEE (7 hier. levels)
(+6.44%)
(+6.43%)
(+6.04%)
15.39
29.83
44.35
(w/o est. meta., tax., 7 hier. levels)
(-6.5%)
(-6.66%)
(+4.6%)
Fast (Cs.)
18.01
34.89
47.53
(w/o est. meta., tax., 15 hier. levels)
(+9.42%)
(+9.41%)
(+12.1%)
Table 8: Results on TechQA Dataset in Customer Support Domain with and without established domain- specific document metadata and taxonomy
A pre-requisite of FastDoc has been the availability of document metadata and taxonomy. In this experi- ment, go beyond that and derive document similarity via similarity based the ROUGE-L. score among
12

---

Published in Transactions on Machine Learning Research (05/2024)
documents, followed by creating custom taxonomy of document category hierarchies using Hierarchical Topic Modeling (Grootendorsi, 2022). Table shows results on the Customer Support (see other domains' results in Section G.3 of Appendiz), and we can see that gives comparable performance when considering same number of hicrarchical levels as FastDoc. However, since the taxonomy derived using topic mod- eling, we are here not constrained by the number of hierarchies. We notice that the performance improves when a larger number of hierarchical levels are used, showing great potential for adapting Fast.Doc any domain of interest. However, note that even though one can devise a (unsupervised) way extract triplets and document hierarchies, it is much more efficient to metadata and taxonomy if and when available, there is some time and CPU involved in deriving content-similarity-e metrics like ROUGE-L score due to the large size of the documents.
7.3 Effect of using larger backbone model for Fast
Model
F1
HA FI
HA F182
CastDocCCus..Jeeee
17.52
33.94
44.96
18.48
35.8
47.8
FastDoe(Cuss)ua
(+5.11%)
(+5.118%)
(++6.32%)
Table 9: Results on TechQA Datasct in Customer Support Domain (RI. ROBERTa-LARGE)
Field
Task
Dataset
FustDoc(Sci))nFir
FastDoe(Sei) Jar.
BC5CDR
88.45 (+0.73%%
NER
JNLPBA
75.81
76.53 (+0.91%)
BIO
NCBI-D
84.33
86.18 (+2.199%
REL
ChemProt
80.48
84 (+ (+33%)%
CS
REL. SCERC
78.95
80.26 (+ (-1.6)%)
Multi
CLS
ScCCite
83.59
85.76 (+2.6%%
Table 10: Results tasks presented in Beltagy et al. 2019 (BL BERT-LARGE)
Precision@
Model
AUPR
80% Recall
FastDoc (eest.o((Leg)) ROBERT
44.8
34.6
45.3
39.5
LasgDoc(Leg.)na
(+1.12%)
(+14.16%)
Table 11: Results CUAD Dataset in Legal Domain (RL RoBERTa-LARGE
Tables g 10 and 11 compare the impact of using a larger backbone compared to the one used in the
proposed Dast (c.g. RoBERTa-LARGE vs. RoBERTa-BASE, BERT-LARGE vs. BERT-BASE). From the results, we can see that using a larger model as a backbone further improves results due to an increased number of trainable parameters.
7.4 Analysis of the interoperability of embeddings
FastDoc shows that using input sentence embeddings during pre-training helps when using token embed- ding inputs during fine-tuning, as evident from the potent downstream task performance. We analyze this interoperability of embeddings by answering the following research questions (observations and exper- iments elaborated in Section G.4 of Appendiz) (a). How does FastDoc learn local context? Similar documents have very-similar local (paragraph-level) contexts, suggesting that, using document-level supervision during pre-training implicitly learns local context. Also, in an experiment, we randomly sample 500 sentences from each of the domains. For each sentence, we mask a random token and calculate the change in its prediction probability on masking other tokens the sentence. Spcarman Correlation of this change between FastDoc and a domain-specific model pre-trained using MLM is moderately high for all domains, showing that local context is learned by Dost to reasonable extent. (b). Are relative
13

---

Published in Transactions on Machine Learning Research (05/2024)
representations preserved across the two embedding spaces? Independent of whether inputs are sentence or token embeddings, documents are clustered in a similar manner across the two representation spaces, hence, relative representations are preserved.
8 Prior Art
Representation Learning using self-supervised learning methods: In recent times, downstream tasks in NLP use representation learning techniques where transformers are pre-trained on large text corpora using
self-supervised learning methods like NSP (Devlin et al.) 2019 MLM (Devlin et al.) 2019 Liu et al., 2019)
contrastive learning 4Giorgi et al., 2021
Yan et al. 2021 Wang et al. 2021 Cohan et al.) 2020 etc.
before fine-tuning on downstream tasks. There are models pre-trained on domain-specific corpora such
etc. E-Manuals (Nandy al. 2021 legal lexts
(Chalkidis et al. 2020), bio-medical documents (Lee et al. 2020
Supervised Pre-training: Feng et al. (2022) proposes supervised pre-training on Leave- (cave-One-Out KNN that improves Transfer to rownst tasks. CLMSM
Nandy el al. 2023) uses recipe meladata as supervision 2023) leverages labeled data from a corpus across 11 tasks for
signal for pre-training. MVP (Tang et al. pro-training, by unifying the data into tcxt-to-text format. The paper also states that unsupervised pre- training likely incorporates noise that affects the downstream performance, making supervised pre-training a better alternative. CLIP (Radford et al. 2021) utilizes the pre-training task of predicting which caption goes with which image (natural language supervision), which is an efficient way to learn image representations.
Incorporating hierarchical information for enhancing representations: Hierarchical information in the form of taxonomy and ontology has been used by some works to enhance learned representations. Barkan et al. 2021 introduces a Variational Bayes entity representation model that leverages additional hierarchical and relational information. [Barkan et al. (2020) also uses a similar Bayesian approach to produce better representations, especially for rare words.
Intra-document Contrastive Learning: DeCLUTR (Giorgi et al. 2021 uses DistilRo
(Sanh et. al. 2019) encoder. Spans overlapping or subsuming each other are considered similar inputs, and other spans are considered as dissimilar imputs. InfoNCE Loss Function (Sohn 2016) brings representations of similar spans closer and pushes reprosentations of dissimilar spans farther away. ConSERT (Yan et al., 2021 also uses contrastive loss, but performs sentence augmentation using adversarial attack (Kurakin et al. 2016), token shuffling, etc. It considers a sentence and its augmented counterpart to be similar, and any other sentence pair dissimilar. CLINE (Wang et al., 2021) creates similar and dissimilar samples from sentence by replacing some word(s) with their synonyms and antonyms using WordNet (Miller 1995 and then uses contrastive loss.
Inter-document contrastive learning: SPECTER (Cohan et al. 2020 uses a triplet margin loss to pull similar documents closer to each other, and dissimilar ones are pushed away. The document representations are obtained using transformer encoder. However, their encoder only able to encode a maximum of 512 tokens of document. SDR (Ginzburg et al., 2021) uses a self-supervised method by combining MLM loss and Contrastive Loss to learn document similarity. LinkBERT (Yasunaga et al. 2022) adds Doeument Relation Prediction Objective to MLM during pre-training, where the task is to predict whether two segments are contiguous, random, or from linked documents. CDLM (Caciularu et al. (021) leverages document-level supervision by applying MLM over a set of related documents using Longformer (Beltagy et al. 2020). These works are in line with our work, but they are unable to tackle the important technical challenges of large input size and scalability and in turn, suffer from the problems of limited input size and high pre-training compute.
9 Summary and Conclusion
Recent studies have repcatedly stressed the importance of domain-spocific pretraining but also pointed to the costly and claborate operation that must be undertaken to achieve reasonable performance. This paper shows that leveraging 1) document-level semantics, and 2) interoperability of input sentence embeddings (during
14

---

Published in Transactions on Machine Learning Research (05/2024)
pre-training) and token embeddings (during fine-tuning), substantially reduces the compute requirements for dooaai--ppccfii pre-training by at least 500 times, even while achieving better results on 6 different downstreamn tasks and 9 different datasets. The frugal pretraining technique has an important side-effect, it shows negligible catastrophic foryetting on the open-domain GLUE Benchmark. We also demonstrate that the existence of well-defined meladata and laxonomy not. mandatory; Past.Dcc performs effectively when discovering such metadata and taxonomy through unsupervised methods, illustrating potential for future application across various domains.
Limitations
FastDoc is robust to wide document similarity range across several domains. However, perfor- mance in presence of high levels of noise in metadata is not guaranteed and further investigation is required to characterize it.
Applicability of the proposed model to decodor-only and encoder-dccoder modcls: FastDoc can be extended to decoder-only models like GPT-2 Radford al. (2019), and encoder-decoder models like BART-BASE Lewis et al. (2020). We apply FastDoc using GPT-2 backbone (referred to FastDocGrr-2) and the BART-BASE encoder as backbonc (referred to as FastDoc Downstream task is dialogue summarization (i.e., a text generation task) on TweetSumm Dataset Feigenblat et al. (2021] in the Customer Support Domain. We compare it with GPT-2 and BART- BASE
Model
ROUGE-1
ROUGE-2
ROUGE-L
GPT-2 FastDocart-2
0.151
0.066
0.119
0.134
0.058
0.104
Table 12: Results of FastDocgrr-2 vs. GPT-2 on TweetSumm Dataset
Model
ROUGE-1
ROUGE-2
ROUGE-L
BART-BASE
0.523
0.314
0.472
FastDocBBRTT BAAtDDoCBART-BARE BASE
0.521
0.315
0.473
Table 13: Results of Fast BASE vs. BART-BASE on TweetSumm Dataset
Tables 12 and show that FastDoc gives poor results when using decoder-only model as the backbone, and gives negligible improvement when using the encoder of an encoder-decoder model as the backbone. Improvement in results needs changing the architecture and document supervision objectives used in FastDoc to adapt to decoder and encoder-decoder models end-to-end. One way to adapt FastDoc to docoder model backbones is hicrarchical decoding (like hicrarchical encoding in FastDoc) in stages decoding special, representative sentence lokens, which are then used to decode subword tokens. This potential future work.
Applicability of the proposed model when downstream tasks are generation tasks: Tables 12 and 13 show that FastDocGPT-2 and FastDocBART- task. Improvement results could be attained in a manner mentioned above.
do not perform well on text generation
Broader Impact Statement
The proposed methodology is, in general, applicable to any domain. Specifically, it can potentially be applied to user- geer-geneaated text available on the web and is likely to learn patterns associated with exposure bias. This needs be taken into considcration before applying this model user-generated text crawled from the web. Further, like many other pre- trained language models, interpretability associated with the output is rather limited, hence users should use the output carefully.
15

---

Published in Transactions on Machine Learning Research (05/2024)
References
Kristjan Arumae, Qing Sun, and Parminder Bhatia. An empirical investigation towards efficient multi- domain language model pre-training. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP)) pp. 1854-4864, 2020.
Oren Barkan, Idan Rejwan, Avi Caciularu, and Noam Koenigstein. Bayesian hierarchical words rep- resentation learning. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pp. 3871-3877, Online, July 2020. Association for Computational Linguisties. doi: URL htttt::://alaathollllooooo
10.18653/v1 http:///1/2220..mmmama...
wwwgg//222..02 acl-
Oren Barkan, Avi Caciularu, Idan Rejwan, Ori Katz, Jonathan Weill, Itzik Malkiel, and Noam Koenigstein. Representation Learning via Variational Bayesian Networks, pp. 78 88. Association for Computing Ma- chinery, New York, NY, USA, 2021. ISBN 9781450384469. URL https://doi.org//// http////...1//////1111 1145/3459637. 3482363
Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3615 3620, 2019.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 2020.
Florian Borchert, Christina Lohr, Luise Modersohn, Thomas Langer, Markus Follmann, Jan Philipp Sachs, Udo Hahn, and Matthieu-P Schapranow. Ggpone: A corpus of german medical text with rich metadata based on clinical practice guidelines. In Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis, pp. 38 48, 2020.
Florian Borchert, Christina Lohr, Luise Modersohn, Jonas Witt, Thomas Langer, Markus Follmann, Matthias Gietzelt, Bert Arnrich, Udo Hahn, and Matthicu-P Schapranow. Ggponc 2.0-the german clinical guideline corpus for oncology: Curation workflow, annotation policy, baseline ner taggers. In Proceedings of the Thirleenth Language Resources and Evaluation Conference, pp. 3650 16600-360,--0 2022.
Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E Peters, Aric Cattan, and Ido Dagan. Cdlm: Cross- document language modeling. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2648 2662, 2021.
Richard Caruana. Multitask learning: A knowledge-based source of inductive bias. In Proceedings of the Tenth International Conference on Machine Learning, pp. 41-48. Morgan Kaufmann, 1993.
Vittorio Castelli, Rishav Chakravarti, Saswati Dana, Anthony Ferritto, Radu Florian, Martin Franz, Dinesh Garg, Dinesh Khandelwal, J. Scott McCarley, Mike McCawley, Mohamed Nasr, Lin Pan, Cezar Pendus, John F. Pitrelli, Saurabh Pujar, Salim Roukos, Andrzej Sakrajda, Avirup Sil, Rosario Uceda-Sosa, Todd Ward, and Rong Zhang. The techqa dataset. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 1269-1278. Association for Computational Linguistics, 2020. doi: 10.18653/vv /v1 100p3//v1122200000 2/11//20.acllcccm www.mmai..nn1.. 117. URL
11w53//11/222 acw.mmam......mmm...... 117
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, and lon Androntsopoulos. Large-scale multi- label text classification on EU legislation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6314-6322, Florence, Italy, 2019. Association for Computational Lin- 11116553/11/11-16B3 1616- URL https://www.aalwveet httpa/////.natooog//////----/
guistics. doi: 10.18653/ v1
Iaas Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Alctras, and Ion Androutsopoulos. Legal-bert: The muppets straight out of law school. In Findings of the Association for Computational Linguisties: EMNLP 2020, pp. 2898 2904, 2020.
16

---

Published in Transactions on Machine Learning Research (05/2024)
Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Läu, and Qun Liu. bert2BERT: Towards reusable pretrained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2134 2148, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1 acl- long. 151. URL https://aclaanthologg http://oo......rr//....alaaa---- 151.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PML.R, 2020.
Elizabeth Clark, Asli Celikyilmaz, and Noah A Smith. Sentence mover's similarity: Automatic evaluation for multi-sentence texts. In Proceedings of the 57th Annual Meeting of the Associalion for Computational Linguistics, pp. 2748 2760, 2019.
Arman Cohan, Walcod Ammar, Madclcine van Zuylen, and Field Cady. Structural scaffolds for citation intent classification in scientific publications. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 3586-3596, 2019.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. SPECTER: Document-level Representation Learning using Citation-informed Transformers. In ACL, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- uine (Long and Short Papers), pp. 4171 4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.11863//11 ///p//111111////11111 1423. URL http:ssss/accaattoooooooo/////// org/N19-1/2
Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. Nebi disease corpus: a resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1 10, 2014.
Guy Feigenblat, Chulaka Gunasekara, Benjamin Sznajder, Sachindra Joshi, David Konopnicki, and Ranit Aharonov. TWEETSUMM- dialog summarization dataset for customer service. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 245-260, Punta Cana, Dominican Republic, November findinng-emmllp.24 URL https:
2021. Association for Computational Linguistics. doi: //aclanthology.org//28 findings-emnlp...
Yutong Feng, Jianwen Jiang, Minggian Tang, Rong Jin, and Yue Gao. Rethinking supervised pre-training for better downstream transferring, In International Conference on Learning Representations, 2022. URL
Saibo Geng, Rémi Lebret, and Karl Aberer. Legal transformer models may not always help. arxiv preprinl 0rXiv:2109.06862 2021.
Dvir Ginzburg, Ttzik Malkiel, Oren Barkan, Avi Caciularu, and Noam Koenigstein. Self-supervised doc- ument similarity ranking via contextualized language models and hierarchical inference. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3088 0888-008,, Online, Au- gust 2021. Association for Computational Linguistios. doi: 10.18653/v1 /2021 fin....ining--cl.225 URL https:://aclantooloo..oo/// wrgp:/21..r/20222.. http://21.....nnnnnseeeeaae findings-acl.272
John M. Giorgi, Osvald Nitski, Bo Wang, and Gary D. Bader. Declutr: Deep contrastive learning for unsupervised textual representations. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI//JCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 979-895, Association for Computational Linguistics, 2021. doi: 10. 10.18653/v1 20/55//vv12211/1 /202121122222 acl-long.72 72. URL http:::tt/:/o...oo/ http://o....rrr........6//////// acl-://.o...........o........
17

---

Published in Transactions on Machine Learning Research (05/2024)
Maarten Grootendorst. Bertopic: Neural topic modeling with class-based tf-idf procedure. arXiv preprint arXiv:2203.057 2022.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models domains and tasks. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342-8360, On- line, July 2020. Association for Computational Linguistics. doi:
URL
https://aacLantollogy www.hhologg...g//22222222 wtt./222..clmmmaannnn.m.. acl-main. 740
Dan Hendryeks, Collin Burns, Anya Chen, and Spencer Ball. Cuad: An expert-annolated nip dataset for legal contract review. NewrIPS, 2021.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openrevient rett:forrumm/dnndeeeeeerrt
Allen H Huang, Hui Wang, and Yi Yang. Finbert: A large language model for extracting information from financial text. Contemporary Accounting Research, 40(2):806-50) 841, 2023.
Giannis Karamanolakis, Jun Ma, and Xin Luna Dong. Txtract: Taxonomy-awart knowledge extraction for thousands of product categories. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8489 8502, 2020.
Jin-Dong Kim, Tomoko Ohta, Yoshimasa Tsuruoka, Yuka Tateisi, and Nigel Collier. Introduction to the bio- entity recognition task at julpba. In Proceedings of the international joint workshop on natural language processing in biomedicine and ils applications, pp. 70 70-75. Citeseer, 2004.
Jens Kringelum, Sonny Kim Kjaerulff, Søren Brunak, Ole Lund, Tudor I Oprea, and Olivier Tabourcau. Chemprot-3.0: global chemical biology diseases mapping. Database: The Journal of Biological Databases and Curation, 2016, 2016.
Alexey Kurakin, lan Goodfellow, Samy Bengio, et al. Adversarial examples in the physical world, 2016.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervieed learning of language representations. In International Conference on Learning Represenlations, 2019.
Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents. In International conference machine learning, pp. 1188-1198 PMLR, 2014.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Doughyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinfor- matics, 36(4):1234 1240, 2020.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising toweennce---ssqquee pre-training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020 acl-main. 703. URL https: //t:c/accllac http://allatthoooooorrrr//////// www.o/22........aia.n.... acl-main. 703
Jiao Li, Yueping Sun, Robin J. Johnson, Danicla Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. BioCreative CDR task corpus: resource for chemical discase relation extraction. Database, 2016, 05 2016. ISSN 1758-0463. doi: baw06s.
20.1093/databaase/baaa baw068. URL
Chin- Yew Lin. ROUGE: A package for automatic evaluation of summarics. In Tert Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguisties. URL https:
wowellantholoog..rrg//ww..
18

---

Published in Transactions on Machine Learning Research (05/2024)
Carolyn E Lipscomb. Medical subject headings (mesh). Bulletin of the Medical Librory Association, 88(3): 265, 2000.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeetteemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXi::1907.110 11692, 2019.
Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Represenlations, 2018.
Eneldo Loza Mencia, Johannes Fürnkranz, and loza. Eur-lex dataset, 2010. URL https://tudatallib. ulb.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 3219 00--22222222 2018.
Ttzik Malkiel, Dvir Ginzburg, Oren Barkan, Avi Caciularu, Yoni Weill, and Noam Koenigstein. Metricbert: Text representation learning via self-super triplet training. In ICASSP 2022-2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2022.
Danicle Margiotta, Danilo Croce, Marco Rotoloni, Barbara Cacciamani, and Roberto Basili. Knowledge- based neural pre-training for intelligent document management. In Stefania Bandini, Francesca Gasparini, Viviana Mascardi, Matteo Palmonari, and Giuseppe Vizzari (eds.), AIJAA 2021 Advances in Artificial Intelligence, pp. 564 579, Cham, 2022. Springer International Publishing. ISBN 978-3-031-08421-88
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In Yoshua Bengio and Yann LeCun (eds.), 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013. URL http:://axii...../////// http://aawxxvv.../aaaaaa htgg//b///ia 1101 3781
George A. Miller. Wordnet: A lexical database for english. Commun. ACM, 38(11):39 41, nov 1995. ISSN 0001-0782. doi: 21.145/219717.2197433 URL htttps://dol http://doooorrg///// 1145/119717.219744
Abhilash Nandy, Sounya Sharma, Shubham Maddhashiya, Kapil Sachdeva, Pawan Goyal, and Nlloy Gan guly. Question answering over electronic devices: A new benchmark dataset and multi-task learning based QA framework. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 4600 4609, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. 20/2 findings-emnlp. 392.
URL
Abhilash Nandy, Manav Kapadnis, Pawan Goyal, and Niloy Ganguly. CLMSM: A multi-task learning framework for pre-training on procedural text. In Houda Bouamor, Juan Pino, and Kalika Bali (ods.), Findings of the Association for Compulational Linguisties: EMNLP 2023, pp. 8793 8806, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. findaa..iiinnn-eee modings-emmlp.55) URL https://aaclanthooogy.yooo wrg.e/23..g2222./. findings-emnlp.
Rodrigo Nogucira, Jimmy Lin, and AI Epistemic. From doc2query to doctttttquery Online preprint, 2019.
Hyun Oh Song, Yu Xiang, Stefanic Jegelka, and Silvio Savaresc. Deep metric lcarning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 4004-4012, 2016.
Malte Ostendorff, Nils Rethmeier, Isabelle Augenstein, Bela Gipp, and Georg Rehm. Neighborhood contrastive learning for scientific document representations with citation embeddings. ar preprint arXiv:2202.06677 2022.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
19

---

Published in Transactions on Machine Learning Research (05/2024)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2883-2392, 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang, Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784 789, 2018.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)), pp. 3982-3992, 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1ix::1910001100 108, 2019.
Carlos N Silla and Alex Freitas. A survey of hierarchical classification across different application domains. Data Mining and Knowledge Discovery, 22(1):31 72, 2011.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29:1857-1865, 2016.
Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. MVP: Multi-task supervised pre-training for natural language generation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguisties: ACL 2023, pp. 8758 8794, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1 12023 windings--al..... URL https: //aclanthollog.org///2 wwwp:/oyy..r///223... findings-acl. 558
Tan Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 4593 4601, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10.18653/v1/P19 1452. URL https: pooclanthology.orr//113.
Betty van Aken, Benjamin Winter, Alexander Löser, and Felix A. Gers. How does bert answer questions? layer-wise analysis of transformer representations. In Proceedings of the 28th ACM International Con- ference on Information and Knowledge Monagement, CIKM '19, pp. 1823 1832, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450369763. doi: 3344/333578443338222 URL https://doi.org// 1111111//33884.336588833
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding, In Proceedings of the 2018 EMNLP Workshop Blackbox.NLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353- 355, Brussels, Belgium,, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/
W18-5446. URL https:://cllanhhoogyy wwt.oo///ooo//////6/g
Dong Wang, Ning Ding, Piji Li, and Haitao Zheng. Cline: Contrastive learning with semantic negative examples for natural language understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 2332-2342. 2021.
Kilian Q Weinberger and Lawrence K Saul. Distance metric learning for large margin nearest neighbor classification. Journal of machine learning research, 10(2), 2009.
20

---

Published in Transactions on Machine Learning Research (05/2024)
Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. NoisyTune: A little noise can help you finetune pretrained language models better. In Proceedings of the 60th Annual Meeting of the Associalion for Computational Linguisties (Volume 2: Short Papers), pp. 680 685, Dublin, Ireland, May 2022, Association for Computational Linguistics. doi: 10.18653/ v22////1/222.aa//// ac--222.c-- URL httpss//aclanttooooy.ooom
1022.acl-short./1
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. Consert: A con- trastive framework for self-supervised sentence representation transfer. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro- cessing, ACL/IJCNLP 2021, (Volume Long Papers), Virtual Event, August 1-6, 2021, pp. 5065- 5075. Association for Computational Linguistics, 2021. doi: 10.18653/ 21//3//11222211/ 2202..acl-oong3333 https://doi.org// 18653vv1/2021 act.//2222...llooggg... 393
URL
Michihiro Yasunaga, Jure Leskovee, and Perey Liang. Linkbert: Pretraining language models with document links. In Proceedings of the 60th Arnual Meeting of the Associalion for Compulational Linguistics (Volume 1: Long Papers), pp. 8003 8016, 2022.
Xingxing Zhang, Furu Wei, and Ming Zhou. HIBERT: Document level pre-training of hierarchical bidi- rectional transformers for document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5059 5069, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19 1499. URL http::://aclaatttooooooooogr/gg
Appendix
The Appendix is organized in the same sectional format as the main paper. The additional material of a section is put in the corresponding section of the Appendix so that it becomes easier for the reader to find the relevant information. Some sections and subsections may not have supplementary material so only their name mentioned. The page numbers are continuation from the Main Paper's end page number.
A
Introduction
B FastDoc Framework
Figure 8 shows the percentage of documents encoded entirely by RoBERTa-BASE encoder when the input is 512 tokens vs. 512 sentences.
Figure shows a dotailed overview of the FastDoc Framework.
C Pre-training Setup
C.1 Pre-training in the Customer Support Domain
Table 14 shows examples of E-manual product categories and the hierarchies assigned to them with the help of the GPrT. We can see that more similar products tend to have more similar hierarchies.
Mapping E-Manual product category to hierarchy: It may happen that the product category of the E-Manual does not have an exact match with any leaf category in the GPrT. In that case, we map it to that leaf category, where cosine similarity of mean word embeddings (Mikolov et al.) 2013) of the product category and the hierarchy's last two entities is the highest. This choice gives qualitatively good mapping when measured using human evaluation.
Triplet Count in Customer Support Note that we do not double the number of triplets by swapping anchor and positive in case of Customer Support, as doing so leads very similar results, while taking double the compute. This is because in case of Customer Support, F-Manuals written based the product
21

---

Published in Transactions on Machine Learning Research (05/2024)
100
<#512 tokens
512 sentences
Percentage documents covered
0
Customer
Scientific Domain
Legal
Support
Domain
Figure Percentage of documents encoded entirely by RoBERTa-BASE encoder when the input is 512 tokens vs. 512 sentences (The red bar of "Scientific Domain" has a negligible height, and hence, not visible.)
Total Loss
Hier. Classfn. Loss
Hier. Classfn. Loss +
Triplet Loss
+ Hier. Classfn. Loss
Anchor Doc.
Positive Doc.
Negative Doc.
Representation
Representation
Representation
hard parameter sharing
hard parameter sharing
Transformer Encoder
Transformer Encoder
Transformer Encoder
Anchor Sentence Embeddings
Positive Sentence Embeddings
Negative sentence embeddings
Pre-Trained Sentence Transformer (Frozen)
Anchor Document
Positive Document
Negative Document
Figure 4: Depiction of Doct Anchor, Positive, and Negative Documents are encoded using Sentence Transformer, followed by transformer encoder, to give document representations. A combination of Triplet and Hierarchical Classification Losses is used to get the Total Loss
category metadata, while the metadata for Scientific and Legal Domains assigned after the document is
22

---

Published in Transactions on Machine Learning Research (05/2024)
Product
Hierarchical categories assigned to the
Category
product
blu-ray
Electronics > Audio > Audio Accessories
player
MP3 Player Accessories MP3 Player &
Mobile Phone Accessory Sets
stereo
Electronics Audio >
Audio Players &
equalizer
Recordens Stereo Systems
laptop
Electronics
Electronics Accessories
docking
Computer Accessories Laptop Docking Stations
station
hot
bever-
Home & Garden
Kitchen & Dining
age maker
Kitchen Appliance Accessories > Coffee
Maker & Espresso Machine Accessories > Cof-
fee Maker & Espresso Machine Replacement
Parts
Table 14: Examples of product categorics and their corresponding hicrarchical categories assigned with the help of the Google Product Taxonomy. Similar products have similar hierarchies
written. Thus, the product category Customer Support is much more precise and well-defined compared to primary calegory and overlap EUROVOC Concepts used in Scientific and Legal Domains respectively, thus requiring lesser data.
D Downstream Datasets/Tasks
Datasets used in our work
Table 15 lists all the downstream datasets used in our work, along with their corresponding tasks and domains.
Domain
Task single-span
Dataset
Customer
TechQA
QA
Support
multi-span
S10 QA
QA
Dataset
BC5CDR
NER
JNLPBA
Scientific Domain
NCBI-D
Relation
ChemProt
Classification
SciERC
Text
SciCite
Classification
Contract
Review
Legal
(Span-basod
CUAD
Domain
Clause
Extraction)
Table 15: List the datasets along with their corresponding tasks and domains.
D.1
Customer Support
$10 Question Answering Dataset. The S10 QA Dataset (Nandy et al. 2021 consists of 904 question- answer pairs curated from the Samsung S10 Smartphone E-Manua along with additional information on
1hhttps://btt.yy 1yy/bggg///...
23

---

Published in Transactions on Machine Learning Research (05/2024)
the section of the E-Manual containing the answer. However, the answer might not be continuous span, T.e., the answer may be present in the form of non-contiguous sentences of a section. The tasks of section and answer retrieval are performed. The dataset is divided in the ratio of 7:2:1 into training, validation, and test sets, respectively.
D.2
Scientific Domain
D.3
Legal Domain
The dataset is split 80/20 into train/test, with a small validation set for the preliminary experiments to perform hyperparameter grid search.
Experiments and Results E
Here we report certain extra experiments which could not accommodated the main paper due to want of space. We also report additional ablation analysis on the TechQA Dataset below.
E.1 Customer Support Domain
Details on SPECTER When pre-training on E-Manuals, instead of initializing the en- coder with SciBERT (Beltagy et al. 2019) (as in Cohan et al. (2020)), initialize the model using EManualsBERT (Nandy el al. 2021). sample about the same number of E-Manual triplets stated in Co- han et al. (2020) as inputs (using product category information), and use the first 512 tokens per input E-Manual.
Baselines:
Fine-tuning Setup
Fine-tuning on SQuAD 2.0 (Rajpurkar et al., 2018). SQuAD 2.0 is a span-based open-domain reading comprehension dataset, consisting of 130, 319 training, 11, 873 dev, and 8, 862 test QA pairs. Before fino-tuning on the task-specific dataset, we fino-tune the encoder on the SQuAD 2.0 training set, as it has been shown to improve the performance QA tasks (Castelli et al. 2020 The hyperparameters used are the same mentioned in Rajjurkar et al. 2018
The encoder is fine-tuned on the TechQA Dataset with the same training architecture used when fine-tuning on SQuAD 2.0. Since this is QA task, a question and one of the candidate technotes scparated by a special token is the input. If the technote contains the answer, the larget output is the start and end token of the answer, and is unanswerable otherwise. The hyperparameters used are the ones mentioned in the default implementation of Castelli al. 2020)).
Fine-tuning on TechQA Dataset:
Finc-tuning on S10 QA Datasct: The S10 Datasct is accompanied by 2 sub-tasks (a) Section Rc- trieval given the question and top 10 candidate sections retrieved using BM25 IR Method (Nogueira et al. the task is to find out the section that contains the answer. (b) Answer Retrieval Given a
2019 15
question and the relevant E-Manual section, the task is to retrieve the answer to the question. For section retrieval, a (question, candidate section) pair separated by special tokens ((CLS)) and '[SEP]' in case of BERT (Devlin et al.] 2019) and '
" and '
" case RoBERTa (Liu et al. 2019h) input to the model, and the ground truth is 0/1 depending on whether the section contains the answer or not. Similarly, in the case of answer retrieval, the question paired with sentence of the section containing the answer (separated by special tokens) is the input to the model, and the ground truth is 0/1 depending on whether the sentence is part of the answer or not. In both sub-tasks, the classification token's (+(CLS)) or '
") encoder output is fed to a linear layer (followed by Softmax function) to get a probability value. Fine-tuning on each of the sub-tasks yields separate models which are used during inference time for completion of the respective task.
https: htaps:://gitaee //t.thub..ttttt wom//IM/techhal Apache-2.0 2.0 License
HMM22 TE-IDF used Nandy al. 2021
24

---

Published in Transactions on Machine Learning Research (05/2024)
HA
FI@1
HA
FIQ5
BERT
8.63
16.72
22.52
RoBEITTaBASE
13.98
27.1
43.02
Longformer
15.39
29.82
42
10.1
19.56
29.87
EMamualsRoBERT.
13.62
26.38
38.67
DeCLUTR
12.52
24.26
29.59
ConSERT
10.78
20.88
31.55
SPECTER
0.69
1.34
7.21
Dast.Doc CouuseCCssseeee /ERST (hirr)
9.12
17.68
26.52
Dost (Cus. (URR))ERr(rriplll
10.76
20.84
31.81
DostDoc )
7.8
15.11
24.78
Doet Cus....BBRRT. (hver.)
13.83
26.8
37.84
Dos Cus, (us.)RRBBRR..
12.93
25.06
40.84
Dos Cus..RoBERTE
14.89
28.85
39.04
Table 16: Results for the QA downstream task on the TechQA Dataset, without intermediate SQuAD 2.0 fine-tuning (Values in red/ green indicate if the values are less than/greater than the values got using intermediate SQuAD 2.0 fine-tuning)
For all the fine-tuning experiments on S10 QA Dataset, we use batch size of 16 (except for the pre-trained
DeCLUTR model with DistilRoBERTagAse backbone, where batch size of 32 is used), and train for 4 epochs with an AdamW optimizer (Loshchilov & Hutter 2018) and an initial learning rate of 4 x 10-5, that decays linearly.
Results
Performance on TechQA Datasct
Analyzing impact of fine-tuning on SQuAD 2.0: Table 16 shows the results on the TechQA Dalaset without intermediate fine-tuning on SQuAD 2.0 Dataset. Intermediate SQuAD 2.0 fine-tuning definitively improves results for out of baselines, and all the FastDoc variants.
Additional Ablation Analysis
We perform the following ablations Fast:DocCuus)) Ro (1) We pre-train both the lower and higher- level encoders and fine-tune the lower encoder. This is referred to as FastDoc (oBt))))BBRR((FUU (2) We replace the lower encoder sRoBERTa (sentence transformer) with RoBERTa-BASE, (stíll keeping its weights frozen) and refer to it as MostDDoc(Cus.)Roele Blower (lower - RoB BR))...
F1
HA
FI@1
HA
F1@5
Dast Cus.) RROBERT.(FUL) FastDoe Cus. MoBBRRaa(oowe) RoBERTa)
17.4
33.71
46.23
15.76
30.54
42.52
Doe Cus., CuS.)ROBERTT. ROBERTA
17.52
33.94
44.96
Table 17: Additional Ablation Analysis on TechQA Dataset.
Table 17 shows the results corresponding to the ablations and FastDoc on TechQA Dataset. We can see that FastDoc performs better than FastDoc(Cuss RoBERr (lower RoBERTa) on all 3 metrics, and gives better Fl and HA F1@1 than Fast.Doc (C.s..
An alternative to the Triplet Loss: We also observe the effect of using an alternative for triplet loss. We use "quadruplet loss", which breaks similarity into categories instead of the binary notion in triplet loss. We sample documents per input Anchor(A), near positive(NP), far positive(FP). negative(N). NP most similar to anchor, followed by FP, and N. E.g. in Customer Support, NP has same brand, category that of anchor, FP has same category, different brand, and N has neither same brand category. Quadruplet
25

---

Published in Transactions on Machine Learning Research (05/2024)
Total
Hier. Classfn. Loss
Hier. Classfn. Loss 2
Hier. Classtn. Loss 3
Hier. Classfn. Loss
+
Quadruplet
Triplet Loss
+ a.
Triplet Loss?
Anchor Doc.
Near Positive Doc. Representation
Far Positive Doc. Representation
Negative Doc. Representation
Representation
hard perameter sharing
hard perameter shartng
hard parameter sharing
Transtormer Encoder
Transformer Encoder
Transformer Encoder
Transformer Encoder
Anchor Sentence Embeddings
Near Positive Sentence Embeddings
Far Positive Sentence Embeddings
Negative sentence embeddings
Pre-Trained Sentence Transformer (Frozen) *
Anchor Document
Near Positive Document
Far Positive Document
Negative Document
Figure 5: FastDoc Pre-training Archilecture It is similar to that of Fast.Dc. Differences are (1)
Instead of using a Triplet Loss Function (as in FastDoc), Quadruplet Loss Function is used in this case. (2) Anchor, Near Positive, Far Positive, and Negative Documents are taken as inputs.
Loss, denoted by L. can be mathematically stated as
Cg -.CAANPN)++
(3)
where K(O < K < 1) is constant to reduce weight of the second loss term in Equation 3 distance between A and NP is to be reduced more than that between A and PP. We denote this variation of FastDoc as FastDoc(()) (Q), as depicted in Figure 5 From Table 18 we observe that FastDoc performs better than FastDoc(Q) for = 0.1, 0.5 w.r.t all 3 metries on the TechQA Dataset.
F1
HA
FI@1
HA FI@5
Fast Cus. ROORRTA (Q, K 0.1)
16.1
31.2
43.04
Fast ( Cus. ROBERTO 0.5) Fast Doc ROBERT'
15.72
30.45
40.42
17.52
33.94
44.96
Table 18: Performance of FastDoc(Q) vs. FastDoc on the TechQA Dataset
Performance on the S10 QA Dataset
Table 19 shows the performance of basclines and proposed variants on Section Retrieval and Answer Retrieval
tasks S10 QA Dataset. Results reported the test set, similar to Nandy al. (2021). For Section
26

---

Published in Transactions on Machine Learning Research (05/2024)
Retrieval we report HITS@K the percentage of questions for which, the section containing the ground truth answer is one of the top K retrieved sections. We report values for K = 1,3. In Answer Retrieval a single answer is retrieved, hence НA FIOI is reported. The other metrics reported are (a). ROUGE-L I6
[inn 2004), and (b). Sentence and Word Mover Similarity (S+WMS) T7 (Clark et al. 2019 TS
We draw the following inferences from Table 19 (1) Longformer does not perform well, as global at- tention does not help in learning local contexts required for answer retrieval. (2) Among the baselines, EMamallRo.nERT gives the best HITS@1, RoBERTaBASH gives the best ROUGE-L F1, and DeCLUTR gives the best S+WMS score and HA FI@1. This shows that MLM and span-based contrastive learn- ing help extract non-contignous answer spans. perform better than FastDoe(Cuss (Cus.) variants. (4) Compared to TechQA, HA FI@1 scores on S10 QA Dataset are better, as answering questions from a single device is easier than answering questions from diverse sources. (5) In Section Retrieval, Fast.Doc Past.DocCuus.Raae (Cus.) gives the best HITS@1, suggest- ing that, pro-training using document-lovel supervision helps generalize
(3) Similar to TechQA, FastDoc RoBERT. variants
a device not seen during pre- training. (6) Dast.Doc MhastDocCCus.)naaaae C gives the best. ROUGE-L F1 and the second-best S+WMS score, while PastDoc(Crin CUw.:......T...... (hier.) gives the best WMS score, and the second- ROUGE-L FI and HA FI@1, as mapping documents to hicrarchical labels during pre-training can generalize to the context of device E-Manual not seen during pre-training. The hierarchy is particularly robust due to the variety in hierarchical labels. Combining it with triplet loss improves the lexical context, as can be seen from the value of ROUGE-L F1.
HITSQK
Answer Retrieval
ROUGE
S+
HA
K=1
K=3 3
-L F1
WMS
F1@1
BERT BASE
76.67
91.11
0.792
0.411
44.87
RoBERTaBASE
80
93.33
0.812
0.454
45.39
Longformer
75.56
93.33
0.768
0.415
41.1
EManualSERR
81.11
93.33
0.8
0.429
44.25
EMamualsRoBERR.
82.22
93.33
0.82
0.444
44.73
DeCLUTr
76.67
92.22
0.818
0.455
46.71
ConSERT
78.89
92.22
0.778
0.389
40.85
SPECTER
77.78
93.33
0.802
0.429
43.59
PastDoc (tripp)))
81_11
93.33
0.791
0.427
43.18
FastDoc (Cuu.) (triplet)
77.78
93.33
0.798
0.419
42.93
DastDoc (Cus.)BER
78.89
93.33
0.79
0.112
11.75
Fast.Dc (Ous.) R ROBERT
78.89
92.22
0.82
0.478
16.69
Dos DaatDDs..uS. ROBERR))oR
80
93.33
0.811
0.437
43.78
Dast (Cus. RoBERTo
82.22
93.33
0.828
0.463
46.22
Table 19: Results on the S10 QA Dataset (Best value for each metric is marked in bold, while the second- best value is underlined).
Qualitative Analysis of answers predicted by a proposed variant and a basclinc
We discuss qualitative results with 2 questions from TechQA Dataset and I question from S10 QA Dataset and the answers FastDOCROBERT' and a consistently well-performing bascline EManualSROBERI for each question. These questions, ground truth and predicted answers are listed in Table 20) The first question is a procodural question ('How' type), where both models give extra information w.r.t the ground truth. However, PastDDcRORERT. performs better extracting the exact number corresponding to the FFxx' which E.MamalsRoBERT. misses. The second question is in essence two questions together where one
19 provide
1sesed http:///ppi.orrr/pppjeett/yoooo/
uused https://github com/ wtte/eacllnn lark0//mms
1RROUGE-L ST WSS are reported, all questions in the S10 QA Dataset answerable, and these metrics make sense when each question has ground truth answer.
19 Note that even though Longformer performs well on TechQA, it does perform S10 QA Dataset. For compactness, chose only one consistently well performing baseline. However, the illustrations will be similar.
27

---

Published in Transactions on Machine Learning Research (05/2024)
is procedural ('How type') and the other is factual ('Is' type) question. Both the models output short answers that have minimal overlap with the ground truth, suggesting that it is difficult to answer multiple questions of different types at one go. However, FastDocRoBERY is nearer to the answer, as it suggests the issuc related to 'WS-Proxies' (mentioned in the ground truth), but that does not appear in the basclinc's answer. The third question a location-based question ('Weree' type). RosEDoCROBERTe answers it almost. correctly, with just some extra information on the paragraph containing the answer, whereas the baseline confuses 'fineeppritt list. for 'fingepprint recognition'. The observations point to the proposition that Dost.DocoBBERTS performs better at extracting numerical entities, tackling multiple questions in a sample, and answering location-based questions.
Dataset
Question
Ground Truth Answer A code hix for this issue is integrated into the Web- Sphereia Portal & WCM Combined Cummlative Fix (PI31198 [http://wee /www-- ol.ibm.com// comppppprt/////////// dneview. wes?
(tUDoooCUU.JJoBBeee EManualettBeeet
We are using Portal Server and when starting the Portal server see follow- mm0r000
A code fix for this is- sue integrated the WahSppereep Por- tal
A code fix for this is- sue integrated the WobSphere'n Por- tal & WCM Combined
ing
MiBaaSttrrter
WCM Com- binedCumulative Fix 05
corn ibmn www.scripting.srvy server. load- Extensions failed to key:
Cumulative
Fix 06
MfReanStarter
(PI31198 [http://wer /www-/1
(PI36060 hhttp://www.dt
uid 19811198) for Portal 8.5.0.0 available from Fix Central:
load extension: ExtverExt.applient
was?uid Baid==swl1PP 1wsi11P1111900
was PeruudsdwwgPPii 1P166060])
Portal), 8.5.0.0 avail- able from Fix Central:
Portally 8.5.0.0 avail- able from Fix Contral:
valne:
ibm. wps scripting AmpiecationServeeSttenn sions How can we resolve this error?
<LONG URL>
<LONG URL>
TeclQA Dataset
have made changes WSDL stored DP lncal filesystem. the WSP reffect this change since DstaPower Gateway 7.1.0 WebService Proxy We- bGUI doesn't offer the RRe- fresh)) button anymore? Is there best practice to re- fresh the WSP modified WSDL?
In WS Proxy, you can refresh WSDL from an
To minimize the impact this issue, consider creating additional WS- Proxies to reduce the number WSDLs per proxy.
Add/remove the WSDL from the WSP Gateway each time you change the WSDL. 2) Add one char summary and
can
URL using WSDL Cache policy. However, if the WSDL hosted on the DataPower appliance, such as in locak///. rather than
to the then backspace it. The ap- ply button will be enabled now. 3) Add WSDL Cache Policy.
external URL, the WSDL Cache Policy does not apply. stance, changes made local://teet wadl are not picked up by WS Proxy To refresh a WSDL the loral:/// dirertory, dis- able and re-enable the ser- vica.
in-
S10 QA
need the registered finger- print list. Where can I find this?
From Settings, tap Biomet- rics and socurity Finger- prints for the following op- tions: The list of registered fingerprints
Add, delete, and rename fingerprints. From Set- tings, tap Biometrics and security > Fingerprints for following options: list of registered finger- prints is at the top this finger-
Use fingerprint recognition to verify your identity in supported apps and tions.
Dataset
From Settings, tap Biometrics rity > FFingerprints. This is not available on the Galaxy S10e. This only available on the Galaxy $10e.
top this list. You can tap a fingerprint to remove or re- name it.
secu-
list. You print to remove or rename
Table 20:
Qualitative Analysis of answers predicted by FastDoc (Cus.) and the F.MamualSn.HRRT. baseline on the TechQA and S10 QA Datasets. (The parts in green overlap with the ground truth, and the parts in red do not overlap.)
In an effort understand the reason behind FastDoc being better extracting numerical entitics or an- swering location- questions, we see how well are such entities shared by anchor and positive documents compared to anchor and negative documents in Table 21
28

---

Published in Transactions on Machine Learning Research (05/2024)
Number of common entities in (anchor, positive) and not in negative
Number of common entities between anchor and positive
Number of common entities between anchor and negative
Numerical Noun Phrase
20.7
10.2
13.3
61.5
6.6
57.9
Table 21: Analysis of the distribution of numerical and noun phrase entities actoss document triplets
We observe that overlap between anchor and positive more than that between anchor and negative. Also, number of entities (positive, anchor) and not in negalive is more than the number of common entities between anchor and negative. Hence, numerical entities and locations (subset of noun phrases) are shared across highly similar documents, which is captured in the contrastive learning task performed during pre- training Doct Hence, FastDoc is proficient at answering numerical entity and location- praation-based questions.
E.2 Scientific Domain
Since recent. works have used citations similarity signal, report. the performance of FastDoc using citations similarity signal in Table 22 This gives satisfactory performance, showing that DostDoc works different metadata types. However, average, system using citations does not perform well when using "primary category'. In line with the recent works on Contrastive Learning, we apply FastDoc on triplets sampled using citations as a similarity signal and denote it as CastDoocSSii.CKi BERT- We can see in Table 22 that on an average, FastDoc (Scc.) performs better than FastDoc(Scii, Cit.) Cit.)BERT
Field
Task
Dataset
FastDDe(Scii. (itei..-itt)p -it.)EETT
FastDoe BERT
BC5CDR
87.55
87.81
NER
JNLPBA
75.9
75.84
BIO
NCBI-D
85. 12
84.33
REL
ChemProt
73.8
80.48
CS
REL
SciERC
80.8
78.95
Multi
CLS
SciCite
84.13
83.59
AVERAGE
81.22
81.83
Table 22: Fast vs. FastDoc(Sci-CKi Cit.) in tasks presented in Beltagy et al. (2019). Wo report macro F1 for NER (span-level), and for REL and CLS (sentence-leve)), except for ChemProt, where we report micro F1.
BERT
Comparison with GPT-3.5
We compare DastDoc with GPT-3.5 in the zero and one-shot settings for some tasks in the Scientific Domain in Tables 23 and 24 respectively. We can see that our proposed FastDoc performs much better compared to the highly capable and much larger GPT-3.5 in both zero and one-shot settings.
Field
Task
Dataset
Fast BERT
GPT-3.5 (Zero-Shot)
BC5CDR
87.81
56.04
(-36.18%)
NER
JNLPBA
75.84
41.25 (-45.61%)
BIO
NCBI-D
84.33
50.49 (-40.13%)
REL
ChemProt
80.48
34.16
(-57.56%)
Table 23: Results of Fast vs. zero-shot GPT-3.5 on some the tasks presented Beltagy (0019)
Field BIO
Task
Dataset
Post
GPT-3.5 (One-Shot)
REL
ChemProt
80.48
48.64 (-39.56%)
Table 24: Results of FastDoc vs. one-shot GPT-3.5 on some of the tasks presented in Beltagy et al. (2019
29

---

Published in Transactions on Machine Learning Research (05/2024)
E.3 Legal Domain
F Pre-training Compute of FastDoc relative to the baselines
Choice of GPU-Hours as a metric for measuring pre-training compute
FastDoc and the basclines in Table i use BERT RRoBERa backbone, making modules of sharding, data
parallelism, elc. uniform across models. Hence, GPU-Hours is appropriate for compute. Also, several works
pre-training such as Devlin et al. 2019) Liu et al. (019)) Cohan et al. (2020); Giorgi et al. (2021) report the number of GPUs and the time needed for pre-training, ie., they support the metric of GPU-Hours
Also, we compare the metrics of GPU-Hours and FLOPS Flloatingpooin operations per second) in Table
25 Note that FLOPS another reliable metric for measuring computer performance 20 and hence, training
compute.
FLOPS
GPU-Hours
EMaaualsR.BRRR.
4.46 x 10'8
980
Doc DocCssDDo CooRR)
2.56 x 10'5
0.75
Speedup
1745
1307
Table 25: Comparison of the speedup obtained using FastDoc in GPU-Hours vs. FLOPS
We observe that the compute speedup (or reduction) in Dost compared to the EManualsROBERT base- line is very close for the two metries of GPU-Hours and FLOPS, suggesting that GPU-Hours, like FLOPS, is indeed a reliable metric for measuring pre-training compute.
Clarification of calculation of pre-training compute of SciBERT
We would like to present the following evidence accompanied by suitable reasoning in support of the caleu- lated value of the pre-training compute-
1. The blog referred to in Footnote 5 of Beltagy et al. (2019) (https://timdettnee) com/ 2018/10/ trtpuus-v-gguu--orr trwassoorersseeeerrrrrrrr
titled "TPUs vs GPUs for Transformers (BERT)', discusses the compute requirements for BERT-LARGE and BERT-BASE using different GPU and TPU configurations and specifications. line from a section of the blog titled "BERT Training Time Estimate for GPUss states "On an GPU machine for V100/RTX 2080 Tis with any software and any parallelization algorithm (PyTorch, TensorFlow) one can expect to train BERT-LARGE in 21 days or 34 days". This does not match the sentence in the footnote, which suggests that it an 8 GPU machine. Hence, believe that the
is expected to take 40-70 days for pre-training sentence in the footnote does not correspond to BERT-LARGE, rather, it intuitively corresponds to SCIBERT. The blog was referred to give the reader an idea of how a comparison of GPU and TPC is made.
We must mention here that the TPU versions used in Beltagy et al. (2019) and Devlin et al. (2019)
are in all probability different. Beltagy et al. (2019) reports its pre-training time corresponding TPU v3, whereas Devlin et al. (019) does mention the exact version of the Cloud TPU used,
Also, according to the Google Cloud TPU Release Notes /https://cloud http::///lloud.ogglle wwt. haanm/tp//doc/// release-notes##ctooer 10__018)
see that the TPU v3 was first introduced (in beta release) on October 10, 2018. However, the first version of the BERT Paper was added to ArXiv (https: http://.rgg/b///////// aas//a10 04805v1) on October 11, 2018, just 1 day after the beta release of TPU v3 indicating that, some earlier version of TPU was used for the pretraining experiments.
20 httpp://kkk. edu/ d/ a
30

---

Published in Transactions on Machine Learning Research (05/2024)
G Analysis and Ablation
G.1 Catastrophic Forgetting in open-domain
Hyperparameters: For all such experiments, we fine-tune for 10 epochs, with learning rate of 3 10-3 input sequence length of 612, and batch size of 32. For task, the best development set results across all epochs is reported.
G.2 Parameter-Efficies training
The Trainable parameters of the encoder during fine-tuning are the same as that in the domain-specific pre- training stage. Hence, as an ablation, we explore the impact of a reduced number trainable parameters during pre-training by incorporating Parameter-Effictee Training.
Field
Task
Dataset
Past (Sci.))ERT
Past (ToRA)
BC5CDR
87.81
88.59 (+0.89%)
NER
JNLPBA
75.84
61.24 (-19.25%)
BIO
NCBI-D
84.33
39.57 (-53.08%)
REL
ChemProt
80.48
74.32 (-7.65%)
CS
REL
SciERC
78.95
62.19 (-21.23%)
Multi
CLS
SciCite
83.59
81.35 (-2.68%)
Table 26: Results of FastDoc trained using LoRA vs. proposed Fast.Dc on tasks presented
Beltagy
et al. (2019)
Table 26 shows the results of the parametee-efffiient training technique of LoRA (Low-Rank Adaptation)
Hu et al. 2022 to observe the effect of using reduced number of trainable parameters during continual domain-specific pre-training of FastDoc in the Scientific Domain. LoRA is applied on the upper encoder of FastDoc during pre-training. Fast performs significantly better compared to when using LoRA in downstream NER, when there are a large number of classes (as in JNLPBA, which has 11 classes), or when the datasot extremely imbalancod (as in NCBI-Discase, where 91.72% of the training samples belong to a single class). We altribute this to insufficient number of trainable parameters when using LoRA during pre-training. Similarly, LoRA performs poorly in Relation and Text Classification. On the contrary, there is meagre reduction in compute from 1.7 to 1.66 GPU-Hours when using FastDoc(Scii) LoRtA) instead of FastDoc(Set Sci.) (ec.SSci..R)))) BERT, suggesting that LoRA is not beneficial from compute perspective well.
G.3 Absence of document supervision
FastDoc
FastDoc
Fost (3 hier. levels)
Dataset
SCIBERT
(w/o est. meta, tax. 3 hier. levels)
(w/o est. meta, tax. 11 hier. levels)
BC5CDR
85.55
87.81
87.6
87.88
JNLPBA
59.5
75.84
75.91
76.06
NCBI-D
91.03
84.33
85,02
85.05
Pho
78.55
80.48
76.9
76.6
SciERC 74.3
78.95
79.26
81.21
SciCite
84.44
83.59
83.6
83.6
Table 27: Results of FastDoc on tasks mentioned in Beltagy et al. (019)) in Scientific Domain with and without established domain-specifie document. metadata and taxonomy, compared to well-performing base-
line
31

---

Published in Transactions on Machine Learning Research (05/2024)
Precision@
Model
AUPR
80% Recall
b+ Contracts Pre-training
45.2
34.1
44.8
34.6
MostDoc(Leg.)RoDD (4 hier. levels)
(-0.88%)
(+1.47%)
FastDoc(Legg (Le.)))Teeggerr RoBERTa (w/o est. meta., tax., 4 hier. levels)
46.7
38.7
(+3.32%%
(+13.49%)
FastDoc( Rog.RRBERT. (w/r meta., tax., hier. levels)
47.9
42
(+5.97%)
(+28.17%)
Table 28: Results of FastDoc on CUAD Dataset in Legal Domain with and without established domain- specific document metadata and taxonomy, compared to well-performing baseline
G.4 Why FastDoc works: Analysis of the interoperability of embeddings
Interoperability of pre-trained encoder parameters for input token and sentence embeddings
We present experiments and observations support the surprising interoperability of input embeddings, response the following research questions
Q1. How does FastDoc learn local contert?
We contrast the paragraph-level similarity between similar and dissimilar input documents used during pre-training, Given a pair of E--Manuls (from the Customer Support Domain), each paragraph in the two E-Manuals is converted to a fixed-size vector using Doc2Vec model (Le & Mikolov 2014) trained on Wikipedia, and the similarity score (cosine) with the most similar paragraph from the other E-Manual is considered as a PParagraph Similarity Score'. The distribution of this score across similar and dissimilar document pairs is plotted in Fig.
Nature of pre-training inputs:
Similar Documents
6-
Dissimilar Documents
5-
4-
Percent
3
1-
-0.2
0.0
0.2
0.4
0.5
0.8
1.0
Paragraph Similarity Score
Figure 6: Distribution 'Paragraph Similarity Score' across similar and dissimilar document pairs
We can infer that paragraph pairs from similar documents are skewed towards higher similarity, with more than half of the samples having 'Paragraph Similarity Score' 0.7. The sections where the similarity is even higher mainly the specific sections where some procedure/task. is specified like 'How to calibrate wircless thermometer?' or the device specifications and/or warnings about using the device/service are given, which would indirectly help the model in the downstream QA lask. Similarly, (even the most similar) paragraph pairs in dissimilar documents are skewed towards lower similarity, with a majority the samples having a 'Paragraph Similarity Score' < 0.5. Thus similar documents have very-similar local (paragraph- level) contexts, suggesting that, using document-level supervision during pre-training also helps in learning local contexts.
32

---

Published in Transactions on Machine Learning Research (05/2024)
Analysis of Local Context learned using FastDoc: Document-level supervision using FastDoc per- forms well on token-level tasks that require learning from local context (such as NER, relation classification, etc.), even though it was not explicitly trained on learning local context. To validate this, we measure the inffuence of local context on a random token vis-a-vis a standard MLM model (that learns from local context in accordance with the supervision signal). We randomly sample 500 sentences from each of the domains. For each sentence, we take random token and calculate the change in its prediction probability on masking other tokens in the sentence. Table 29 shows the Spearman Correlation of this change between two models (FastDoc and sppaif- model pre-trained using MLM) for each domain. We observe that the correlation is moderately high for all domains, showing that the local contexts is learned by FastDoc to a reasonable extent.
Domain Customer Support Scientific
Model using FastDoc
Model using MLM
Correlation
MnstDoccCCus.RoBBERR)
EMamualsROBERD
0.368
FastDoe(Sci.) BER) BERT
ScIBERT
0.481
Domain
Legal
RoBERTaBASE
LestDDoo(Leg)))BRRRRR RoBBRR
0.393
Domain
Contracts Pre-training
Table 29: Correlation of the change in masked token prediction probability between FastDoc and MLM, corresponding to other masked tokens, across domains.
Q2. Are the relative represenlations preserved across the two embedding spaces?
Qualitative Evaluation of Relative Document Representations: We analyze the relative document representations learnt by the pre-trained encoder in FastDoc MaatDoccCCus.).uue (Cus.) for both sentence-level and token-level input embeddings, For 4 different product categories (printer, plumbing product, battery charger, indoor furnishings), we consider E-Manuals cach containing between 400 512 tokens so that it complies with the maximum number of tokens accepted by BERT BASE or RoBERTaRASE as inputs. The cosine 20 similarity between the E-Manual representations of each of the C2 = 190 E-Manual pairs corresponding to both types of input emboddings obtained, and normalized (using max-min normalization). The similarity values corresponding to the two types of input embeddings are positively correlated each other, with the Pearson Corrclation value being 0.515. We further take the category-wise average representations, and repeat this experiment. We find that the Correlation for the similarity values is 0.977. Hence, the relative representations in both the representation spaces are highly correlated, which justifies good downstream performance when an encoder pre-trained on sentence embedding inputs is fine-tuned on token embedding inputs.
For a visual analysis of these F-manuals, PCA (Principal Component Analysis) is applied over the document representations to reduce the vector dimension from 768 to 2. These representations are then plotted (as shown in Fig. 7 for the pre-trained encoder in Fast ) RoBERTo and two types of input embeddings (sentence and token level). Different product categories are shown in different colors. We infer that indepen- dent of whether the inputs are sentence or token-level embeddings, the E-Manuals are clustered in a similar manner across the two representation spaces, and hence, the relative representations are preserved across the two embedding spaces.
Q3. How are pre-trainèng and fine-tuning compatible?
Compatibility between pre-training using FastDoc and downstream tasks via few-shot fine- tuning:
To test this compatibility, we fine-tune on small number of samples in a few-shot setting. We perform 50-shot fine-tuning (i.c., fine-tuning on 50 training samples) on 3 tasks from 3 domains Span- Based Question Answering on TechQA Dataset from Customer Support (with no intermediate SQuAD Fine-tuning), Text Classification on SciCite Dataset from Scientific Domain, and Span Extraction on CUAD Dataset from Legal Domain. We compared FastDoc (pre-trained using document-level supervision) with ROBERTaBASE /BERTBASE (pre-trained without any document- supervision). Tables 30 31 and 32 show the results, along with the respective improvements when using Fast.Doc. Better performance of
33

---

Published in Transactions on Machine Learning Research (05/2024)
1.5
Product Category
1-E
1-printer
2 2-plumbing product
3 3-battery charger
1.0
4-indoor furnishings
Principal Component
4-C
4-D
0.5
2-8
4-A
3-D 3-B
2-A
3-E
3-C
0.0
2-E
-0.5
1-c
2-D
2-C
-1.0
1-8
1-D
1-A
-2
Principal Component
(a) with sentence emboddings as inputs
1-D
Product Category
1-B
1-printer
1-E
2-plumbing product
0.4
3-battery charger
4-indoor furnishings
Principal Component 2
1-A
1-c
0.2
4-D
4-C
4-B
4-A
0.0
3-B
3-E
3-C
3-A
-0.2
2-D
2-C
-0.4
2-A
2-B
-1.5
-1.0
-0.5
0.0
1.0
Principal Component
(b) with token embeddings as inputs
Figure 7: 2D Plots of first two principal components of the document representations of 20 E-Manulls from 4 product categories using Fast.Dee (Cuss.)))) ROBERTA for different types of input embeddings
FastDoc across all domains in few-shot fine-tuning setting suggests that (1) Pre-training using document- level supervision is effective across 3 domains (2) FastDoc Pre-training and Finc-tuning are compatible.
Additional Analysis
Experiment on analyzing local context similarity of input embeddings: Fig. shows the distribu- tion of WL (Window Length) corresponding to input sentence and token embeddings for RoBERTa- based Past.Doe(uus.. RoBERTa encoder for similar and dis-similar document pairs, for documents that have be-
34

---

Published in Transactions on Machine Learning Research (05/2024)
Model
HA FIQ1
HA FIQ5
RoBERTaBASE
0.71
1.38
2.71
0.86
1.66
4.85
Fast.Doc(uus.)
(+21.1%%
(+20.33%
(+79%)
Table 30: Results on TechQA Dataset in Customer Support Domain
Model
Macro F1
BERTRASE
37.75
40.16
(+6.4%)
Table 31: Results on SciCite Dataset in Scientific Domain
Model
AUPR
RoBERTaBASE
0.13
0.14
PastDooc(Leg))neea ROBERTA
(+7.69%)
Table 32: Results on CUAD Dataset in Legal Domain
tween 400 512 tokens. Given pair of documents, the first being anchor, WL is 1 more than the distance between an input embedding of the first document, and the most similar input embedding of the second document, averaged across all embeddings of the first document. If similar embeddings are present at nearby positions in two documents, WL will tend to be smaller. Thus WL quantifies the local context similarity of the input embeddings. We observe that the distribution of WL is skewed towards smaller values (i.c., inputs are locally more similar) similar pairs compared dis-similar pairs, irrespective of the input (loken sentence) embeddings. Additionally, this also suggests that similar documents inherently induce learning of similarity between sentences and tokens when using Fast thus learning from local contexts.
35

---

Published in Transactions on Machine Learning Research (05/2024)
35
14
30
Percentage
25
Percentage
10
20
15
10
5
2-
8
16
11
15 16
Window Length
Window Length
(a) Sentence Embeddings of similar pairs
(b) Sentence Embeddings of dis-similar pairs
17.5
17.5
15.0
15.0
Percentage
12.5
Percentage
12.5
10.0
10.0
7.5
5.0
2.5
0.0
110 120 140 Window Length
80
90
100
170 180 190 200 Window Length
110
(c) Token Embeddings of similar pairs
(d) Token Embeddings of dis-similar pairs
Figure 8: Distribution of WL for sentence and token embeddings as inputs pre-trained MostDDoeCuusJJnnee ROBRTTa encoder for similar and dis-similar document pairs
36


Reviews:
[{"review_id": "IlJjxBcfsY", "reviewer": "Reviewer_hrPo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. Current pre-training objectives are very localized and do not consider document context. Therefore, it is worthwhile to explore directions for incorporating document-level similarity.\n\n**Weaknesses**\n\n1. Is it pre-training? - My main concern is about the way the paper is positioned. The document similarity and document hierarchy objectives are supervised objectives for which you need training data. Yes, for some domains, you can derive it via distant supervision (e.g., via arxiv hierarchy such as stat.ML), but that doesn’t mean the document hierarchy information is widely available for all domains. I also think that even though the model is trained via these two objectives and is tested on other tasks, I believe it is not pre-training but transfer learning. I would like to know more about why the authors are positioning this paper as a pre-training paper, as the objectives introduced are not self-supervised at all.\n\n2. Some claims regarding baselines are unfair: The paper asserts that it can drastically reduce pre-training compute (500x) compared to the amount of compute required for pre-training the base models. I am not sure if that's justified, as the models proposed in the paper are built on top of pre-trained models and not trained from scratch. Moreover, claims such as “The amount of data used is negligible compared to the 1.14M Papers used by SCIBERT (Beltagy et al., 2019) during its pre-training.” are also not justified, since SCIBERT used a self-supervised objective and trained the model from scratch. The self-supervised objective allows it to scale, and pre-training from scratch on a new domain makes it necessary to pre-train on a large scale.\n\n3. The paper trains different models for each domain, i.e., one model for customer, one for science, and one for legal. I don't think this is a scalable approach. What is the reason for not training one model that works across all domains?\"\n\n4. Catastrophic forgetting experimental setting: I am not sure if the catastrophic forgetting (CF) setting is accurate. Typically, to test CF, we consider an online setting where the model parameters are updated on new tasks, and performance is evaluated on old tasks to see if there is any degradation. In Table 7, the results are on the GLUE dev set, while the fine-tuning was on the train set of GLUE. Why is this considered catastrophic forgetting?\n\n5. For the contrastive loss, I am curious why the max-margin loss is selected instead of the more popular contrastive objective, as seen in eq 1 of the simCLR paper (Chen et al., 2020). Additionally, was a margin of 1 sufficient for all tasks?\n\n6. In Table 1, the papers 'Proximal Policy Optimization Algorithms' and 'Generating Natural Adversarial Examples' are considered as a positive pair. Are they truly similar? In other words, is using the same primary category enough to signal that documents are similar? I would encourage the authors to consider richer signals such as document representations, topic modeling, etc., to determine document similarity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces FastDoc, a novel pre-training framework for NLP that replaces traditional objectives such as masked language modeling and next sentence prediction  and  with document similarity learning and hierarchical classification tasks using domain-specific supervision. Utilizing a hierarchical architecture with lower and higher-level encoders, it significantly reduces pre-training compute requirements (up to 500x) while maintaining or improving performance across domains. \n\nThe hierarchical architecture in FastDoc incorporates two levels of encoders to enhance pre-training efficiency. The lower-level encoder is initialized using a pre-trained sentence transformer (sBERT/sRoBERTa) and is fixed during training. The higher-level encoder is initialized with a pre-trained BERT/RoBERTa encoder, operating with sentence embeddings from the lower level. This design choice allows for direct processing of sentence embeddings, enabling the model to capture larger contexts in a single input (e.g. 512 sentences with Bert-base models). Finally, the paper also explores further fine-tuning the higher level network after replacing it with the original token embeddings. The higher-level network is trained with two losses: a contrastive or triplet loss based on document similarity and a supervised loss for classifying documents into a domain-specific taxonomy. \n\nExperiments on various tasks (e.g. QA, classification, NER, etc) are run on 3 domains - scientific, legal, and customer support (e-manuals of products). The proposed model gets moderate improvements over the performance of base-models (BERT-base and Roberta-base)", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe the paper requires a significant round of revision. Addressing or clarifying all six points in the weakness section is necessary for me to consider recommending acceptance.\"", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nt0GAOeXZg", "reviewer": "Reviewer_Z3uB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents two interesting ideas and together (FastDoc) they work well: much lower training cost and as good (and even better) than several baselines in 3 different domains (9 datasets). \n\nMy first major concern is that it is unclear about the contribution of using sentence embeddings to the empirical results. What if only the proposed loss is used? We should easily test that by training the baselines with the proposed loss.\n\nMy second major concern is that the experiments aren't adequate. \n* Why did the authors experiment with only BERT and RoBerta-base? Why not with other models and sizes? This is crucial because: 1) as the authors claim that the proposed method is very fast, experimenting with other LLMs should  be manageable. 2) In some cases, e.g. Roberta-large outperforms Roberta-base + contract-pretraining (see Hendrycks et al 2021, tab 2)\n* Why LinkBERT, CDLM (sec 8) weren't used as baselines? \n* Numbers for SciBERT on table 3 don't match the numbers in Beltagy et al. Tab 1. \n\nMy other concerns are: \n* It's important to understand how FastDoc works but unfortunately most information is in Appendix rather than the main text. Still, I found the explanation of how FastDoc in the Appendix G.4 unsatisfying. In Table 22, as the models in comparison are based on the same LLMs, isn't that their distributions should be correlated? Besides, the answer for Q3 doesn't explain why FastDoc performs better than several baselines.\n*  In Table 7, on average, FastDoc is even better than Roberta-base on several datasets unrelated to Cus. domain. This is counter-intuitive. How could the Cus. domain help boost the performance of Roberta-base?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new method to quickly continuously pre-train a LLM towards a specific domain. The key contributions are two-fold.\n* A methodology to update a LLM using sentence-embedding input (like hierarchical transformers), but that LLM later on will be used with token embedding input  \n* A loss based on document similarity labels and hierarchical labels.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "I expect the authors clarify those points raised above.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SZs2cLKpwY", "reviewer": "Reviewer_WGMw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts**\n\nAligned with the contributions that I highlighted above, the paper (i) proposes an easy way to incorporate new information in the form of triplets and hierarchies, (ii) a new hierarchical model that learns from the new structured information, and (iii) extensively shows that it improves compared to baseline models.\n\n**Weaknesses**\n\nThere are several points that requires improvement, clarification/discussion.\n\n*Improvement*\n1. One major concern is that baselines seem to not trained with the extra information that FastDoc utilizes. This makes it hard to assess whether the architecture of FastDoc is significant or just the extra information that brings the most benefit. Can you utilize the extra information in pre-training of other baselines to see whether they also improve? Table-5 also lists some examples where it is clear that hierarchy can help but not necessarily a hierarchical model.\n\n2. Another major concern is that GPU hours is an inaccurate way of measuring the compute even when everything about the hardware is the same. It depends a lot on the implementation such as sharding, data parallelism, fast attention, etc. that makes it really hard to compare. Why not use a measure that is hardware independent as in scaling literature (like 6ND) [2] ? Similarly, in Section-F you reference open source blogs to justify your compute results including mapping from TPU hours to GPU hours or understanding the TPU version. This also suggests that hours is not a reliable and accurate measure of compute across a huge range of hardware, implementation, data, etc.\n\n3. Related to the compute concern, $EManuals_{BERT}$ is trained on the whole EManuals corpus (33.3 times larger) than FastDoc but its compute requirement is 1000 times more. Can you explain the reason why?\n\n4. You mention that SPECTER's main problem is due to lack of long context. This suggests that with longer context pre-training, it can achieve better results. Is this a general problem for the domains that you study? If you would use a model that can handle long contexts, can that replace FastDoc? For example, can you try LongT5 [1] for your setting?\n\n5. Can you explain why FastDoc is better at extracting numerical entities or location-based questions? Are these things that are shared across documents that would inform the model via hierarchies or categorical triplets?\n\n6. Does your model benefit from scale -- mostly model size?\n\n*Clarification/Discussion*\n1. Taxonomies or metadata are not always available. While the paper extends the results by showing a hierarchical topic modeling can help improve the results, it is not clear if this is applicable to the pre-training in general or would be helpful at all. Should we all be extracting hierarchies for C4 or code corpora?\n\n2. Would this only work for encoder-only models?\n\n3. What-if you had the extra information also during inference time?\n\n4. Section E.3 is empty.\n\n[1] [LongT5: Efficient Text-To-Text Transformer for Long Sequences] (https://aclanthology.org/2022.findings-naacl.55) (Guo et al., 2022)\n[2] [Training Compute-Optimal Large Language Models] (Huffman et al, 2022)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an alternative pre-training mechanism for encoder only models when additional information about the task is present. This additional information is either positive/negative examples or hierarchy of domain specific categories or both. These additional information suggests a different training mechanism than MLM or NSP objectives that are typically used in training encoder or decoder models. Here are more specific contributions of the paper:\n\n**A new training mechanism that incorporates example triplets and domain specific hierarchies of categories into the pre-training of BERT-like encoder models.** Triplets for each domain are generated based on corresponding categories such as product categories. This triplets are used with a margin-based contrastive loss. Hierarchies are taken from the corresponding domains when they are available; otherwise, Google Product Taxonomy of Arxiv Category Taxonomy are used to generate hierarchies. These hierarchies are used in a classification objective.\n\n**A hierarchical architecture for pretraining.** A sentence encoder model, such as ROBERTa with an output aggregation such as mean, is used to generate sentence encodings in a document. This encodings are used as inputs to a higher level BERT model that generates sentence level encodings conditioned on other sentences within a document. The higher level encoder is trained with the contrastive learning and classification objectives while the lower level encoder is frozen. This hierarchical approach with sentence level input encodings extend the context length of the model substantially compared to a token level model. Finally, the higher level transformer blocks are combined with the frozen token level embeddings to fine-tune on a downstream domain.\n\n**Extensive experiments on three different domains.** The paper studies three different domains that share positive/negative splitting of examples and information hierarchies. It shows improvement in all three domains compared to other encoder baselines while training a tiny fraction of the available data -- on customer support domain, it is only 3% of the entire corpus. I think this result nicely explains why rather than increasing the size of a corpus, adding auxiliary information can not only drop the labeling requirement but also give better results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Based on the above points that I made, a few changes in the paper are critical.\n\n1. Architectural changes or extra information, which one is more critical? Baselines should also use the extra information for a more fair comparison.\n\n2. Estimating compute using GPU hours is unreliable and inaccurate. A better way that is not dependent on the hardware and implementation is needed.\n\n3. A clarification and additional experiments on the long context aspect is needed.", "broader_impact_concerns": "There are no additional ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper contains extensive experiments showing that one can continue pre-training of a masked LM backbone (bert, roberta) on three domains using sentence embeddings as input and using document-level similarity losses using document taxonomy information. Despite training the backbone with sentence embeddings as input, experiments show that it can still be further fine-tuned with token embeddings as input for downstream tasks. The proposed document-level training objective is much faster than the masked LM (MLM) objective, but yields better or comparable performance on downstream tasks than continued pre-training at the token level with MLM. The authors also experiment with a heuristic that removes the need for document-level taxonomy information.

Overall, I think this paper is slightly above the bar of acceptance.
I *strongly* suggest authors to include discussions about the limitations in this paper in a limitations section, namely:

- Applicability of the proposed model to decoder-only models
- Applicability of the proposed model when downstream tasks are generation tasks

Reviewers asked also for the following points to be revised that are still not addressed in the rebuttal:

- Tone down (or remove) the claim that the method is faster than SciBert, which starts from scratch, while the method starts from Bert.
- Discuss what is the reason for not training one model that works across all domains.

Please, make sure to incorporate all the feedback from the reviewers in the paper along with the additional experiments performed.
Decision: Accept (poster)