W H I T E PA P E R

Boost Accuracy inDomain-Specific NLP With
SambaNova’s Solutions

Modern industrial and research NLP models follow
a cadence for their domain fine-tuning task training
pipelines. They usually are initialized from off-the-shelfpre-trained weights — such as BERT-large-uncased from

What’s Inside
The Power of Domain
Knowledge2

HuggingFace’s model zoo — and then are fine-tuned
on task-specific and domain-specific datasets before
deployment to serve massive online inference requests
from application users.
Recent research has demonstrated that this industrystandard pipeline does not capture domain-specificknowledge during language modeling, and can lead tosuboptimal statistical metrics on any domain-specificfine-tuning task.1 2 3
Gururangan et al.1 demonstrate how class-wise F1scores on Hyper-Partisan Detection,4 a task from theNews Domain, can be increased from 86.6 to 88.2 bysimply tuning the pre-training corpus to have more
relevant domain exposure.

Domain Pipelines

3

Out-Performance ofDomain-Customized
Pipelines4What Do You Needto Do?6

Boost Accuracy in Domain-Specific NLP With SambaNova’s Solutions

Introduction
One of the main reasons behind the lack of widespread
adoption of domain-focused training is due to theenormous time, compute, and human capital investments

minimal effort.

”

By enabling domain-specific pre-training, we demonstratepre-training, wenecessary to implement this specialized training.
SambaNova Systems, a full-stack hardware-software
company, provides simple one-click solutions for both thehardware and software challenges involved in building
these pipelines. This allows users to maximize downstream
statistical accuracy for domain-specific NLP pipelines withthat such solutions can offer superior statistical performancecompared to the standard off-the-shelf industry

By enablingdomain-specificdemonstrate thatstandard pipelines.

such solutions

The Power of Domain Knowledge

can offer

Current industrial workflows initialize their fine-tuning

superior statisticalpipelines from weights such as the bert-large-uncased
weights from HuggingFace.5 These weights do not have

performancedomain exposure as they tend to be from the generaldomain, often pre-trained on corpuses like Wikipedia
English6 and the Books Corpus.7
These corpuses have a limited exposure to domain-specificvocabulary. For instance, only about 0.004% of English
Wikipedia’s sentences contain the word “plaintiff,” a word
relevant to nearly all legal proceedings. On the other hand,
if we consider a legal dataset such as CaseHOLD,8 over 33%
of that dataset’s sentences contain the same word.
It is clear that a model exposed to CaseHOLD would havemore legal domain emphasis than one trained on English
Wikipedia. We showcase how this increased emphasis
manifests itself by outperforming the standard industry
pipeline on tasks from the legal, financial, andbiomedical domains.

2Boost Accuracy in Domain-Specific NLP With SambaNova’s Solutions

Domain Pipelines
In our solution, we implement two pipelines to enable domain exposure during pre-training
and compare them to the standard pipeline utilized by many industrial cadences today,
whereby bert-large-uncased from HuggingFace is used to fine-tune without any consideration
to domain exposure.
The first pipeline pre-trains on Eleuther’s PILE dataset,9 a massive corpus with exposure to BioMedicaland Legal domains. The second pipeline introduces an intermediate stage to the NLP cadence
after the pre-training but before the fine-tuning stage, where we do language modeling on thefine-tuning dataset itself, as described in Gururangan et al.1
SambaNova’s two pipelines, along with the current industry standard pipeline, are shown in Figure 1,
with the differences highlighted in the dark rectangular boxes. We refer to these two pipelines
as “Domain-Customized Pipeline 1” and “Domain-Customized Pipeline 2.”
To emphasize the singularly important role of data in this process, we maintain the samevocabulary and model architectures across all three pipelines in this blog post. The vocabulary
and model architectures can be further tuned with SambaNova’s solutions for potentially evenbetter performance.

Current Industry Standard Pipeline
Pre-training (Language Modeling)~WikiCorpus-En + Books Dataset~

Fine-tuning (Classification)~Domain Dataset~

Model: BERTLarge
Vocabulary: BERT-large-uncased

Fine-tTuning (Classification)~Domain Dataset~

Model: BERTLarge
Vocabulary: BERT-large-uncased

Pre-training (Language Modeling)~Domain Dataset~

Fine-tuning (Classification)~Domain Dataset~

Domain-Customized Pipeline 1
Pre-training (Language Modeling)~PILE Dataset~

Domain-Customized Pipeline 2
Pre-training (Language Modeling)~WikiCorpus-En + Books Dataset~

Model: BERTLarge
Vocabulary: BERT-large-uncased

Figure 1: Augmented parts of SambaNova’s pipelines, compared to the industry standard, are shown in the dark rectangular boxes.

3Boost Accuracy in Domain-Specific NLP With SambaNova’s Solutions

Out-Performance of Domain-Customized Pipelines
We demonstrate superior statistical performance on text classification on CaseHOLD (Legal),
FiQA+PhraseBank10 11 (Finance), and ChemProt12 (BioMedical) using either of the two DomainCustomized pipelines, compared to the current industry standard pipeline, without any changes tothe model architecture or the vocabulary the model learns.
Legal Domain — CaseHOLD
We observe a 1.87% increase in seed-average test accuracy score when using Domain-Customized
Pipeline 2 over the industry standard pipeline, and a 1.36% increase in seed-average test accuracy
score when using Domain-Customized Pipeline 1 over the industry standard pipeline.

Figure 2: The seed-average test accuracy on CaseHOLD classification

4

Boost Accuracy in Domain-Specific NLP With SambaNova’s Solutions

BioMedical Domain — ChemProt
We observe a 1.37% increase in seed-average test accuracy when using Domain-Customized
Pipeline 2 over the industry standard pipeline, and a 0.27% increase in seed-average test accuracy
when using Domain-Customized Pipeline 1 over the industry standard pipeline.

Figure 3: The seed-average test accuracy on ChemProt relation classification

Finance Domain — FiQA+PhraseBank
Due to the small size of FiQA and PhraseBank, we combined the datasets together before
performing sentiment classification. We observe a 1.88% increase in seed-average test accuracy
when using Domain-Customized Pipeline 2 over the industry standard pipeline. As the PILE dataset
has no significant financial exposure, we do not report the performance of Domain-Customized
Pipeline 1 as it is not relevant to the central message.

Figure 4: The seed-average test accuracy on FiQA+PhraseBank combinedclassification

5

Boost Accuracy in Domain-Specific NLP With SambaNova’s Solutions

What Do You Need to Do?
You should consider the following areas inpreparation for NLP:
• Determine whether the business problem you are trying
to solve is domain-specific or not. For example,are you
trying to develop a general-purpose sentiment analysis
solution or a domain-specific one?
• If you are trying to solve a domain-specific problem,
like most SambaNova customers do, such as sentiment
analysis in finance, you will need to ensure you haveaccess to NLP models pre-trained based onpipelines
with domain exposure.
• SambaNova offers a purpose-built hardware andsoftware stack that makes it possible to substantiallydecrease the amount of time, compute, and humancapital investment necessary to formulate and set up
these domain-exposed pipelines for a given workflow.
To learn more, please visit our NLP solutions web page.

https://arxiv.org/pdf/2004.10964.pdf
https://arxiv.org/pdf/1908.10063.pdf
3https://arxiv.org/pdf/1903.10676.pdf
4https://pan.webis.de/semeval19/semeval19-web/5
https://huggingface.co/BERT-large-uncased
6
https://en.wikipedia.org/wiki/English_Wikipedia
7
https://yknzhu.wixsite.com/mbweb
8
https://arxiv.org/pdf/2104.08671.pdf
9https://pile.eleuther.ai
10https://sites.google.com/view/fiqa/
11https://arxiv.org/pdf/1307.5336.pdf
12https://pubmed.ncbi.nlm.nih.gov/20935044
12

6

Learn more at SambaNova.ai
linkedin.com/company/sambanova@SambaNovaAI@SambaNovaAI

info@sambanova.ai

AI is here. With SambaNova, customers are deploying the power of AI and deep learning in weeks rather than years to meet the demandsof the AI-enabled world. SambaNova’s flagship offering, Dataflow-as-a-ServiceTM, is a complete solution purpose-built for AI and deeplearning that overcomes the limitations of legacy technology to power the large and complex models that enable customers to discover
new opportunities, unlock new revenue and boost operational efficiency. Headquartered in Palo Alto, California, SambaNova Systems
was founded in 2017 by industry luminaries, and hardware and software design experts from Sun/Oracle and Stanford University. Investors
include SoftBank Vision Fund 2, funds and accounts managed by BlackRock, Intel Capital, GV, Walden International, Temasek, GIC,Redline Capital, Atlantic Bridge Ventures, Celesta, and several others. For more information, please visit us at sambanova.ai or contact us
at info@sambanova.ai. Follow SambaNova Systems on LinkedIn.

Copyright © 2022 SambaNova Systems, Inc. All rights reserved. | Palo Alto, CA and Austin, TX | (650) 263-1153