how can i use the following information: ""
Case Study: Innovating Domain Adaptation through Continual Pre-Training and Model Merging
We show how Arcee uses the most innovative Continual Pre-Training and Model Merging techniques to deliver high-quality domain-specific language models at a fraction of the cost of our competitors–using Medical and Patent data.

Shamane Siri
Malikeh Ehghaghi
Luke Meyers
Vladimir Karpukhin
Charles Goddard
Mark McQuade
Shamane Siri, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Charles Goddard, Mark McQuade
Mar 19, 2024 • 9 min read
Case Study: Innovating Domain Adaptation through Continual Pre-Training and Model Merging
In the realm of specialized and secure language models, Arcee stands out with its focus on tailoring solutions that operate within the client's own cloud, leveraging their proprietary data. A cornerstone of our approach is domain adaptation, a critical yet resource-intensive process which maintains a balance between the general language capabilities and the specialized domain expertise of language models. This case study delves into how Arcee harnesses Continual Pre-Training (CPT) and Model Merging for cost-effective domain adaptation, showcasing our cutting-edge strategies in the Medical and Patent domains.

The Challenge of Domain Adaptation
Domain adaptation is paramount at Arcee, yet traditional methodologies demand considerable time and resources. In addition, a significant challenge arises with catastrophic forgetting, wherein post-pretraining often results in a deterioration of the model's original general abilities–hindering its fine-tuned performance across various tasks. This underscores the need for a method capable of incorporating domain-specific knowledge while mitigating forgetting and other deterioration. Our breakthrough lies in integrating two key methodologies: Continual Pre-Training (CPT) and Model Merging, designed to enhance efficiency and efficacy in adapting language models to specific domains.

Our Approach
Continual Pre-Training (CPT)
In language, CPT was studied under the name of domain adaptation pre-training where the new dataset comes from a new domain.[1] For instance, PMC-LLaMA,[2] an open-source medical-specific large language model, incorporates data-centric knowledge injection with pure CPT and medical-specific instruction tuning. It stands out as the first of its kind, showcasing superior performance on diverse medical benchmarks with significantly fewer parameters compared to both ChatGPT and LLaMA-2. As another example, ChipNeMo investigates the utility of large language models (LLMs) in industrial chip design, employing a domain-adaptive CPT approach in their adaptation process. They assess their model across three specific chip design applications: an engineering assistant chatbot, EDA script generation, and bug summarization and analysis. Their findings demonstrate that their domain adaptation pipeline enhances LLM performance substantially compared to general-purpose models, achieving up to a 5x reduction in model size while maintaining or improving performance across various design tasks.[3] Inspired by prior work, CPT at Arcee involves extending the training of a base model, such as Llama-2-base or Mistral-7B-base, using domain-specific datasets. This process allows us to fine-tune models to the nuances of specialized fields.

Model Merging
Model Merging involves synthesizing the capabilities of multiple pre-trained models into a single, more versatile checkpoint. This technique enables us to combine domain-specific models with general-purpose chat models, leveraging the strengths of both.[4][5][6]

Benefits of Our Method
• Domain-Specific Data Utilization: By employing CPT, we can incorporate proprietary client data, ensuring models are finely-tuned to specific requirements.
• Efficiency in Model Development: Utilizing existing chat models accelerates development, avoiding the need for complex and expensive model tunings to have chat-like capabilities.
• Cost Effectiveness: Fine-tuning smaller language models (SLMs) for specific domains yields substantial cost savings, with SLMs requiring only thousands of dollars for training compared to the billions needed for large language models (LLMs). Through Model Merging, our approach combines the specialized expertise of public SLMs with the broad domain-adapted SLMs, ensuring cost-effective and high-performance language model development.

Case Study Highlights
Continual Pre-Training Stage
Medical Domain
• Our project in the medical domain entailed the development of a CPT checkpoint from a vast dataset sourced from medical articles and books, as per the PMC-Llama[2:1] paper protocol. This initiative generated a dataset which is similar to the Meditron[7] dataset, which was then utilized to enhance a Llama-2-7B base model, without employing traditional data cleaning techniques like de-duplication and topic filtering. We stopped the training process after 3500 steps when approximately 27 billion tokens of the dataset were processed.

• The model was trained using a packed strategy, with each example containing 4096 tokens. This approach was implemented with a learning rate of (1.5 \times 10^{-5}) and batch sizes of 2048, utilizing the Trainium architecture. For additional hyperparameters, we used the methodologies outlined in Gupta et al.'s[2:2] work.

• Note: Our strategy did not extend to training beyond the 3500 steps due to the existence of Meditron,[7:1] an open-source PMC Llama-2 chat model trained on a curated and well-cleaned 48B token medical dataset, compared to our former dataset. Given Meditron’s exemplary performance, we acknowledge it as the pinnacle of CPT achievements in the medical domain and use it in place of the model our CPT efforts would have converged to.

• Both of the models helped in facilitating our exploration into how the quality of a CPT checkpoint impacts the task performance of a downstream merged model.

Patent Domain
• A similar approach was taken in the patent domain, adapting the methodology to the unique content and requirements of the United States Patent and Trademark Office (USPTO) dataset.[8] We took 10B patent tokens, as well as general tokens to reduce catastrophic forgetting, and did continual pre-training runs using Llama-2-7B as a base model. This resulted in a domain-adapted 7B patent model that performed exceptionally well on patent QA (synthetically generated), which was synthetically generated by us using new patents (held out patents), much better than a closed-source model with the same query.
• The model training was conducted in accordance with the DOREMI[9] settings, blending domain-specific data with a broad dataset of general red pajama data, totaling 30 billion tokens.
• We also created an instruction-tuned version of our domain-adapted patent base model with the use of a synthetically-generated instruction dataset.

Merging Stage
Leveraging MergeKit, we explored various merging techniques, such as Linear,[10] SLERP,[11], TIES,[6:1] and DARE,[4:1] to integrate our CPT checkpoints with general-purpose chat models. Model Merging maintains a balance between general and domain-specific knowledge while mitigating the risk of catastrophic forgetting, as the weights in the foundational general model can remain frozen. This stage was crucial for enhancing the model's adaptability and performance in specific domains.

Experiments and Results
Our research assessed the effectiveness of Continual Pre-Training (CPT) models and Model Merging strategies in the medical and patent domains. Performance of our final merged models on medical and patent benchmarks showcases our pipeline’s' ability to adapt to a certain domain.

Relationship between Medical and General Benchmarks and Checkpoint Steps
To assess the quality of our CPT efforts, we focused on the medical domain, recognizing the Meditron-7B[7:2] checkpoint for its superior refinement and domain-specific performance. This checkpoint served as a benchmark for evaluating the effectiveness of our CPT process. Our analysis spanned medical and general benchmarks[12]: USMLE, MedMCQA, PubMedQA, Arc Challenge, HellaSwag, MMLU.

blog chart 1.png

Observations
• Better CPT checkpoints can improve the final results after the merging stage.
• Final evaluation with the Meditron checkpoint emphasized the importance of carefully-selected CPT settings and high-quality datasets.
• Comparative results revealed that the quality of CPT checkpoints is vital for superior model performance after merging.

Which Merge Methods Work Well for the Medical Domain?
With a selection of refined checkpoints at hand, our next goal was to determine the most effective Model Merging techniques for the medical domain. We experimented with various methods, including SLERP, TIES, and Linear, to merge the Meditron-7B[7:3] checkpoint with Llama2-chat models, the base model of both being the Llama2 base model.

blog chart 1 (1).png

Observations
• The region between and around the models seems to be filled with low loss models as verified by evaluating various exploratory configurations. [5:1]
• Linear interpolation (Lerp) merge and Spherical Interpolation (Slerp) merge end up doing the same weight interpolation because the weights represented as flattened vectors are pretty much collinear (at which point Slerp transforms to Lerp).
• As per observations, linear merge works the best in this scenario.
• Ties merge method doesn’t have any edge here as the task vectors for the two models (Meditron and Llama-7B-chat) relative to the Llama2 base model are mostly orthogonal and as a consequence there is very little interference to resolve. And in the scenario of conflict resolution, Meditron larger magnitude most likely wins statistically maintaining Meditron as the larger contributor as is the case with the Lerp scenario
• Through various experiments with different Ties-merging configurations, reducing the threshold value which is a redundant weights filter seems to add to performance making for better evals (especially when it comes to the thresholding Meditron associated task vector). This could be explained by the product space distance between Meditron and Llama2-chat-hf is vast enough that: 1) The weight distributions are different, and 2) that missing portions of either model induced by the Ties mechanism (trimming) cannot be made up by fractional values by the corresponding model’s delta weights especially in the case when the induced values are those of Meditron’s.

Comparing the Patent Domain Checkpoints
As part of our exploration, we also worked in the patent domain. This area, unlike the medical domain, lacks public benchmarks–leading us to develop our own evaluations for this domain.

We introduced two benchmarks: 1) generating a patent abstract summary, and 2) a closed-book Question Answering where the questions were synthetically generated using a powerful LLM and manually-curated prompts. The former involves feeding the system the main content of a patent and requesting it to produce an abstract. The latter benchmark entails generating synthetic question-answer pairs and assessing the model's capability to provide accurate answers.

We use perplexity as the base primary metric in the absence of human expert labels. Using classical long text generation task metrics like ROUGE/BLUE are known to provide inconsistent correlations with actual quality for complex domains such as legal texts.

The results, detailed in the table below, compare the performance of domain-adapted patent models and instruction-tuned versions against the baseline performances of the LLama2-7B model and Llama2-7B chat. This comparison highlights the tailored models' effectiveness in navigating the complex and nuanced patent domain.

6.png

Which Merge Methods Work Well for the Patent Domain?
Similar to the medical domain, we also explore different merging methods with our CPT and Instruction checkpoints.

7.png

Observations
• As with the medical domain, the immediate region between the models seems to be filled with low loss models as verified by evaluating different configurations.

• As with the previous scenario (medical domain), Linear interpolation (Lerp) merge and Spherical Interpolation (Slerp) merge end up doing the weight interpolation because the weights represented as flattened vectors are collinear.

• In contrast to the previous domain where Ties-merging did not seem to make a significant positive difference, here it does make a significant difference. This is because the distance between Llama2-chat and patent-instruct is somewhat the same relative to the Llama2-base model.

Conclusion
The integration of Continual Pre-Training and Model Merging at Arcee.ai represents a significant leap forward in domain adaptation. Our case studies in the Medical and Patent domains demonstrate the potential of these methodologies to enhance the relevance and performance of language models across specialized fields. By leveraging domain-specific data, existing open-source checkpoints, and innovative Model Merging techniques, we are delivering cost-effective, high-quality models tailored to our clients' unique needs.

References
Gupta, Kshitij, et al. "Continual Pre-Training of Large Language Models: How to (re) warm your model?." arXiv preprint arXiv:2308.04014 (2023). ↩︎

Wu, Chaoyi, et al. "Pmc-llama: Towards building open-source language models for medicine." arXiv preprint arXiv:2305.10415 6 (2023). ↩︎ ↩︎ ↩︎

Liu, Mingjie, et al. "Chipnemo: Domain-adapted llms for chip design." arXiv preprint arXiv:2311.00176 (2023). ↩︎

Yu, Le, et al. "Language models are super mario: Absorbing abilities from homologous models as a free lunch." arXiv preprint arXiv:2311.03099 (2023). ↩︎ ↩︎

Stoica, George, et al. "ZipIt! Merging Models from Different Tasks without Training." arXiv preprint arXiv:2305.03053 (2023). ↩︎ ↩︎

Yadav, Prateek, et al. "Ties-merging: Resolving interference when merging models." Advances in Neural Information Processing Systems 36 (2024). ↩︎ ↩︎

Chen, Zeming, et al. "Meditron-70b: Scaling medical pretraining for large language models." arXiv preprint arXiv:2311.16079 (2023). ↩︎ ↩︎ ↩︎ ↩︎

Marco, Alan C., et al. "The USPTO patent assignment dataset: Descriptions and analysis." (2015). ↩︎

Xie, Sang Michael, et al. "Doremi: Optimizing data mixtures speeds up language model pretraining." Advances in Neural Information Processing Systems 36 (2024). ↩︎

Wortsman, Mitchell, et al. "Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time." International Conference on Machine Learning. PMLR, 2022. ↩︎

https://github.com/Digitous/LLM-SLERP-Merge ↩︎

Gilson, Aidan, et al. "How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment." JMIR Medical Education 9.1 (2023): e45312. ↩︎


Start building your SLMs today
Book a call with our team to discuss how Arcee can help you train, deploy, maintain, and continuously improve your smaller, secure, specialized, and scalable language models.
"" 

...to develop a llm to act as an advisor (plus sub-advisors)for the following tasks listed in this financial prospectus?: 

""PRINCIPAL INVESTMENT STRATEGIES
The Fund is an actively-managed exchange-traded fund (“ETF”). The Fund will invest in U.S.-listed equity securities
that Sparkline Capital LP (the “Sub-Adviser”) believes are attractive relative to its proprietary measure of “intangibleaugmented intrinsic value.”
- 2 -
Unlike most traditional quantitative value strategies, the Sub-Adviser’s definition of intrinsic value includes an
assessment of both tangible assets and intangible value. Intangible value is growing increasingly important as the
economy shifts from industrial to information-based. The Sub-Adviser focuses on four pillars of intangible value: (1)
human capital, (2) brand equity, (3) intellectual property, and (4) network effects, each of which are described more
below.
1. Human capital: Human capital is the value embodied by human beings. In the modern economy, the ability
to attract and retain top talent can be an important source of competitive advantage, as are company cultures
that motivate and nurture workers.
2. Brand equity: Well-known brand names are often able to generate sales simply due to strong consumer
recognition and loyalty. Companies may invest considerable resources in building their brands, which can
constitute a large component of their market value.
3. Intellectual property: Intellectual property encompasses creations of the human intellect. It includes both
legally-protected patents and proprietary trade secrets. As science and technology plays a larger role in human
society, intellectual property has increasingly become the primary source of value for many companies.
4. Network effects: Network effects are a phenomenon by which users of a product or service derive incremental
value from the addition of other users to the network. This can make it challenging for new entrants to unseat
firms with dominant market positions. As globalization and the internet increase the potential scale of
networks, network effects are becoming an important type of “moat.”
The Sub-Adviser employs a proprietary quantitative methodology to determine an estimated value of the foregoing
four pillars for each company as well as to determine an estimated value of each company’s tangible assets – the fifth
pillar.
The Sub-Adviser uses, among other sources, companies’ public accounting disclosures to analyze tangible assets.
However, the Sub-Adviser has concluded that most companies’ accounting disclosures omit or give only cursory
mention to their intangible value. The technical accounting definition of “intangible assets” is quite specific and
captures only a narrow subset of the Sub-Adviser’s broader concept of intangible value. As a result, a key component
of the Sub-Adviser’s process is its use of non-accounting data (or “alternative data”) to measure intangible value. In
general, such metrics are quite varied because each intangible pillar must be measured differently.
Because alternative data is often unstructured (e.g., text) and very large, the Sub-Adviser uses natural language
processing (NLP) (a form of machine learning) in addition to traditional quantitative investment techniques to
incorporate the data into its investment process. NLP is specifically designed to deal with unstructured text. The SubAdviser generally uses open-source NLP frameworks, which are widely used and vetted, and adapts them to the unique
use case of investing.
This investment process is applied to a starting investment universe of the approximately largest 1,000 publicly listed
U.S. securities (by market capitalization). The Sub-Adviser may remove companies from the universe if the SubAdviser determines they do not have a meaningful quantity of intangible value. For each company in the investment
universe, the Sub-Adviser considers multiple metrics for the company’s attractiveness according to each of the five
pillars, and then averages those metrics to produce a score for each of the five pillars. This is because the Sub-Adviser
believes that no one data source or metric is infallible and that by combining many metrics, a better result can be
obtained. Finally, the composite score is created by summing across the five pillars. The Fund will then generally seek
to hold the securities of the companies with the highest total scores.
The Sub-Adviser is not constrained by the number of portfolio holdings, except that the Fund will generally hold at
least 50 securities. The Fund’s investments may include common stocks and Real Estate Investment Trusts (REITs).
Although the Fund will not concentrate its investments in a particular industry, the Sub-Adviser anticipates that Fund
will hold a meaningful amount of stocks in the technology, communications, healthcare, and consumer discretionary
sectors.
- 3 -
The Sub-Adviser will seek to continually improve its valuation models used for the Fund as new datasets,
methodologies and research become available. The Sub-Adviser will also employ active risk management techniques.
As a result and because the Fund seeks to be fully invested at all times, the Sub-Adviser may recommend changes to
the Fund’s individual positions during dynamic market conditions.
PRINCIPAL RISKS
An investment in the Fund involves risk, including those described below. There is no assurance that the Fund will
achieve its investment objective. An investor may lose money by investing in the Fund. An investment in the Fund is
not a bank deposit and is not insured or guaranteed by the FDIC or any government agency. More complete risk
descriptions are set forth below under the heading “Additional Information About the Fund’s Risks”.
Investment Risk. When you sell your Shares of the Fund, they could be worth less than what you paid for them. The
Fund could lose money due to short-term market movements and over longer periods during market downturns.
Securities may decline in value due to factors affecting securities markets generally or particular asset classes or
industries represented in the markets. The value of a security may decline due to general market conditions, economic
trends or events that are not specifically related to the issuer of the security or to factors that affect a particular industry
or group of industries. During a general downturn in the securities markets, multiple asset classes may be negatively
affected. Therefore, you may lose money by investing in the Fund.
Value Style Investing Risk. A value stock may not increase in price if other investors fail to recognize the company’s
value and bid up the price, or the markets favor faster-growing companies. Investing in or having exposure to “value”
stocks presents the risk that the stocks may never reach what the Sub-Adviser believes are their full market values,
either because the market fails to recognize what the Sub-Adviser considers to be the companies’ true business values,
including its assessment of their intangible value, or because the Sub-Adviser misjudged.
Alternate Valuation Risk. The Sub-Adviser assesses the intrinsic values of companies by incorporating alternate,
non-traditional measurements, within its calculations. There is a risk that the alternate measurements may be incorrect
or the Sub-Adviser’s assessment of them may not be reflected in the company’s stock price. In addition, there is a risk
that some alternate valuation data for particular companies may be impossible or difficult to obtain, or difficult to
analyze even with the aid of NLP tools. As a result, the Sub-Adviser may need to rely on different data sources when
valuing differing companies. Therefore, the Sub-Adviser’s strategy of incorporating alternate valuations with
traditional valuations may not produce the desired results and may not perform as expected.
Equity Investing Risk. An investment in the Fund involves risks similar to those of investing in any fund holding
equity securities, such as market fluctuations, changes in interest rates and perceived trends in stock prices. The values
of equity securities could decline generally or could underperform other investments. In addition, securities may
decline in value due to factors affecting a specific issuer, market or securities markets generally.
Technology Sector Risk. The Fund will have exposure to companies operating in the technology sector. Technology
companies, including information technology companies, may have limited product lines, financial resources and/or
personnel. Technology companies typically face intense competition and potentially rapid product obsolescence. They
are also heavily dependent on intellectual property rights and may be adversely affected by the loss or impairment of
those rights.
Communications Sector Risk. The Fund will have exposure to companies operating in the communications sector.
Communication companies are particularly vulnerable to the potential obsolescence of products and services due to
technological advancement and the innovation of competitors. Companies in the communications sector may also be
affected by other fierce competitive pressures, including pricing competition. They may also be adversely affected by
research and development costs, substantial capital requirements, and increased governmental regulation.
Consumer Discretionary Sector Risk. The Fund will have exposure to companies operating in the consumer
discretionary sector. The consumer discretionary sector may be affected by changes in domestic and international
economies, exchange and interest rates, competition, consumers’ disposable income and consumer preferences, social
trends and marketing campaigns.
- 4 -
Healthcare Sector Risk. The Fund will have exposure to companies operating in the healthcare sector. Companies in
the healthcare sector, including drug related companies, may be heavily dependent on clinical trials with uncertain
outcomes and decisions made by the governments and regulatory authorities. Further, these companies are dependent
on patent protection, and the expiration of patents may adversely affect the profitability of the companies.
Additionally, the profitability of some healthcare and life sciences companies may be dependent on a relatively limited
number of products, and their products can become obsolete due to sector innovation, changes in technologies or other
market developments.
Quantitative Security Selection Risk. Data for some companies may be less available and/or less current than data
for companies in other markets. The Sub-Adviser uses quantitative models, and its processes could be adversely
affected if erroneous or outdated data is utilized. In addition, securities selected using a quantitative model could
perform differently from the financial markets as a whole as a result of the characteristics used in the analysis, the
weight placed on each characteristic and changes in the characteristic’s historical trends.
Machine Learning Risk. The Fund relies heavily on a proprietary “machine learning” selection process as well as
data and information supplied by third parties that are utilized in that process. To the extent the machine learning
process does not perform as designed or as intended, the Fund’s strategy may not be successfully implemented and
the Fund may lose value. If the input data is incorrect or incomplete, any decisions made in reliance thereon may lead
to the inclusion or exclusion of securities that would have been excluded or included had the data been correct and
complete.
Alternative Data Risk. The Sub-Adviser employs so-called “alternative data,” which generally refers to data that is
not the traditional exchange or accounting data that has been widely used by the mainstream investment industry.
Risks associated with alternative data include the possibility of new legal and regulatory frameworks targeting the
collection and use of the data or technological changes that may make the data less useful or available. There is also
the possibility that the organizations providing alternative data may cease operations, change business models, or
suffer temporary outages due to technical issues. Insider trading and “fair practice” laws are generally untested in this
area. Investment decisions based on alternative data may be flawed for various reasons, such as incomplete, “dirty”
or misunderstood data, or problems with the technology used to collect and analyze it.
Management Risk. The Fund is actively managed and may not meet its investment objective based on the Adviser’s
or Sub-Adviser’s success or failure to implement investment strategies for the Fund. In addition, the Fund’s principal
investment strategies are dependent upon the Sub-Adviser’s use of its proprietary machine learning security selection
process and, as a result, the Sub-Adviser’s skill in understanding and utilizing such process.
Small- & Mid-Capitalization Companies Risk. Investing in securities of small- and medium-capitalization
companies involves greater risk than customarily is associated with investing in larger, more established companies.
These companies’ securities may be more volatile and less liquid than those of more established companies. Often
small- and medium-capitalization companies and the industries in which they focus are still evolving and, as a result,
they may be more sensitive to changing market conditions.
REIT Risk. A Real Estate Investment Trust (REIT) is a company that owns or finances income-producing real estate.
Through its investments in REITs, the Fund is subject to the risks of investing in the real estate market, including
decreases in property revenues, increases in interest rates, increases in property taxes and operating expenses, legal
and regulatory changes, a lack of credit or capital, defaults by borrowers or tenants, environmental problems and
natural disasters. Investments in REITs may be volatile. REITs are pooled investment vehicles with their own fees
and expenses and the Fund will indirectly bear a proportionate share of those fees and expenses.
Geopolitical/Natural Disaster Risks. The Fund’s investments are subject to geopolitical and natural disaster risks,
such as war, terrorism, trade disputes, political or economic dysfunction within some nations, public health crises and
related geopolitical events, as well as environmental disasters, epidemics and/or pandemics, which may add to
instability in world eco""