Stock Price Impact of Drug Event through LLM Analysis of Specialized Texts
from FactSet StreetAccount Drug Events & Pipeline
Mitsubishi UFJ Trust Investment Technology Institute Co., Ltd. (MTEC)1

1. Introduction
The aim of this paper is to analyze drug event data, which is fed by Factset as FactSet
StreetAccount Drug Events & Pipeline [FACTSET]. According to the Factset Marketplace,highlights of its data are:Measure company and market impact of past and future events within thehealthcare industry by monitoring regulatory decisions, filings, meetings, andclinical trials for specific pharmaceutical products.
Capitalize on alerts about changes in a drug life cycle that may affect manufacturing
company performance involving patent filings for medication, patent expirations on
drugs, current approval stage, and release of new indications. Review detailed
footnotes that provide context and relevance of a drug event. Access curated FDA
Orange Book patent and exclusivity expiration dates.
Gain a holistic view of the pharmaceutical market by analyzing the pipeline for toplevel indications and classes.

1The information contained in this report is provided "as is" and does not carry any

warranties or guarantees of accuracy or completeness. The information is provided solely for
informational purposes and should not be considered as professional advice. Any actions ordecisions based on the information in this report are at the reader's own risk. This report isintended to provide general information and does not constitute specific advice orendorsements for any particular situation or requirements. Mitsubishi UFJ Trust Investment
Technology Institute Co., Ltd. shall bear no responsibility or liability for any actions taken
based on this analysis or for any loss or damage incurred based on this report. None fromamong Mitsubishi UFJ Trust Investment Technology Institute Co., Ltd., its suppliers, nor
any of their respective directors or employees accept any liability towards any third party
who has obtained this document with respect to any loss or damage directly or indirectly
arising out of the use of all or any part of this report. ©2024 Mitsubishi UFJ TrustInvestment Technology Institute Co., Ltd. All Rights Reserved.
1

The importance of data analysis in the biopharmaceutical arena cannot be overstated.
As the industry continues to advance at a rapid pace, the ability to accurately interpret
various events and predict their impact on stock prices becomes increasingly important.
The purpose of this paper is to focus on the impact of clinical trials and other significant
events in the biopharmaceutical industry on subsequent market value using the event datacontained in the StreetAccount drug event data. By introducing large language models(LLMs) such as GPT-4 into the analysis process, we present a new approach that
increases the accuracy and depth of data interpretation of previously difficult shorttechnical texts.
The organization of this paper is delineated as follows: Section 2 explores thedistribution patterns of event types on an annual basis, among other aspects. Additionally,
it provides a discourse on the selection of the appropriate 'event time' from the multitudeof time-series variables. In Section 3, the paper elucidates the methodology implemented
for the development of the event shock measure and examines the specific event types
that exert influence on the market. In Section 4, the study implements and evaluates anexperimental approach using GPT-4's few-shot learning to predict stock market shocks
from event descriptions. The conclusion, Section 5, summarizes the paper and discuss
how to improve these methods.

2. Data at a Glance
In this section, we will present the key data points used for our analysis. Before theanalysis, we have to set the event occurrence time. In the data, there were several columns
which we can consider as 'event_date', 'event_exact_date', 'event_modified_datetime' andso on. However, 'event_exact_date' often recorded dates earlier than'event_modified_datetime' (Table 1), and there was a high possibility that dates prior tothe available information date were mixed in, and due to the number of missing recent
data. 'Event_create_datetime' only had timestamps from 2022 onwards, which is when thedatabase is believed to have been created. Therefore, we adopted'event_modified_datetime' as the event date.

2

Table 1: Relation between exact_date and modified_date
Count

Modified

Exact

2013

2014

2015

2013
2014

2016

2017

2018

2019

2020

2021

2022

2023

2024 total

3
2

2015

138

4

184

5

287

528

10972

2016
2017

3
7335
820

97875

17

2018

973

2019

998
892
21
9092020

994
18996

2021

17
1125

1

928

3

1016

161

1

1097

117

9

1223

2023

897

220

1117

2024

19

218

237

2022

N/A
total

2

143

301

1287

8

1

11

7

24

15

110

518

476

1170

1695

885

1015

937

1038

1157

1372

1552

923

11020

Figure 1: Distribution of Data per Year (Total Count)

Figure 1 represents the annual distribution of event data, aggregated by the number ofevents per event type until 2023. Here we use 'event_modified_datetime' as the event date.
It becomes apparent that there is a significant amount of data from 2016. However, whencomparing 'event_exact_date' as in Table 1, it seems that many events whoseevent_modified_date are in 2016 actually have their 'event_exact_date' in 2015. One
hypothesis for this discrepancy could be that from 2016 onwards, data retrieval from a main
data source of this dataset became possible, leading to a large inclusion of data from prioryears, particularly 2015, within the data whose ‘event_modified data’ is in 2016.
Except that, while the number of events has been increasing year by year, thedistribution of event types has not changed significantly. The most common types of events,3

in descending order, are Phase 2, Phase 3, and Regulatory decisions, with a slight recent
increase in Phase 1 events and Regulatory filings.
Figure 2 aggregates the number of companies recorded in the data on an annual basis.
As expected, the number of companies has been increasing with each passing year.
Figure 2: The Number of Companies by Year

3. Measuring the Market Impact of Events
To measure the market impacts of events, we employ the following method:I.

Calculate the stock price return over a set number of days (1day, 3days, 7days,30days and 90days) starting from the day before 'event_modified_datetime.'

II.

Compute the excess return over the average return of the universe of companies
whose events were recorded.

III. Normalize the excess return using the variance of the excess returns of thecompany.
This approach allows us to standardize the event shocks and compare them acrossdifferent events and time periods. The dataset employed in this analysis covers theperiod from 2017 through to 2023.
According to this approach, the study measured the stock price impact of clinical
trial events and the results are as follows. Table 2 presents t-values for different event
types across various time horizons. The significance levels at 5% and 10% are also
indicated.
4

Table 2: Impact of Clinical Trial Events on Stock Prices (t-values)
event_type

ReturnHorizon

Phase 1

Phase 2

3days

0.591

-1.096

7days

Phase

Phase 3Phase 4Phase 1/2

Phase 2/3

0.259

-0.4000.041-0.646-0.5600.839-0.179 1.668*

0.095-0.672

0.640

0.535

30days0.110-2.400** -1.427

0.418-0.531-0.6100.017

90days-2.297**-4.430** -1.084-4.473-0.228-0.764

0.571

Unknown*, ** indicate significance at 10%, 5% levels, respectively.

From the results of the study, several insights can be discerned regarding the impact ofclinical trial events on stock prices over different time horizons. Notably, the findings
suggest that the market reacts slowly for clinical trial events.
For the shorter 3-day and 7-day horizons, the t-values indicate that there is no
consistent pattern of significant stock price impact across the different phases of clinical
trials. The t-values are relatively low, and none reach the 5% significance level, suggestingthat the market does not react strongly to clinical trial announcements in the immediateshort term.
However, over the 30-day horizon, Phase 2 events show a significant negative impact
on stock prices at the 5% level, as indicated by a t-value of -2.400. This suggests thatnegative outcomes or uncertainties associated with Phase 2 trials can lead to a notabledecline in stock prices.
The most pronounced effects are observed over the longer 90-day horizon, where Phase
1 and Phase 2 events show significant negative impacts at the 5% level. This observationsuggests that announcements of clinical trial results, other than those directly leading toproduct commercialization (such as Phase 3), may be negatively evaluated by the market as
they imply a longer time for investment recovery. This is likely because, in the early stagesof clinical trials, even if successful, there is a high level of uncertainty and significant risk
associated with progressing to commercialization. Consequently, the study revealed that
there is a negative stock price impact associated with the announcement of clinical trial
events in these early phases. Next, the impact of events except for clinical trial events on
stock prices is the following.

5Table 3: Impact of Biopharmaceutical Events on Stock Prices (t-values)

Litigation

Conference-0.683-1.037

1.418

0.813

1.487

-0.579-0.361 13.823**

30days-0.546-0.037 2.868**-0.529

1.418

0.526

-1.9680.354

90days-0.815-0.359 5.970**-0.0570.929

2.278*-1.454-0.595submitted

Filing

7days

Regulatory-0.274

Meeting-0.735

Regulatory

1.952*

Decision

1.653

Regulatory-1.009 2.692**

discontinued

Clinical Trial --0.130

Horizon3days

ReturnPre-Clinical

Pending - NDA

event_type

1.536*, ** indicate significance at 10%, 5% levels, respectively.

The results indicate that Regulatory Decisions have a significantly positive impact on
stock prices over both short and long-term horizons, particularly at the 90-day mark with at-value of 5.970, indicating a strong positive reaction. On the other hand, Regulatory Filings
show a positive impact in the shorter term. It makes sense that Regulatory Decisions havepositive impact for stock prices because they confirm the market potential and profitabilityof a company's product, especially with new drug approvals. Such decisions validate thecompany's R&D efforts and can resolve uncertainty, often leading to increased investor
confidence and investment. In essence, they are key milestones that can unlock substantial
value for pharmaceutical companies.
Litigation events mean specifically patent litigations in this context. This type of events
also shows a positive impact on stock prices. This is because successful resolution can secure
a company's exclusive marketing rights and protect its revenue from generic competition,
thus naturally leading to a positive stock market response over the long term.
It was observed that there are significant positive disparities in long-term returns
specifically associated with Regulatory decisions and Litigation event types. However, only afew event types demonstrated a statistically significant impact on long-term returns.
Notably, this significance was not evident in pivotal events such as Phase 3 clinical trials.
The details of Phase 3 clinical trials are expected to have a significant impact on the stock
market as they will reveal how close the drug is to commercialization. However, whether this
impact will be positive depends on a variety of factors, including the feasibility of the drug,
the number of people with the potential disease, and the degree of cure. The direction of theimpact cannot be determined by event type alone and may not yield a significant return.
6

Although this information is encapsulated within the ‘event_description’, the utilization ofspecialized, terse language presents challenges for classification without pre-existingknowledge.
At this point of view, it may be proposed that Large Language Models (LLM) like
GPT-4, adeptly trained on texts which contain medical papers and articles, have thepotential to enrich the categorization process by inferring from the knowledge it has
previously acquired, despite limited information provided by the event descriptions. The
ensuing section intends to explore the outcomes of applying an LLM (GPT-4 in thisanalysis) to assign polarity to events, with an in-depth analysis of the findings.

4. Event Shock Prediction Using Few-Shot Learning with GPT-4
In this section, we have implemented an experimental setup that leverages thecapabilities of GPT-4's few-shot learning to see if event descriptions have any information
to predict event shocks for the stock market.
The 'Few-shot Universe' dataset used in this experiment included event samples up
until 2021, with a maximum of 200 samples per event type. The test data covered the period
from 2022 to 2023.
The prompt used in the analysis was as follows:###Instruction
Evaluate the impact of the events described in the context on the stock returns of thecompanies, categorizing your response as 'Positive', 'Neutral', or 'Negative', whilereferencing the samples bellow.
### Constraints
You must not provide answers other than 'Positive', 'Negative' or 'Neutral’.
You must respond with only one sentiment.
(few-shot settings)
In Few-shot Universe, event shocks are categorized as 'Positive', 'Neutral', and 'Negative'.
The five samples with the highest similarity to the 'SD_EVENT_DESC' of the test samples
were selected from both the 'Positive' and 'Negative' categories, respectively, based on the'embedding-ada-003' measurement for few-shot prompts.
The Few-shot samples and test data descriptions are given as follows:### Description
1.Event_type: {‘SD_EVENT_TYPE_DESC’}
7

2.PRODUCT_NAME: {‘SD_PRODUCT_NAME’}
3.EVENT_DESCRIPTION: {‘SD_EVENT_DESC’}### SENTIMENT:
Positive (for ‘Positive’ category) or Negative (for ‘Negative’ category)In first step, we look at the characteristics of the output. Figure 3 displays the sentiment
distribution for different event types. It is observed that the output has a bias towards
positive or negative classifications. The low assignment of neutral sentiment may be due tothe lack of few-shot samples which have the neural sentiments. However, even in theabsence of few-shot examples (zero-shot learning), the GPT-4 model rarely expresses
neutral sentiment, which suggests that it may believe drug events generally have some
impact on stock prices. Additionally, positive and negative sentiments are output for all
event types, indicating that the GPT-4 sentiment may be retrieving information beyond theevent type provided in the descriptions.
Figure 3: The Distribution of Sentiments via Event Types

In the subsequent analysis, the sum of market impacts defined in the previous section
was calculated by event type and polarity (Figure 4). Calculations were conducted with
the inclusion of caps and floors, whereby the 10th percentile was set as the floor and the90th percentile as the cap for the range of event shocks. The most discernible impact is8

observed at Phase 3. The sum of event shocks anticipated to be positive exhibits asubstantial positive total, while those predicted to be negative result in a negative total.
This suggests that even for events where the polarity could not be determined based solely
on the event type information, it is possible to assign polarity using LLM. Meanwhile, forevents like regulatory decisions, which were deemed to have a significantly positivecontribution based solely on the event type, positive returns were still observed, even forshocks classified as negative. However, the group of events classified as positive clearlyyielded larger returns, which to some extent validated the outcomes of the classification.
Furthermore, when a test was conducted to compare the mean values of event shocks
between the groups classified as positive and those classified as negative by GPT-4, theresults were significant with a p-value of 0.004, indicating significance at the 1% level.
This offered substantial support for the effectiveness of polarity assignment using GPT-4.
Figure 4: Sum of Positive/Neutral/Negative Event-Induced Shocks by Event Type

Finally, the study computed F1 scores for both positive and negative predictions ofthe estimation results and compared them to other models. The model used forcomparison was as follows:9A fine-tuned BERT with event description as input
LightGBM with Event_type and Drug_Class as categorical data and explanatory
variables for classification
GPT-4 with Drug-Class, which was tested by adding Drug-Class to the prompt
instead of Product-name.
The results are presented in Table 4. The main outcomes exhibit the most balanced
accuracy between Positive and Negative sentiments, being the only instance where both F1scores exceed 0.4. In the three-category classification model for balanced samples, theexpected F1 score would be 0.4 if binary outcomes were predicted randomly and withoutbias. Consequently, an F1 score falling below this threshold would result an absence ofdiscriminative ability, in cases that the outputs are primarily binary, only classified as either
positive or negative. At this point of view, the main result is the only model2 which is able todiscriminate both positive events and negative one.
Additionally, the output trends of the models may indicate the following: models thatuse ‘event_desc’ as text increase the effectiveness of identifying positive shocks; Conversely,models that use ‘event_type’ will be more effective at identifying negative shocks. There areseveral possible reasons for the reduced identification accuracy of Positive Shock withrespect to estimation using 'Drug-Class'. One hypothesis is that the similarity selections offew-shot samples containing 'Drug-Class' information reduce the accuracy of extracting
similar ‘event_descriptions’ that have important information. Further experiments areneeded to test this hypothesis.
Table 4: Comparative F1 Scores for Sentiment Classification by Model Type
P/N2Main

BERTlightGBM

Drug-classPositive

0.45

0.45

0.30

0.38

Negative

0.44

0.29

0.46

0.44

Note: Strictly speaking, it is difficult to make a fair comparison with models that can adjust

the ratio of output labels, as the LLM cannot control this ratio. Here, we are conducting acomparison with traditional models in a simple setting.
105. Conclusion
The analysis of FactSet StreetAccount Drug Events & Pipeline data has revealed
significant insights into the impact of various events within the biopharmaceutical industry
on stock market values.
For event type study, the findings indicate that the market's response to clinical trial
events is generally slow, with no consistent pattern of significant stock price impact
observed in the short term. However, over longer time horizons, certain events such as somePhase clinical trials and regulatory decisions have shown significant impacts on stock prices,highlighting the importance of these events in influencing market valuation.
More importantly, this paper demonstrates the importance of text analysis in thebiopharmaceutical industry and the potential for using advanced large language models toincrease the depth and accuracy of such analysis. Further research and experimentation areneeded to refine these methods and fully realize the benefits of incorporating AI-driven
sentiment analysis into market impact studies, and continued advances in language models
offer a promising means to improve the predictive power and depth of interpretation of
event-driven market analysis provides. For instance, one might employ a domain-specificlanguage model tailored for medical applications, such as Med-PaLM [GOOGLE], oralternatively, a mixture or ensemble approach leveraging both a medical-specific and afinancial-specific language model. Presently, a multitude of methodologies is vying forsupremacy as evidenced by benchmark scores' competition, with future research anticipatedto ascertain the superior algorithm for tasks such as the one in this study.

References
FACTSET. FACTSET STREETACCOUNT DRUG EVENTS & PIPELINE.

OPEN:FACTSET MARKETPLACE. [Online]
https://go.factset.com/marketplace/catalog/product/factset-streetaccount-drug-eventsand-pipeline.
GOOGLE. med-palm. [Online] https://sites.research.google/med-palm/.

11