GPT Deciphering Fedspeak: Quantifying Dissent Among Hawks and Doves
Authors: Denis Peskoff
Adam Visokay
Sander V Schulhoff
Benjamin Wachspress
alan blinder
Brandon M. Stewart
Venue: EMNLP 2023 Findings (2023)
Keywords: FOMC
Fed
GPT
LLM
dissent
TL;DR: We study dissent in the statements and transcripts of the Federal Open Market Committee with GPT-4.

Abstract: Markets and policymakers around the world hang on the consequential monetary policy decisions made by the Federal Open Market Committee (FOMC). Publicly available textual documentation of their meetings provides insight into members’ attitudes about the economy. We use GPT-4 to quantify dissent among members on the topic of inflation. We find that transcripts and minutes reflect the diversity of member views about the macroeconomic outlook in a way that is lost or omitted from the public statements. In fact, diverging opinions that shed light upon the committee’s "true" attitudes are almost entirely omitted from the final statements. Hence, we argue that forecasting FOMC sentiment based solely on statements will not sufficiently reflect dissent among the hawks and doves.

GPT Deciphering Fedspeak: Quantifying Dissent Among Hawks and Doves
Denis Peskoff
Adam Visokay
Sander Schulhoff
Office of Population Research
Sociology
Computer Science
Princeton University
University of Washington
University of Maryland
avisokay @ @uw.edu
@echulho@umd.ce.
Benjamin Wachspress
Alan Blinder
Brandon M. Stewart
Computer Science
Economics
Sociology and OPR
Princeton University
Princeton University
Princeton University
bjw6@ princeton.edu
blinder princeton.edu
bms4 @ @ww.@@perr.iioeoooommm
Abstract
yea
Ecconoy
federal funds
Suuteeeeer federa
före cas
don
ind
federal al open
Markets and policymakers around the world hang on the consequential monetary policy de- cisions made by the Federal Open Market Com- mittee (FOMC). Publicly available textual doc- umentation of their meetings provides insight into members' attitudes about the economy. We use GPT-4 to quantify dissent among mem- bers on the topic of inflation. We find that transcripts and minutes reflect the diversity of member views about the macroeconomic out- look in way that lost or omitted from the public statements. In fact, diverging opinions that shed light upon the committec's "true" atti- tudes are almost entirely omitted from the final statements. Hence, we argue that forecasting FOMC sentiment based solely on statements will not sufficiently reflect dissent among the hawks and doves.
the
WAY
risk
aee open ma ket
n
nkk
inancia
mone
acry po icy
now
Ehange
mar ket Coomi tre
me
questi SniienSeeee
see
nuch
one
Longer
Labor market
c
daT Fanancial per cent
inflation
funds race
going
price stabi
Saar me par teltt 753
Figure 1: The transcripts (left) contains opinions and disagreements but statements (right) are concise. We analyze both datasets using GPTT-4 prompting.
one page statements are released immediately after each FOMC meeting, followed about three weeks later by lengthier minutes, and finally five years later by full, verbatim transcripts. Subsequently this triplet is referred to as documents. We find minutes closely reflect the content of transcripts, so to avoid redundancy, we focus our analysis on transcripts and statements.
1 The Road to FOMC Transparency
The Federal Open Market Committee (FOMC) is responsible for controlling inflation in the United States, using instruments which dramatically affect the housing and financial markets, among others. For most of the 20th century, conventional wisdom held that monetary policy is most effective when docisionnmaking was shrouded in secrecy; the tight- lipped Alan Greenspan, past chairman of the Fed, quipped about "learning to mumble with great in- coherence." " But times change.
Increased FOMC communication has prompted social science research spanning the disciplines of economies, sociology, finance and political science (Section 2). Financial market participants are also keenly interested. Billions, if not trillions of dollars are traded on the Fed's words. The interpretations- right or wrong-of what the FOMC "really means" move markets and affect the economy. However, relying upon documents as data in the social sci- ences is challenge due to the lack of structure and the cost of annotation (Grimmer and Stewart, 2013; Gentzkow et al., 2019; Ash and Hansen, 2023).
Blinder et al. (2008) show how the emergence of greater transparency and strategic communication became an important feature of 21 century central banking. Fed communication is now an integral component of monetary policy, and "Fed watchers" dote on every word. The FOMC first started releas- ing public statements following their meetings in February 1994. This meager documentation grew and now consists of three types for each official meeting: carefully produced and highly stylized
Hansen and Kazinnik (2023) show that Gener- ative Pre-training Transformer (GPT) models out- perform a suite of commonly used NLP methods on text quantification. Motivated by these results, we set out to quantify the language of the FOMC using GPT-4 (OpenAI, 2023) by preparing a com- bined data set of FOMC documents from 1994.

---

Speaker
Speaker 2
Speaker
The gradual increase in core inflation over the past year is a con- cern to me
do not belleve that there is a very great risk of an unmanage- able outbreak of Infla- tion during the rele- vant pollcy horizon.
I think we should not slow our pace of easing moves at this meeting.
PRANSCRIPT
HUMAN: 1.0 Hawkdsh
HUMAN:-0.5 Mostly Dovish GPT-4:
HUMAN: Dovish
GPT-4 0.5 Mostly Hawkish
Neutral
GPT-4; -1.0 Dovish
STATEMENT
Deliberation in the transcript
is obscured in the statement.
The Committee continues to believe that against the background of its long-run goals of price stability and sustalnable economic growth and of the Information currently available, the risks are weighted mainly toward conditions that may generate economic weak- ness in the foreseeable future."
HUMAN:-100
GPT-4: -1.0
Figure Text taken from an FOMC meeting on December 11, 2001. A dovish statement does not reflect the bawkish sentiment of Speaker GPT-4 can quantify dissent lost from transcripts statements.
We conclude that transcripts contain more dissent than statements (Section 3).
2016 (Section 2).'
Mazis and Tsekrekos (2017) apply Latent Seman- tic Analysis to FOMC statements to identify the main "themes" used by the committee and how well they explain variation in treasury yields. Gu et al. (2022) use minutes to investigate how the tonal- ity of committee deliberations impacts subsequent stock market valuations. Political scientists transcripts estimate committee members' prefer- ences on inflation and unemployment (Baerg and Lowe, 2020). Economists have assessed the role of communication in achieving monetary policy objectives by looking at similar documents (Romer and Romer, 2004; Handlan, 2020). Hüpper and Kempa (2023) investigate the extent to which shift- ing inflation focus is reflected in full transcripts. Edison and Carcel (2021) apply Latent Dirichlet Allocation (LDA) to transcripts detect the lution of prominent topics. Hansen al. (2017) use LDA to quantify transcripts and identify how transparency affects the committee's ' deliberations.
2 FOMC Data: Transcript to Statement
The FOMC normally meets eight times per year in order to assess current economic conditions, ulti- mately deciding upon the path for monetary policy, We aggregate and release the official publicly avail- able text documenting these deliberations by the Fed as an aligned corpora of documents from 1994 2016.2 These documents are similar in con- tent, but transcripts and statements are dramatically different in style and detail (Figure 1). For our pur- poses, the statements required no pre- pre-processing For the transcripts, we use regular expressions to partition and then re-aggregate the text by each unique speaker. See Appendix D for example of each document type.
2.1 Lessons from Social Science
Past work has used at most one form of FOMC meeting documentation, but rarely multiple in con- junction. For example. in the finance literature,
2.2 Hawks and Doves
We take the transcripts to best represent FOMC members' underlying attitudes and think of the statements stylized representations of what they wish communicate publicly. To identify disagree- ment, we need to go beyond the statements and look closely at the language employed by members
'https://github. htttp::/ggichuu.ccm//mnissssssss www.DeenisPessoff/Feeelooo hhttps://ww/ www.eeareeeeeeeeee...m
'Transcripts only became being released in 1994 and due to the year lag the latest transcripts are not yet available.

---

in their remarks throughout meeting transcripts. Dissenting voles are rare because of a strong histor- ical norm: members dissent only if they feel very strongly that the committee's s decision is wrong. Modest disagreements do not merit dissent. That said, members of the committee do frequently voice detectable disagreements with one another at meetings. Disagreements frequently concern the state of the economy, the outlook for inflation, and many other things, including where the range for the federal funds rate should be set that day. Such debate and deliberation among members is a rou- tine and productive element of the meetings. These disagreements are more clearly expressed in the transcripts. Daniel Tarullo's comments in a 2016 transcript illustrate the point:
Nothing since then has increased my con- fidence. To the contrary, a few more doubts have crept in."
If the meeting statement was an accurate represen- tation of what transpired at the meeting, it would follow that the uncertainty of the committee regard- ing their inflation forecast would be communicated. Instead, the diverging individual opinions are omit- ted in the final statement, where:
"The Committee currently expects that, with gradual adjustments in the stance of monetary policy, economic activity will expand al moderate pace and la- bor market indicators will continue to strengthen. Inflation is expected re- main low in the near term, in part be- cause of the further declines in energy prices, but to rise to percent over the medium term as the transitory effects of declines in energy and import prices dis- sipate and the labor market strengthens"
"it is institutionally important for us to project an ability to agree, even if only at fairly high level, and that is why ab- stained rather than dissented over each of the past several years [...] have gone out of my way in the past four years not to highlight publicly my points of differ- ence with the statement."
We apply this level of granular analysis to create a gold label for statements, classifying each one according to Hawk and Dove definitions proposed by Hansen and Kazinnik (2023): Dovish (-1.0), Mostly Dovish (-0.5), Neutral (0), Mostly Hawkish (.5), Hawkish (1.0). A trained undergraduate does first pass and escalates any borderline cases a FOMC expert for adjudication. We manually review the January 2016 transcript and pair it with a simple computational analysis, which finds that most dissent at that time surrounded topics relating to inflation (Appendix C).
2.3 Manual Analysis to Create a Gold Label
Dissent amongst speakers is normally concentrated on the discussion of the economic and financial situation of the U.S. specifically inflation target- ing. For example, in the January 2016 meeting transcript, the committee discusses their 2 percent inflation projection in the context of factors such as oil prices, the job market, and the Chinese economy. Many of the members argue that the inflation pro- jection of 2 percent will not be accurate, while oth- ers who do support the 2 percent projection qualify their support with varying degrees of uncertainty. In this meeting, President Mester concludes,
2.4 GPT-4 "Reads" Terse Documents
GPT-4, and Large Language Models more broadly. suitable tool for rapid linguistic processing at scale. We produce three different measurements of hawk/dove sentiment using statements and two using transcripts. Hansen and Kazinnik (2023) use GPT-3 to quantify 500 sentences selected uni- formly at random from FOMC statements between 2010 and 2020. We extend this by using GPT-4 to quantify all 3728 sentences in statements from 1994 to 2016 (Appendix B). A limitation of this approach is that the holistic sentiment of the meet- ing not captured because each sentence is scored context.
"My reasonable confidence that inflation will gradually return to our objective over time recognizes there and has al- ways been large uncertainty regarding inflation forecasts."
While Mr. Tarullo argues in opposition,
"... didn't it have reasonable confidence that inflation would rise to 2 percent.
independently-
"Notably, this is not true of the monetary policy com- mittees of other nations; on some of them, dissent is com- mon- expected (Tillmann, 2021).
The first statement measurement we propose is to simply take an unweighted mean of all individual

---

Methodology
1.0
Manual: Overall
0.8
GPT4; By Sentence
GPT4: Overall
0.6
Hawk-Dove Score
0.2
Hawkish Dovish
-0.0
-0.2
-0.4
-0.6
-0.8
1994
1996
1998
2000
2002
2004
2006
2008
2010
2012
2014
2016
Year
Figure 3: Our analysis of statements finds that averaging at the sentence-level (gray) loses information since the average sentence Neutral. Ingesting the overall statement (blue) better mirrors the manual gold label (gold).
Proportion of Scores by Speaker
0.8
Hawk Dove Score
0.6
Mostly Hewkltsh
Neutral
Mostlly Dovish
Bosteh
0.4
0.1
0.0
1994
1996
1998
2000
2002
2004
2006
2008
2010
2012
2014
2016
Year
Figure 4: Our analysis of transcripts with GPT-4 the speaker level shows that meetings consistently have dissenting opinions. Similarly to the previous figure, 2001 to 2004 is dovish (blue), while 2004 to 2006 is hawkish (orange).
sentence scores for cach meeting. Because most sentences have nothing to do with inflation (62% of sentences scored as Neutral), we hypothesize that this method has further limitations. We resolve this by ingesting each entire statement into our GPT-4 prompt. Both of these measurements, along with manual gold labels, can be seen in Figure 3.
ish and dovish scores, respectively. In this approach g(*)) ignores sentences scored as Neutral, placing more emphasis on the relative rather than abso- lute differences between hawkish and dovish sen- timent. Furthermore, since (LL) has no predefined end points, this allows us to generate positions at any level of extremity, which more appropriately reflects the outlier meetings.
For the final statement measurement, we con- struct logit-scaled score (Lowe et al., 2011),
When measuring hawk/dove sentiment using transcripts, the amount of text adds an addi- tional challenge. Rather than evaluating 100+ page transcript at the sentence level, we instead evaluate transcripts at the speaker level. That is to
(Hawk + 0.5
g(L) logg
Dove + 0.5
where Hawk and Dove are the sums of the hawk-

---

say, we use GPT-4 quantify hawk/dove sentiment for each distinct speaker within each transcript, and then aggregate those by meeting date. contrast to the 3728 individual sentence observations across all statements, we evaluate 5691 speaker observa- tions across all transcripts. We then follow the same steps outlined above to calculate unweighted average and logit-scaled scores.
3.2 Conclusion and Next Steps
Our method of ingesting the entire statement for an aggregate prediction better captures the extremes, which more closely mirror the gold label human annotation and suggests that Large Language Mod- els can avoid the noise in this nuanced context. The Fi score for this comparison is 0.57. While this rather low as a measure of model "fit", it is important to note that the results rarely flip senti- ment (from hawkish to dovish, or mostly hawkish to mostly dovish), rather, it just seems to mostly disagree on adjacent categories. See Figure 3 for a visual comparison of the sentence-level, entire text. and manual scores. Of note, the inconsistent pro- vision of statements and relatively high volatility in hawk/dove sentiment before 2000 consistent with Meade and Stasavage (2008) and Hansen et al. (2017) who have also studied the 1993 change in FOMC communication strategy. We demonstrate that GPT-4 can identify the extremes in dissenting hawk and dove perspectives despite the indications of clear consensus in the statements. This empiri- cal finding supports our manual analysis.
The logit-scaled measurements for transcripts and statements track one another quite closely over most of the years, demonstrating that GPT-4 is effective at identifying similar content across doc- ument types of dramatically different length and style. It is worth noting, however, that the logit- scaled transcript scores have larger extremes than statements, especially the upper bound: (-3.66, 3.37) and (-3.40, 2.40), respectively. This under- scores how the FOMC devotes considerable atten- tion to curating their communication strategy to convey confidence and unity despite dissent among the hawks and doves. The logit-scaled scores begin to diverge in 2012, with transcripts trending more neutral/hawkish while statements remain mostly dovish. Hence, we propose direct comparison.
While we focus on transcripts and statements, fu- ture work may consider an even more fine grained analysis, incorporating minutes as well. We found the content of the minutes to more closely resemble the transcripts than the statements, but differences do exist and remain underexplored.
3 LLMs Quantify Economic Text
3.1 Measuring Dissent
We can use the sentence-level statement and speaker-level transcript scores from GPT-4 to com- pute measure of dissent for each meeting using the following algorithm:
Additionally, we note that GPT-4 scores made more neutral predictions than the gold standard manual labels. To improve upon this, we created a balanced few-shot example using sentences from FOMC statements not included in our sample meetings since 2020. This marginally improved the prediction "fit" (Fi of 57% to 58%), but we expect that this could be improved much further with additional prompt engineering.
1. From the list of scores for each meeting, count the number of hawkish/mostly hawkish and the number of dovish/mostly dovish scores. 2. If there is at least one hawkish score and at least one dovish score within the same meet- ing, assign Dissent
Else, Dissent = O.
We find that 47% % of statements and 82% of tran- scripts contain dissent. We also compute the con- ditional probability of transcript containing dis- sent given the associated statement binary. P(T = i|s = 1) and P(T = 1|$ = 0). We find that when a statement contains dissent, P(T = 1|S = 1). the transcript agrees more than 97 % of the time However, for statements scored as having no dis- sent, P(T 1|$ 0) we find that more than 69% % of associated transcripts are scored as containing dissent. This means that for the 53% of statements that don't show signs of dissenting opinions, there is likely dissent in the transcript as evidenced by the speaker-level hawk/dove scores.
GPT-4 is able to quickly quantify stylized eco- nomic text. Our results from quantifying dissent support the hypothesis that dissenting opinions on the topic of inflation omitted from FOMC state- ments can be found in the associated transcripts. As LLMs continue to improve, we expect that it will be possible study even more nuanced questions than the ones we answer here.
Limitations
Substantively, strategic signaling in the FOMC is a challenging topic and this is only an initial inves- tigation. Dissent does not have clear ground truth

---

labels and thus we are reliant on human judgment and our team's substantive expertise on monetary policy. Finally, as with much current research, our work relies on OpenAT's GPT API, which poses challenges to computational reproducibility, as relies on the stability of an external system that we cannot control.
Nicole Baerg and Will Lowe. 2020. A textual taylor rule: estimating central bank preferences combining topic and scaling methods. Political Science Research and Methods, 11)):10--12.
Alan Blinder, Michael Ehrmann. Marcel Fratzscher, Jakob De Haan, and David-Jan Jansen. 2008. Central bank communication and monetary policy: A sur- vey theory and evidence. Journal of Economic Literature, 46(4):910-45.
Ethies Statement
Hali Edison and Hector Carcel. 2021. Text data anal- ysis using latent dirichlet allocation: an application to fome transcripts. Applied Economics Letters, 28( 1):3844..
The FOMC is a high-stakes body whose activities are already subject to substantial scrutiny. There is some ethical risk in exploring linguistic signals of hidden information. For example, based on the substantive literature we believe that dissent is in- up
Matthew Gentzkow, Bryan Kelly, and Matt Taddy. 2019. Text as data. Journal of Economic Literature. 53577535-77
tentionally signalled in meetings in order future discussions or lay claims to particular po- sitions. Nevertheless, attribution of intention (as implied by 'dissent') always involves some level of error that could be uncomfortable to meeting participants who feel mischaracterized. We also emphasize that our approach to capturing dissent would not be appropriate to use outside this spe- cific context without careful validation. Finally, by making the FOMC data more easily available to the NLP community, we also assume some ethical responsibility for the potential uses of that data (see e.g. Peng et al., 2021). We spend under $1500 on computation and under $1000 on annotation and believe our results to be reasonably reproducible. We feel that these concerns are ultimately minor given that all participants are public officials who knew their transcripts would ultimately be released.
Justin Grimmer and Brandon M. Stewart. 2013. Text as data: The promise and pitfalls of automatic con- tent analysis methods for political texts. Political Analysis, 21(3):267-29
Chen Gu, Denghui Chen, Raluca Stan, and Aizhong Shen. 2022. It is not just what you say, but how you say it: Why tonality matters in central bank commu- nication. Journal of Empirical Finance, 68:216-231.
Zihan Guan, Zihao Wu, Zhengliang Liu, Dufan Wu, Hui Ren, Quanzheng Li, Xiang Li, and Ninghao Liu. 2023. Cohortgpt: An enhanced gpt for participant recruitment in clinical study.
Amy Handlan. 2020. Text shocks and monetary sur- prises: Text analysis of fome statements with ma- chine learning. Published Manuscript.
Anne Lundgaard Hansen and Sophia Kazinnik. 2023. Can chatgpt decipher fedspeak? Available at SSRN.
Stephen Hansen, Michael McMahon, and Andrea Prat. 2017. Transparency and Deliberation Within the FOMC: A Computational Linguistics Approach*, The Quarterly Journal of Economics, 133(2):801- 870.
Acknowledgements
This material is based upon work supported by the National Science Foundation under Grant #2127309 to the Computing Research Association for the CIFellows 2021 Project. We thank the Ini- tiative for Data-Driven Social Science at Princeton University for grant support and OpenAT for tech- nical support. Visokay is supported by the National Institute of Mental Health of the NIH under Award Number #DP2MH122400 and by the Center for Statistics and the Social Sciences at the University of Washington.
C. Hutto and Eric Gilhert. 2014. Vader: A parsimonious mule-baeed model for sentiment analysis of social media text. Proceedings of the International AAAI Conference on Web and Social Media, 8(1):216-225.
Florian Hüpper and Bernd Kempa. 2023, Inflation tar- geting and inflation communication of the federal reserve: Words and deeds. Journal of Macroeco- nomics. 75:103497.
Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang- Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He. Zhengliang Liu. Zihao Wu, Lin Zhao, Dajiang Zhu, Xiang Li. Ning Qiang. Dingang Shen. Tianming Liu, and Bao Ge. 2023. Summary of chatgpt-related research and perspective towards the future of large language models. Meta- Radiolooy 1(2):10001) 10001..
References
Elliott Ash and Stephen Hansen. 2023. Text algorithms in economics. CEPR Discussion Paper, DP18125.

---

Will Lowe, Kenneth Benoit, Slava Mikhaylov, and Michael Laver. 2011. Scaling policy preferences from coded political texts. Legislative Studies Quar- terly, 36(1):123-155:
A Appendix
Our appendix contains the GPT prompt we used, section
computational analysis, and ample of the statement (Figure 6), transeript (Fig- ure 7), and minutes (Figure 8) for the January 29- 30, 2008 meetings.
P. Mazis and Tsekrekos. 2017. Latent semantic anal- ysis of the fome statements. Review of Accounting and Finance, 16:179-217.
Ellen E. Meade and David Stasavage. 2008. Publicity of Debate and the Incentive Dissent: Evidence from the US Federal Reserve. The Economic Journal, (186578)695--177
B GPT-4
We used GPT-4 heavily for our experiments and analysis. Recent work (OpenAI, 2023; Liu et al., 2023; Guan al., 2023) that successfully uses GPT- 4 for classification gave confidence in its quality. Additionally, human expert
OpenAl. 2023. Gpt-4 technical report.
Kenny Peng, Arunesh Mathur, and Arvind Narayanan. 2021. Mitigating dataset harms requires steward- ship: Lessons from 1000 papers. arXiv preprint arXiv:2108.02922.
team examined GPT-4 generated labels, and found that in sample of 25, our expert agreed with 19 labels with high confidence, and 22 labels with at least moderate confidence.
Christina D Romer and David H Romer. 2004. new measure of monetary shocks: Derivation and implica- tions. American economic review, 94(4):1055-B 1084.
B.1 Methodology
Peter Tillmann. 2021. Financial markets and dissent the ecb's governing council. European Economic Review, 139: 103848.
We used variants a single prompt template for all of our tasks. It contains the relevant labels (Hawk- ish, Dovish, etc.) as well as their definitions. It includes space for some INPUT and asks which label best applies the input. When processing different documents, such transcripts, would switch the word statements in the prompt for the appropriate document word.
For statements, we 0-shot prompted GPT-4 to la- bel each statement as one of the five labels. We also ran an experiment in which we classified each sen- tence of each statements as one of the labels, then averaged the sentence scores get statement score. Finally, we reran this sentence classification with a 10-shot prompt. The prompt was similar below, except with 10 examples of sentences and their classifications at the beginning.
For minutes, we 0-shot prompted GPT-4-32K to label each statement as one of the five labels.
For transcripts, O-shot prompted GPT-4-32K to examine all of each speakers speech, and provide each speaker single label for each transeript.
For all API calls to OpenAI, we only modified the model (either GPT-4 or GPT-4-32K), We did not change any other settings.
prompt_template
<statement>
INPUT
</tatmment>>
<labels

---

Dovish:
Strongly expresses belief that the economy may be growing too slowly and may need stimulus through mon-
sentiment of each sentence within the nine topics. Since VADER is trained on web-based social me- dia content, which is typically more abrupt than the formal language appearing in the FOMC transcript, we conducted the sentiment analysis by sentence to optimize the method's performance.
etary policy.
Mostly dovish: Overall message expresses a belief that the economy may
To analyze dissent more specifically, com- puted the fraction of negative sentences in each topic. For this analysis, ativity score be 0.1. That is, sentences with a negativity score of 0. or higher were classified negative while all others were not. This number determined by manually reviewing what sentences were captured by varying thresholds and evaluat- ing whether or not they conveyed dissent. When the threshold was set too low (0.05), four out of ten randomly selected sentences conveyed dissent. When too high (0.15), seven out of ten randomly selected sentences conveyed dissent, but many sen- tences that indicated dissent were omitted. At the threshold of 0.1, still seven out of ten randomly selected sentences conveyed dissent, and more sen- tences that conveyed dissent were captured.
be growing too slowly and may need stimulus through monetary policy.
set the threshold neg-
Neutral: Expresses neither a hawkish nor dovish view and is mostly objective.
Mostly hawkish: Overall message expresses a belief that the economy is growing too quickly and may need to be slowed down through monetary policy. Hawkish: Strongly expresses a belief that the economy is growing
too quickly and may need to be slowed down through monetary policy.
D Document Examples
See Figures 7, and 8 for examples of the docu- ments.
</labels>
Which label best applies applies to the statement (Dovish, Mostly Dovish, Neutral, Mostly Hawkish, Hawkish)?
C Computational Analysis
We paired our manual review of the January 26-27, 2016 transcript with computational analysis of dissent in the meeting. We stratified the meeting into nine topics, each corresponding to portion of the transeript content. As baseline, we counted the number of speakers in each section see this metric could reflect dissent. This technique, however, seemed to reflect the length sation as opposed to the degree to which members disagreed with one another.
conver-
Our next approach was to do sentiment anal- ysis of each topic to see if the prevalence of neg- ativity could indicate dissent. We supposed that negative sentiment would be high if the speakers opposed the stance either other individuals or the committee as a whole. Using the VADER lex- icon (Hutto and Gilbert, 2014) 4), we calculated the

---

Topic
Cont: Bconomic and Financial Situation with Vollents findt remories
Roloese roganding Statemont Langen-Ruri
Eronomic and Finaneial Situition
Fan Charts
Ratio Negative Sentences
0.25
Develspments Open
Hnusekseping
Nateria Tcr Briefing o Monetary Palicy A taanative
0.20
Overnight Martrees, Banwing Regu
Statement un Knngorge Run Gods Mor Pulkcy
0.15
Mead
0.10
0.05
0.00
Figure 5: Discussion of inflation (Red and Blue topics) is more contentious than other topics and the average of all data (dotted line).
Press Release
January 27, 2016
Federal Reserve issues FOMC statement
For release at 2:00 p.m. EST
Share
Information received since the Federal Open Market Committee met in December suggests that labor market conditions improved further even economic growth slowed late last year. Household spending and business fixed investment have been increasing at moderate rates in recent months, and the housing sector has improved further; however, net exports have been soft and inventory investment slowed. A range recent labor market indicators, including strong job gains, points to some additional decline underutilization of labor resources. Inflation has continued run below the Committee's 2 percent longer- run objective, partly reflecting declines energy prices and in prices of non-energy imports. Market-based measures inflation compensation declined further, survey-based measures of longer-term inflation expectations little changed, balance, in recent months.
Consistent with its statutory mandate, the Committee seeks to foster maximum employment and price stability. The Committee currently expects that, with gradual adjustments in the stance of monetary policy, economic activity will expand at a moderate pace and labor market indicators will continue to strengthen. Inflation is expected remain low the near term, in part because the further declines energy prices, to rise 2 percent over medium term as transitory effects of declines energy and import prices dissipate the labor market strengthens further. Committee closely monitoring global economic and financial developments and is assessing their implications for the labor market and inflation,
Figure Released statement for the January 26-27, 2016 meeting.

---

January 26-27,200 2016
155 of
CHAIR YELLEN. So maybe it should be "the Committee."
MS. MESTER. Right.
CHAIR YELLEN. Let's try that again: "the Committee's inflation goal is symmetric,
but believed that the statement should more clearly express that the Committee is primarily"-
MR. LACKER. Right.
MS. MESTER. Right.
MR. WILLIAMS. No.
MR. EVANS. You're starting talk about Committee actions at that point.
PARTICIPANTS. It's the statement.
CHAIR YELLEN. "But believed that the statement should more clearly express that"-
what? Wth Committee is primarily focused"
MR. LOCKHART. "That the Committee's 's primary focus be on expected futtrre
deviations."
MR. WILLIAMS. No.
MR. LACKER. Madam Chair, could try the passive voice, because avoids that. "Id
should more clearly express focus on" or "a primary focus on."
MR. POWELL. Dan's language avoided this whole mess.
MR. WILLIAMS. Yes. We were good before.
MR. POWELL. "But believed the Committee's 's language is not adequately," or maybe
"the statement language is not adequately focused on expected future deviations."
VICE CHAIRMAN DUDLEY. How about "sufficiently" rather than "adequately"?
CHAIR YELLEN. "That statement language not"
MR. POWELL. ""sufficiently" or "adequately focused expected" -t cetera.
Figure 7: A page the January 26-27, 2016 meeting transcript that shows dissent.

---

Minutes of the Meeting of January 2012 2016
Page 9
Infiation, employment, and long-term interest finettate over time in response economic and finan- cial distubaacces. Morsover, monetary policy actions tend to infiuence economic activity and prices with lag. Therefore, the Committee's policy decisions reflect its longer-mu goals, its medium term outlook, and its sessments of the balance of risks, including tisks to the financial system that could impede the attainment of the Committee's goals.
projected to return levels judged consistent with its mandate.
The Committee intends to reaffirm these principles and to make adjustments appropriate at annual or- ganizational meeting each January." "
All Committee members but one voted to adopt the re- vised statement. Although Mr. Bullard supported the statement without the changes and agreed that the Com- mittee's is infaatio goal is symmetric, dissented because he judged that amended language focused expected future deviations of inflation from the 2 percent objective. In addition, because the Com- mittee's s past behavior had demonstrated the emphasis it places on expected future inflation, Mr. Bullard viewed amended language as potentially confusing to the public.
The infiation rate over the longer rn is primarily de- termined by monetasy policy, and hence the Committee has the ability to specify longer-rum goal for inflation. The Committee reaffirms its judgment that inflation at the rate of 2 percent, measured by the annual change the price index for personal comsumption expendi- tures, most consistent over the longer run with the Federal Reserve's statutosy mandate. The Committee would be concerned if infiation were running persis- tently abore or below this objective. Communicating this symmetric infiation goal cleady to public helps keep longer-term inflation expectations firmly anchozed, thereby fostering price stability and moderate long-termn interest rates and enhancing the Committee's 's ability to promote maximum employment the face of signifi- cant economic distubancess. The maximum level em- ployment is largely determined by nommonetary factors that affect the structure and dynamies of the labor mar- ket. These factors may change over time and may not be directly mexmuaale. Consequently, it would not be appropriate to specify a fixed goal for employment; ra- ther, the Committee's policy decisions must be informed by assessments the mıximum level of employment, recognizing that such assessments necessarily uaces- tain and subject to revision. The Committee considers wide range of indicators in making these assessments. Information about Committee participants' estimates of output growth and unem-
sufficiently
Developments in Financial Markets, Open Market Operations, and Policy Normalization The SOMA manager reported on developments in do- mestic and foreign financial markets, including changes in the expectations of market participants for the trajec- tory of monetary policy. The deputy manager followed with briefing on money market developments and Sys- tem open market operations conducted by the Open Market Desk during the perio since the Committee met on December 15-16, 2015. The report included an as- sessment of the response of money market interest rates to the increase in the taget range for the federal funds rate announced following December meeting. Over- all, the rate increase was implemented smoothly and money markets responded as anticipated. Take-up of overnight reverse repurchase agreement (ON RRP) op- erations over this period was consistent with that ob- served in the testing phase of operations over the second half of last year. The deputy mamager also reviewed plans for reinvestment of the proceeds of upcoming matuations of SOMA holdings of Treasuyy securities, for small-vaine tests of vatious System operations and faciiities during 2016, and for quartedly tests of the Temm Deposit Pacility.
the longer-run nomal ployment is published four times per year in the POMC's Summary of Economic Projections. For ex- ample, in the most recent projections, the median of POMC participants' estimates of the longer-run normal zate of unemployment was 4.9 percent.
In setting monetary policy, the Committee seeks to mitigate deviations of inflation from its longer-run goal and deviations of employment from the Committee's 's sessments of its maximum level. These objectives generally complementary However, under cireun- stances in which the Committee judges that the objec- tires are not complementay, it follows a balanced ap- proach in promoting them, taking into account the mag- nitude of the deviations and the potentially different time hozizons over which employment and inflation are
The Committee then resumed its consideration of mat- tess related to the System's reverse repurchase agree- ment (RRP) facilities, foeusing in partieular on the xp- propriate aggregate capacity of the ON RRP facility go- ing forward. Previous communications had indicated that the Committee intended to allow aggregate capacity of the ON RRP faellity to be temporarily elevated after policy firming had commenced to support monetary policy implementation and expected that would be xp- propriate to reduce capacity faidy soon thereafter. A
Figure 8: A page of the January 26-27, 2016 meeting minutes that discusses inflation targeting.


Reviews:
[{"review_id": "dB7Dg7QYyK", "reviewer": "Reviewer_dn37", "summary": "", "questions": "* Could you explain what measures or systems we could use to verify the quality of the use of GPT in such an analysis (in terms of reproducibility)? I was wondering if you have also tried an open-source system that provides more transparency etc.?\n* The phrase 'escalates any borderline cases' in your annotation process is a bit vague; could you provide more clarification on this aspect and the annotation process?\n* Could you expand on the comparison between manual and GPT-based annotations in your work? What kinds of differences/disagreements did you mainly observe in their evaluation?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 1 for the clarifying questions and constructive criticism.  We agree that studying dissent on inflation in the Federal Reserve is highly relevant and that creating a resource for the NLP community would allow us to better understand it.  \n\nWe wholeheartedly agree with the need for quality annotation (*Reasons to Reject #1, #4, Question #2*).  We manually review every FOMC issued statement as a whole, evaluating the overall Hawkishness or Dovishness of the document (*Section 2.3*). We then compare these manual annotations to the GPT-based annotation. Of the two GPT methods, prompting to evaluate statements based on the average of its individual sentences and prompting it to evaluate the statement as a whole, the latter matched more closely to the manual annotation. The main disagreement we observed was that the GPT evaluation would label sentences as more neutral than the manual analysis concluded.   Additionally, we manually read a transcript (*Lines 185-188, Appendix C, Figure 4*).  Since the transcript is hundreds of pages in length, this required five hours and specialized knowledge of the Federal Reserve, making the process intractable to repeat for all transcripts in the same way as for statements.  \n\nEscalation on borderline cases was intentionally vague in order to preserve anonymity, and was done from a sociology undergraduate to a graduate student. Results of the annotation were discussed with a leading expert, omitted in this version for the sake of anonymity.  Most cases are obvious so this allows scalability while ensuring a genuine gold standard (15 cases needed escalation, of which 2 were escalated to the leading expert).  We strongly agree with the author’s request to include a more detailed analysis of disagreements between GPT and human annotation. We will take this advice and prioritize explaining the annotation process and the analysis in the additional 1-page. \n\nWe ground our work in existing literature, including Hansen and Kzinnik (2023).  Given the close proximity of their contributions and ours, we agree with the reviewer that it is important to emphasize our distinct contribution which we will do so in the revised version.  In short, Hansen and Kzinnik test the application of multiple models, including GPT-3 and GPT-4, on several different language tasks using FOMC materials. We do a robust analysis on a more specific task - quantifying dissent among the FOMC. Where they annotate only 500 individual statements selected at random from statements over a 10 year period and have no time dimension in their analysis, we go much further. First, we demonstrate an improvement over their methodology (Figure 3A) by evaluating the statement as a whole. Second, we do this at a considerably larger scale, annotating all 3728 individual sentences in statements from 1994-2016 compared to their 500 sentences at random.  Third, we create a gold standard to assess their methodology using non-trivial manual annotation. Fourth and most important, we introduce a previously unstudied dimension and dataset, by analyzing transcripts and minutes, which are magnitudes larger than just the statements and evaluating changes over time.  While the statements are short enough that economists could plausibly use manual annotation to investigate, we demonstrate the feasibility of using GPT-4 to vastly expand the quantitative analysis of Federal Reserve deliberations. As a side benefit, our methodology demonstrates a powerful new scale of analysis for social science research more broadly.  \n\nWe agree that a longer paper format would be excellent (*Reasons to Reject #3*).  However, since this is building on past work, we focused our contribution on dissent within the Federal Reserve, which should be possible to convey with the additional page for the final version. To deal with the page limit, we optimized space in this submission and submitted a 5 page appendix (with real examples of our data, technical details of prompting, and a sanity check to support the findings of our manual analysis of the transcripts).  We will include any additional details that arise from the additional requested analysis.\n\nWe understand and share your interest in open-source alternatives (*Question #1*). A human expert on our team examined GPT-4 generated labels, and found that in a sample of 25, our expert agreed with 19 labels with high confidence, and 22 labels with at least moderate confidence. This, as well as recent work that successfully uses GPT-4 for classification (References [1][2][3] below) gave us confidence in its quality. Reproducibility is still an issue; although we set Temperature and Top-P to 0, we understand that randomness in GPU computations still causes variability in responses. Additionally, OpenAI models change over time. As such, it may not be possible to exactly reproduce these results. We experimented with open source models such as FLAN-T5-XXL, but found that the limited performance and context length rendered it insufficient for our task. We are optimistic about newer open models such as Llama 2, but at the time of submission we feel that GPT-4 is the strongest viable option. \n\nReferences:\n\n[1] OpenAI. GPT-4 Technical Report. 2023.\n\n[2] Liu et al. Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models. 2023.\n\n[3] Guan et al. CohortGPT: An Enhanced GPT for Participant Recruitment in Clinical Study. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* relevance of the problem: The paper addresses a relevant research topic - understanding the dynamics of dissent within the Federal Open Market Committee, through the use of advanced technology in the form of GPT-4\n* great resource: expansion of a previous work by Hansen and Kazinnik (2023), by extending the analysis to 3728 sentences (1994-2016), providing a more solid grounding for a comprehensive understanding of dissent within the FOMC", "reasons_to_reject": "* lack of quality measure: for me, a significant drawback of the paper is the absence of quality measures for sanity checks. The use of GPT-4 is indeed promising; however, without a proper system for verifying the quality and accuracy of the results, there is a risk of drawing inaccurate conclusions. Such a measure would have ensured the reliability of the findings and added more credibility to the analysis.\n* incremental nature of the paper: the paper's research is an extension of the work by Hansen and Kazinnik (2023), with the primary difference being the shift from GPT-3 to GPT-4. The contribution could be seen as relatively incremental.\n* (format and depth of analysis: given its complexities and the volume of data considered, the study would benefit from a longer paper format. A more comprehensive format would allow for a more in-depth analysis, discussion of potential limitations, and more detailed contextualization of GPT-4 compared to other models) - for the short paper version, this is just an optional aspect or something for the appendix\n* annotation: I think the study could be enriched further by an extensive comparison between manual and GPT-based annotations, examining aspects such as error types and agreement or others to quantify any potential discrepancies. Furthermore, a more detailed description of the annotation process would be beneficial, as terms like 'escalates any borderline cases' are somewhat ambiguous and leave room for clarification.", "questions_to_authors": "* Could you explain what measures or systems we could use to verify the quality of the use of GPT in such an analysis (in terms of reproducibility)? I was wondering if you have also tried an open-source system that provides more transparency etc.?\n* The phrase 'escalates any borderline cases' in your annotation process is a bit vague; could you provide more clarification on this aspect and the annotation process?\n* Could you expand on the comparison between manual and GPT-based annotations in your work? What kinds of differences/disagreements did you mainly observe in their evaluation?", "paper_topic_and_main_contributions": "This paper presents a novel study to understand patterns of dissent among members on the topic inflation within the Federal Open Market Committee (FOMC) by using GPT-4.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ONn5lAqKqN", "reviewer": "Reviewer_TSKC", "summary": "", "questions": "A: Figure 3 has NBER recessions as grey bars. It would be helpful to note that.  Related, one of the colors of the lines is also grey and that can lead to confusion.\n\nB: Figure 3 and in general, define \"hawk\" and \"dove.\"  (And a few sentences explaining why people might have a different opinion on how aggressive monetary policy should be towards inflation might help the audience.  Perhaps how this is different/similar to \"sentiment\".\n\nC:  It might be helpful to include some measure of inflation over this history.  (It seems odd in 2023, but in some of those meetings, they are arguing about how to _increase_ inflation to the 2% goal).\n\nD: Similarly, much of Fed policy in 2008 was not focussed on (just) inflation.  That was the period of \"unconventional monetary policy\" related to the financial crisis (2007 to 2009).\n\nE:  What is the rationale for: \"Our analysis of transcripts is of an unprecedented scale relative to past social science work.\"  There such a large amount of economics about monetary policy and many scholars have quoted from the transcripts.  Or, research that uses the text in the Beige Book (or other Fed forecasts books).  ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 2 for their detailed review.  We agree that monetary policy is extremely important and that analyzing transcripts with GPT is an important application.  \n\nWe acknowledge that “The paper is an application of GPT-4 and not producing a new nlp model.”  While we could create a new model, it is not obvious to us that this would be preferable to creating prompts for an existing model.   We are not able to train a model as large as GPT (or any other comparable large language model).  Furthermore, use of GPT-4 makes replications easier. That said, we are organizing data for NLP purposes, which could ultimately be used in training the next generation of larger language models.  The technical language of the Federal Reserve is not likely to be found in standard common crawl resources.  We release all our annotations (both manual and by GPT), as well as the prompts.  \n\nWe will work to clarify in future drafts why the overall conclusions should be seen as compelling. We introduce the hypothesis that there is dissent in transcripts that is not present in statements—a key point missed by the prior, near exclusive, focus on statements.  We support this hypothesis with quotes found through manual reading of the transcripts (*Lines 132-139, 144-176*).  Figure 3b empirically shows that both the peak of hawkish and dovish sentiment occurs in the transcripts rather than the statements (*Lines 264-267*). Additionally, this entire experiment is a successful use of large language models at scale for an important social science question which itself provides additional evidence for the growing body of work emphasizing the feasibility of these approaches.\n\nTo answer the questions:\n1. We note this confusion and thank you for your feedback. \n2. This is a good suggestion to improve readability.  (Please see *Lines 178-182* for a reference to our definition of Hawk and Dove labels). With the additional 1 page of space, we will write out the  Hansen and Kazinnik (2023) definitions in the intro and reference them in Figure 3.  \n3. We agree that it might be interesting to note how inflation changed over time, but our analysis focused on how dissent was captured, more generally, in meeting transcripts and statements. Inflation was a key topic during many of these meetings (see Figure 4 in the appendix), but changes in the inflation rate were not the main focus of our paper.  We will investigate to see if the actual inflation for the year is interesting relative to the discussions.  \n4. We think this would be excellent follow-up work.  We used inflation as an example for how dissent in the transcripts was diluted or omitted in the statement but did not exclusively evaluate Hawkishness or Dovishness in relation to inflation. (Real growth was also relevant.) We considered different areas of investigation but thought inflation would be the best starting point given the existing literature and our desire to demonstrate the scalability of large language models for this task; more niche discussions such as \"unconventional monetary policy” would only apply to recent years rather than to the entire transcript history.  \n5. You inquired about our use of unprecedented scale.  We mean for this problem, no other work that has looked at every word in every transcript. One of our contributions is creating a combined data set of all three FOMC documents (statements, minutes, and transcripts) in coordination with Federal Reserve staff. With this, we were able to conduct the first analysis of multiple types of FOMC documents (statements and transcripts) from 1994 to 2016. No analysis has previously analyzed these documents in tandem or with this quantity of data. Given the venue, we hope this will encourage other work in social science.  We will clarify this in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The annotated dataset has lots of interesting applications\n- Monetary policy is an important topic.  In particular, the transcript that documents decision-making and the statement of the \"consensus\" is a rich setting for NLP models.\n- documenting that the transcript and statement have similar sentiment (in Fig 3b) is interesting.\n- The use of the GPT is clearly explained and the fact it works well (relative to labeled data) at capturing the sentiment is interesting.", "reasons_to_reject": "- The paper is an application of GPT-4 and not producing a new nlp model.\n- The overall conclusions of the paper are not compelling. For example:  What do we learn from Figure 3b?  Why is this relevant? Does the transcript refect differences of opinion?  ", "questions_to_authors": "A: Figure 3 has NBER recessions as grey bars. It would be helpful to note that.  Related, one of the colors of the lines is also grey and that can lead to confusion.\n\nB: Figure 3 and in general, define \"hawk\" and \"dove.\"  (And a few sentences explaining why people might have a different opinion on how aggressive monetary policy should be towards inflation might help the audience.  Perhaps how this is different/similar to \"sentiment\".\n\nC:  It might be helpful to include some measure of inflation over this history.  (It seems odd in 2023, but in some of those meetings, they are arguing about how to _increase_ inflation to the 2% goal).\n\nD: Similarly, much of Fed policy in 2008 was not focussed on (just) inflation.  That was the period of \"unconventional monetary policy\" related to the financial crisis (2007 to 2009).\n\nE:  What is the rationale for: \"Our analysis of transcripts is of an unprecedented scale relative to past social science work.\"  There such a large amount of economics about monetary policy and many scholars have quoted from the transcripts.  Or, research that uses the text in the Beige Book (or other Fed forecasts books).  ", "paper_topic_and_main_contributions": "US Monetary Policy is conducted through the Federal Reserve System, specifically the \"Federal Open Markets Committee\" (FOMC).  The FOMC meets 8 times per year and produces a \"statement\" and a \"transcript.\"  The FOMC also has an action in setting the Fed Funds Rate (not used here).   The paper uses GPT-4 model to chart the \"hawk/dove\" sentiment in US monetary policy \"statement\" (i.e., press release) and the meeting's \"transcript.\"  The main question -- does the issued statement reflect the sentiment of the transcript of the meetings.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "\nRomer, C. D., Romer, D. H., 2004. A new measure of monetary shocks: Derivation and implications. American Economic Review 94 (4), 1055–1084.  (One of many papers in the area -- hawkish and dovish depend on the data the fed is looking at)\n\nThe cite to Blider et. al. is a good cite.  But note that is a survey article.  People have discussed the transparency of monetary policy for a long time."}}, {"review_id": "LGVFxIcCjl", "reviewer": "Reviewer_gKYa", "summary": "", "questions": "See questions embedded in the section above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 3 for their feedback.  We appreciate the acknowledgement of the novelty and for stressing the importance of pairing technical expertise with domain knowledge.  \n\nRegarding Figure 3a, we appreciate the questions and concerns you raise. While the sentence-level method and the full document method are both measuring the same thing (hawk/dove sentiment), simply rescaling the former to try to capture the latter would not be appropriate. The contrast highlights an important result about which method most closely reflects gold standard human annotation. Figure 3a is an empirical assessment of our proposed methodology compared to Hansen and Kazinnik (2023)'s method which scored at the sentence level. Section 2.4 (*Lines 203-223*) currently explains this.  Furthermore, the fact that our preferred methodology considers every available meeting rather than a random sample of them, like the original work, is non-trivial (*Lines 194-199*).  \n\nWe agree with your suggestion to include a measure quantifying the deviation between the sentence-level measure à la Hansen and Kazinnik (2023) and our preferred measures that ingest the entire statement. We will include this measure in-line as part of our discussion of Figure 3a.\n\nRegarding Figure 3b, the equation on Line 214 is used to construct the logit-scaled hawk/dove score. This is neither a new evaluation nor analysis, simply a rescaling of the hawk/dove scores to better reflect extreme values. Re-scaling the scores in this way puts more emphasis on relative rather than absolute differences (*Lines 216-223*).\n\nFinally, the question you raise about why particular differences emerge in specific years is a good one. We note that our measure identifies a change in the volatility of hawk/dove sentiment after 2000 that aligns with the historical record and past empirical work on FOMC communication strategy (*Line 245*). Indeed, there are many other interesting specific years of interest we would like to highlight, but we kept it to this one example for the sake of brevity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ Important and novel question and data\n+ Nice combination of domain and technical expertise", "reasons_to_reject": "- Ultimately, I really don't understand how this was evaluated. Figure 3a) shows a visual result that I presume could be turned into some kind of quantitatve analysis with limited effort (although I'd think that one would want to account for time in that analysis, which makes things a bit trickier), but as it is I am just left to compare pairs of lines and trust intuition. I can see the authors point about extreme values, but I don't really understand why I couldn't just rescale the sentence-level measure and get the same effect\n- Figure 3b) seems like maybe another evaluation, or just an analysis? If the former, I don't really know what is being evaluated. If the latter, more details on why differences between the transcript and statement measures emerges in specific years would be interesting (I get the gist from the rest of the paper about why we should see differences, but would be curious to hear more about specific years)", "questions_to_authors": "See questions embedded in the section above.", "paper_topic_and_main_contributions": "The paper presents an application of LLMs to identifying dissent within FOMC transcripts, and use this method to explore the Hawkishness/Dovishness of FOMC policy attitudes over time", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a study leveraging GPT-4 to analyze the patterns of dissent within the Federal Open Market Committee (FOMC) on the subject of inflation. The paper addresses an important research problem, i.e., understanding the dynamics within FOMC, and employs a novel approach using a state-of-the-art model, GPT-4, which is noteworthy. The paper presents an annotated dataset that can be valuable for a variety of applications, including economics, finance, and NLP research. The most glaring weakness is the lack of quality measures or sanity checks to ensure the reliability of the analysis. Given the high-stakes nature of the topic, this is crucial for both academic and policy implications. The research primarily differs from the previous work by the transition from GPT-3 to GPT-4. This could be seen as an incremental step, diminishing the paper's overall impact. In light of the identified merits and shortcomings, the collective judgment leans toward accepting the paper for the "Findings" section.
Decision: Accept-Findings