[Federal Register Volume 88, Number 176 (Wednesday, September 13, 2023)]
[Notices]
[Pages 62850-62863]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2023-19728]


-----------------------------------------------------------------------

SECURITIES AND EXCHANGE COMMISSION

[Release No. 34-98321; File No. SR-NASDAQ-2022-079]


Self-Regulatory Organizations; The Nasdaq Stock Market LLC; 
Notice of Filing of Amendment No. 2 and Order Granting Accelerated 
Approval of a Proposed Rule Change, as Modified by Amendment No. 2, To 
Amend Rules 4702(b)(14) and (b)(15) Concerning Dynamic M-ELO Holding 
Period

September 7, 2023.

I. Introduction

    On December 21, 2022, The Nasdaq Stock Market LLC (``Nasdaq'' or 
``Exchange'') filed with the Securities and Exchange Commission 
(``Commission''), pursuant to Section 19(b)(1) of the Securities 
Exchange Act of 1934 (``Act'') \1\ and Rule 19b-4 thereunder,\2\ a 
proposed rule change to replace the static holding period requirements 
for Midpoint Extended Life Orders and Midpoint Extended Life Orders 
Plus Continuous Book with dynamic holding periods. The proposed rule 
change was published for comment in the Federal Register on January 10, 
2023.\3\ On February 22, 2023, pursuant to Section 19(b)(2) of the 
Act,\4\ the Commission designated a longer period within which to 
approve the proposed rule change, disapprove the proposed rule change, 
or institute proceedings to determine whether to disapprove the 
proposed rule change.\5\ On March 9, 2023, the Exchange filed Amendment 
No. 1 to the proposed rule change, which amended and superseded the 
proposed rule change as originally filed. On April 7, 2023, the 
Commission provided notice of filing of Amendment No. 1 and instituted 
proceedings to determine whether to approve or disapprove the proposed 
rule change, as modified by Amendment No. 1.\6\ On July 6, 2023, 
pursuant to Section 19(b)(2) of the Act,\7\ the Commission designated a 
longer period on proceedings to determine whether to approve or 
disapprove the proposed rule change.\8\ On July 18, 2023, the Exchange 
filed Amendment No. 2 to the proposed rule change, which amended and 
superseded the proposed rule change as amended by Amendment No. 1. The 
Commission received comments on the proposed rule change.\9\ The 
Commission is publishing this Notice and Order to solicit comment on 
Amendment No. 2 in Sections II and III below, which sections are being 
published verbatim as filed by the Exchange, and to approve the 
proposed rule change, as modified by Amendment No. 2, on an accelerated 
basis.
---------------------------------------------------------------------------

    \1\ 15 U.S.C. 78s(b)(1).
    \2\ 17 CFR 240.19b-4.
    \3\ See Securities Exchange Act Release No. 92844 (January 4, 
2023), 88 FR 1438.
    \4\ 15 U.S.C. 78s(b)(2).
    \5\ See Securities Exchange Act Release No. 96963, 88 FR 12710 
(February 28, 2023).
    \6\ See Securities Exchange Act Release No. 97263, 88 FR 22498 
(April 13, 2023).
    \7\ 15 U.S.C. 78s(b)(2).
    \8\ See Securities Exchange Act Release No. 97844, 88 FR 44423 
(July 12, 2023).
    \9\ All comments received by the Commission on the proposed rule 
change are available on the Commission's website at: https://www.sec.gov/comments/sr-nasdaq-2022-079/srnasdaq2022079.htm.
---------------------------------------------------------------------------

II. Self-Regulatory Organization's Statement of the Terms of Substance 
of the Proposed Rule Change

    The Exchange proposes to amend Rules 4702(b)(14) and (b)(15) of the 
Exchange's Rulebook to replace the static holding period requirements 
for Midpoint Extended Life Orders and Midpoint Extended Life Orders 
Plus Continuous Book with dynamic holding periods. This Amendment No. 2

[[Page 62851]]

supersedes the original filing and Amendment No. 1 \10\ in their 
entireties.
---------------------------------------------------------------------------

    \10\ See SR-Nasdaq-2022-079 Amendment No. 1 (March 9, 2023), at 
https://www.sec.gov/comments/sr-nasdaq2022-079/srnasdaq2022079-20159016-327215.pdf.
---------------------------------------------------------------------------

    The text of the proposed rule change is available on the Exchange's 
website at https://listingcenter.nasdaq.com/rulebook/nasdaq/rules, at 
the principal office of the Exchange, and at the Commission's Public 
Reference Room.

III. Self-Regulatory Organization's Statement of the Purpose of, and 
Statutory Basis for, the Proposed Rule Change

    In its filing with the Commission, the Exchange included statements 
concerning the purpose of and basis for the proposed rule change and 
discussed any comments it received on the proposed rule change. The 
text of these statements may be examined at the places specified in 
Item IV below. The Exchange has prepared summaries, set forth in 
sections A, B, and C below, of the most significant aspects of such 
statements.

A. Self-Regulatory Organization's Statement of the Purpose of, and 
Statutory Basis for, the Proposed Rule Change

1. Purpose
    The Exchange proposes to amend Rules 4702(b)(14) and (15) of the 
Exchange's Rulebook to replace the static 10 millisecond holding period 
requirements for its Midpoint Extended Life Order (``M-ELO'') and 
Midpoint Extended Life Order Plus Continuous Book (``M-ELO+CB'') Order 
Types with dynamic holding periods (``Dynamic M-ELO and M-ELO+CB'' or 
collectively, ``Dynamic M-ELO'').
Background
    In 2018, the Exchange introduced the M-ELO, which is a Non-
Displayed Order priced at the Midpoint between the National Best Bid 
and Offer (``NBBO'') and which is eligible for execution only against 
other eligible M-ELOs and only after a minimum of one-half second 
passes from the time that the System accepts the order (the ``Holding 
Period'').\11\ In 2019, the Exchange introduced the M-ELO+CB, which 
closely resembles the M-ELO, except that a M-ELO+CB may execute at the 
midpoint of the NBBO, not only against other eligible M-ELOs (and M-
ELO+CBs), but also against Non-Displayed Orders with Midpoint Pegging 
and Midpoint Peg Post-Only Orders (``Midpoint Orders'') that rest on 
the Continuous Book for at least one-half second and have Trade Now 
enabled.\12\
---------------------------------------------------------------------------

    \11\ See Securities Exchange Act Release No. 34-82825 (March 7, 
2018), 83 FR 10937 (March 13, 2018) (SR-NASDAQ-2017-074) (``M-ELO 
Approval Order'').
    \12\ See Securities Exchange Act Release No. 34-86938 (September 
11, 2019), 84 FR 48978 (September 17, 2019) (SR-NASDAQ-2019-048) 
(``M-ELO+CB Approval Order'').
---------------------------------------------------------------------------

    When the Exchange designed M-ELO, it originally set the length of 
the Holding Period at one-half second because it determined that this 
time period would be sufficient to ensure that likeminded investors 
would interact only with each other, and with minimal market impacts. 
The Exchange believed that the longer length of the M-ELO Holding 
Period and its simplicity in design would provide greater protection 
for participants than they could achieve through competing delay 
mechanisms.
    In 2020, however, the Exchange shortened the length of the Holding 
Period to 10 milliseconds.\13\ The Exchange did so after studying two 
years of actual use and performance of M-ELOs, as well as customer 
feedback. That is, the Exchange came to understand that, while users of 
M-ELO and M-ELO+CB are less concerned with achieving rapid executions 
of their Orders than are other participants, they are not indifferent 
about the length of time in which their M-ELOs and M-ELO+CBs must wait 
before they are eligible for execution. Indeed, participants informed 
the Exchange that in certain circumstances, such as when they sought to 
trade symbols that on average had a lower time-to-execution than a 
half-second, they were reticent to enter M-ELOs or M-ELO+CBs. They 
indicated that the associated Holding Periods for these Order Types 
were longer than necessary to achieve the desired protections and that, 
during the residual portion of the Holding Periods, they risked losing 
out on favorable execution opportunities that would otherwise be 
available to them had they placed a non-MELO order.
---------------------------------------------------------------------------

    \13\ See Securities Exchange Act Release No. 34-88743 (April 24, 
2020), 85 FR 24068 (April 30, 2020) (SR-NASDAQ-2020-011) (``M-ELO 
Timer Approval Order'').
---------------------------------------------------------------------------

    Based upon this feedback, the Exchange studied the potential 
effects of reducing the length of the Holding Periods for both M-ELOs 
and M-ELO+CBs (as well as for Midpoint Orders that would execute 
against M-ELO+CBs). Ultimately, the Exchange determined that it could 
reduce the Holding Periods to 10 milliseconds without compromising the 
protective power that M-ELO and M-ELO+CB are intended to provide to 
participants and investors.\14\ Thus, the Exchange determined that 
shortening the Holding Periods to 10 milliseconds for M-ELOs and M-
ELO+CBs would increase the efficacy of the mechanism while not 
undermining the power of those Order Types to fulfill their underlying 
purpose of minimizing market impacts. At the same time, the Exchange 
determined that a reduction in the Holding Periods to 10 milliseconds 
would dramatically add to the circumstances in which M-ELOs and M-
ELO+CBs would be useful to participants. In its M-ELO Timer Approval 
Order, the Commission agreed with the Exchange:
---------------------------------------------------------------------------

    \14\ The Exchange examined each of its historical M-ELO 
executions to determine at what Midpoints of the NBBO the M-ELOs 
would have executed if their Holding Periods had been shorter than 
one-half second (500 milliseconds). After examining the historical 
effects of shorter Holding Periods of between 10 milliseconds and 
400 milliseconds, the Exchange determined that a reduction of the M-
ELO Holding Period to as short as 10 milliseconds would have caused 
an average impact on mark-outs of only 0.10 basis points (across all 
symbols). In other words, compared to the execution price of an 
average M-ELO with a one-half second Holding Period, the Exchange 
found that a M-ELO with a 10 millisecond Holding Period would have 
had an average post-execution impact that was only a tenth of a 
basis point per share--a difference in protective effect that is 
immaterial. See Nasdaq, ``The Midpoint Extended Life Order (M-ELO); 
M-ELO Holding Period,'' available at https://www.nasdaq.com/articles/the-midpoint-extended-life-order-m-elo%3A-m-elo-holding-period-2020-02-13 (analyzing effects of shortened Holding Periods on 
M-ELO performance).
---------------------------------------------------------------------------

    The Commission notes that, with the proposed ten-millisecond 
Holding Period and Resting Period, M-ELOs and M-ELO+CBs would continue 
to be optional order types that are available to investors with longer 
investment time horizons, including institutional investors. The 
Commission also believes that the proposal could make M-ELOs and M-
ELO+CBs more attractive for securities that on average have a time-to-
execution of less than one-half second and, for investors who currently 
do not use M-ELOs and M-ELO+CBs for these securities, provide optional 
order types that could enhance their ability to participate effectively 
on the Exchange. The Commission notes that, if market participants 
determine that the proposal would make M-ELOs and M-ELO+CBs less 
attractive for their particular investment objectives, such market 
participants may elect to reduce or eliminate their use of these 
optional order types. Moreover, as noted above, the Exchange will 
continue to conduct real-time surveillance to monitor the use of M-ELOs 
and M-ELO+CBs to ensure that such usage remains appropriately tied to 
the intent of the order types. If, as a result of such surveillance, 
the Exchange determines that the shortened

[[Page 62852]]

Holding Period does not serve its intended purpose or adversely impacts 
market quality, the Exchange would seek to make further 
recalibrations.\15\
---------------------------------------------------------------------------

    \15\ M-ELO Timer Approval Order, supra, at 85 FR 24069.
---------------------------------------------------------------------------

    For similar reasons and with even better potential results for 
participants, the Exchange now proposes to further refine the length of 
the Holding Periods for M-ELOs and M-ELO+CBs, this time through the 
application of innovative and patent pending machine learning 
technology.
Dynamic M-ELO
    After receiving feedback from participants that even 10 millisecond 
Holding Periods for M-ELO and M-ELO+CB may, at times, exceed what is 
necessary to accomplish the underlying intent of these Order Types, the 
Exchange began to experiment with making further refinements to the 
duration of the Holding Periods. Ultimately, the Exchange concluded 
that shorter Holding Periods could achieve the same, if not better 
results for participants in terms of mark-outs, but not in all 
circumstances. That is, where prices of the underlying securities are 
stable, and not subject to imminent unfavorable changes, M-ELOs and M-
ELO+CBs face lower risks of confronting spread-crossing orders, such 
that shorter Holding Periods could suffice to protect M-ELOs and M-
ELO+CB from such orders. In periods of heightened price volatility, 
however, M-ELOs and M-ELO+CBs also face heightened risks, such that 
longer Holding Periods would continue to be beneficial in protecting M-
ELOs and M-ELO+CBs from such risks. Thus, the Exchange determined that 
another across-the-board reduction of the static 10 millisecond Holding 
Periods would be sub-optimal because it could impact the performance of 
the M-ELO and M-ELO+CB Order Types during periods of heightened 
volatility.
    In light of these observations, the Exchange tasked its artificial 
intelligence and machine learning laboratory (the ``AI Core Development 
Group'') to explore whether it could employ these innovative 
technologies to optimize the length of M-ELO and M-ELO+CB Holding 
Periods during various states of price volatility, and then to vary the 
lengths of the Holding Periods dynamically during the lifecycles of M-
ELOs and M-ELO+CBs, with the objectives of improving the performance of 
these Order Types while also further reducing opportunity costs.
    As the Exchange explains in greater depth in the attached White 
Paper,\16\ the AI Core Development Group proceeded to develop an 
artificial intelligence-based timer control system that will achieve 
these objectives.\17\ The AI Core Development Group did so by using 
reinforcement learning techniques--machine learning paradigms which 
develop optimal solutions to problems over time by taking actions to 
solve them, generating feedback on the results of such actions, 
applying that feedback to direct and improve the next round of 
solutions, and then repeating the feedback loop until the paradigm 
achieves optimized solutions.
---------------------------------------------------------------------------

    \16\ See Diana Kafkes et al., ``Applying Artificial Intelligence 
& Reinforcement Learning Methods Towards Improving Execution 
Outcomes,'' SSRN, October 19, 2022, available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4243985 (attached hereto 
[sic] without modification from the prior version as Exhibit 3(a)) 
(the ``White Paper'').
    \17\ Although the AI Core Development Group acknowledges that an 
optimal Holding Period would update with every incoming order, it 
determined that training a reinforcement learning model on every 
order would be too difficult to program and too difficult to 
implement given the nanosecond latency requirements of the Exchange. 
The Group then investigated more feasible update cadences and 
determined the point at which optimal outcomes were best balanced 
with the level of programming and implementation difficulty to be 
between 15 and 30 second updates. Ultimately, the Group chose a 30 
second update cadence to give the model the greatest opportunity to 
learn between potential actions.
---------------------------------------------------------------------------

    In this instance, the AI Core Development Group applied 
reinforcement learning techniques to a simulation of the M-ELO Book 
that it constructed using a representative data set from the first 
quarter of 2022 (the ``Training Period''). The Training Period data 
consisted of 380 out of the 6,257 symbols on the M-ELO Book (accounting 
for approximately 67 percent of M-ELO volume). The symbols chosen 
reflect both actively-traded and thinly-traded securities, and both 
low-priced and high-priced securities.
    The AI Core Development Group then developed a machine learning 
model and applied it to the Training Period data. The Group programmed 
the model to value the achievement of higher fill rates or lower mark-
outs than that which occurred in a historical simulation of M-ELOs and 
M-ELO+CBs involving the Training Period data.\18\ The Group then 
programmed the model to seek to achieve its goals by taking one of five 
possible actions with respect to the duration of the Holding Periods at 
30 second intervals \19\ for each symbol during each trading day of the 
Training Period. That is, at each 30 second internal, the model 
evaluated market conditions for each symbol over the prior 30 second 
period and either kept the Holding Periods the same, increased/
decreased them by 0.25 milliseconds, or increased/decreased them by 
0.50 milliseconds.\20\ After each decision-making round, the model 
utilized the results to inform its actions at the next 30 second 
increment.
---------------------------------------------------------------------------

    \18\ As the White Paper explains, the Group developed a model to 
simulate activity on the Exchange involving M-ELOs and M-ELO+CBs 
during the Training Period. See White Paper, supra, at 10.
    \19\ See id.
    \20\ The AI Core Development Group experimented with a range of 
permissible Holding Period durations. Ultimately, it concluded that 
it could produce better outcomes for M-ELO and M-ELO+CB participants 
than the existing approach using Holding Periods as low as 0.25 
milliseconds and as high as 2.5 milliseconds, under normal market 
conditions.
---------------------------------------------------------------------------

    In making its decisions, the model (again, drawing upon a 
combination of historical SIP and M-ELO-specific data) considered 142 
categories of data points.\21\ A confluence of data points that 
correlated with an increase in volatility tended to cause the model to 
increase the durations of Holding Periods, including increases in the 
standard deviation of NBBO prices, the number of unique participants 
placing sell orders on M-ELO and M-ELO+CB, and the volume-weighted 
average of the NBBO spread. Conversely, a confluence of data points 
that correlated with greater price stability tended to cause the model 
to decrease the durations of Holding periods, such as an increase in 
the median and max number of shares per trade and the number of resting 
bids left in the M-ELO and M-ELO+CB Book.
---------------------------------------------------------------------------

    \21\ Nasdaq attaches a full list of these data elements 
(attached hereto [sic] as ``Exhibit 3(b))'', along with an 
observation of the strength of the correlations that currently exist 
between changes to those data values and decisions the system makes 
to set the duration of Holding Periods at any given time. The 
Exchange notes that the version of this list attached to this 
Amendment No. 2 supersedes prior versions attached to prior versions 
of this filing. This version of the list includes expanded 
explanations of the terminology used therein. See also White Paper, 
supra, at 31, for a description of these features.
---------------------------------------------------------------------------

    The AI Core Development Team produced variations of its model that 
prioritized achievement of the lowest mark-outs, the highest fill 
rates, and a blend of these two objectives.\22\ Through

[[Page 62853]]

a process of learning and experimentation involving a combination of 
historical and simulated data, the AI Core Development Group settled on 
a Dynamic M-ELO model that achieved substantial simulated performance 
improvements for users of M-ELO and M-ELO+CB--both in terms of mark-
outs and fill rates--as compared to the static 10 millisecond Holding 
Periods. As the White Paper explains in greater detail, Dynamic M-ELO 
yielded an average combined volume-weighted (simulated) improvement of 
31.7 percent, including a 20.3 percent increase in fill rates and a 
11.4 percent reduction in mark-outs.\23\ The White Paper provides a 
more fulsome explanation of these improvements.\24\
---------------------------------------------------------------------------

    \22\ The AI Core Development Group also applied to the model a 
paradigm called ``retraining'' to combat the degradation of model 
performance that can otherwise occur as the reference data it uses 
for initial comparison becomes stale. Finally, the AI Core 
Development group added a stability protection mechanism to the 
model to provide maximum production to participants in the event 
that the model observes extraordinary levels of instability in the 
National Best Bid and Offer during the prior three seconds as 
compared to reference data. When the model detects such instability, 
it is programmed to increase the length of the Holding Period to 12 
milliseconds for a period of 750 milliseconds.
    \23\ See White Paper, supra, at 22.
    \24\ See id.
---------------------------------------------------------------------------

    Based upon these exciting results, the Exchange now proposes to 
amend Rule 4702(b)(14) and (15) to replace the static 10 millisecond 
timers applicable to M-ELO and M-ELO+CB with Dynamic M-ELO Holding 
Periods. Using the Exchange's ``proprietary assessment of market 
conditions'' \25\ and patent pending technology, the Dynamic M-ELO 
system will evaluate and, as it deems necessary, adjust the length of 
the Holding Periods for each symbol comprising M-ELOs and M-ELO+CBs 
(and Midpoint Orders on the Continuous Book that opt to interact with 
M-ELO+CBs after resting on the Book) every 30 seconds throughout the 
Market Hours (each such 30 second interval, a ``Change Event''). In so 
doing, Dynamic M-ELO will help participants to achieve a more optimized 
blend of the underlying purposes of the M-ELO and M-ELO+CB Order Types: 
protection against adverse selection (low mark-outs) without 
sacrificing opportunities to achieve high-quality executions (high fill 
rates).
---------------------------------------------------------------------------

    \25\ As set forth in the proposed rule text, the phrase 
``proprietary assessment of market conditions'' refers to the 
Exchange's evaluation of prevailing market conditions for a given 
symbol using an algorithm programmed to set a Holding Period 
duration which, at each Change Event, achieves an optimal blend of 
two objectives: maximization of M-ELO fill rates; and minimization 
of M-ELO mark-out rates. As the rule text states and as is discussed 
below, the algorithm ingests and analyzes 142 data points, which the 
Exchange identifies and describes in Exhibit 3b hereto. The Exchange 
derives these data from a combination of public data and M-ELO data 
feeds. Furthermore, the Exchange conducts weekly re-trainings of the 
algorithm, outside of Market Hours, to improve its performance 
relative to the immediately preceding period (in terms of the two 
aforementioned objectives). The Exchange deploys a retrained version 
of the algorithm only if it determines that doing so will, in fact, 
improve its performance relative to the immediately preceding 
period. The Exchange provides further information about the 
algorithm and the retraining process in a White Paper attached 
hereto [sic] as Exhibit 3a.
---------------------------------------------------------------------------

    A proposed M-ELO or M-ELO+CB with a Dynamic Holding Period will 
operate as follows. At the outset of Market Hours (approximately 
9:30:00 a.m.), the Exchange will impose initial Holding Periods of 1.25 
milliseconds for M-ELOs and M-ELO+CBs in all symbols. Thereafter, 
Holding Periods for a given symbol will become eligible to change 
dynamically from the initial duration beginning at 9:30:30 a.m. and 
then at 30 second intervals thereafter during Market Hours. The 
Exchange will then apply to the M-ELO or M-ELO+CB Order a Holding 
Period that is of the duration that prevailed at the time of entry. For 
example, if participant A enters a M-ELO for symbol XYZ at 9:30:25 
a.m., then Holding Period for that M-ELO will be 1.25 milliseconds. If 
at 9:30:30:00 a.m., the System decides to lower the duration of the 
Holding Period by 0.50 milliseconds, and then participant B enters a M-
ELO for symbol XYZ at 9:30:45 a.m., then the System will assign a 0.75 
millisecond Holding Period to participant B's M-ELO. To be clear, the 
System will determine Dynamic M-ELO Holding Periods independently for 
M-ELOs and M-ELO+CBs in each symbol.
    During normal market conditions, the range of potential Holding 
Period durations for M-ELOs and M-ELO+CBs will be between 0.25-2.50 
milliseconds, with the Holding Period duration being eligible to change 
by increments of either 0.25 or 0.50 milliseconds at each Change Event. 
Thus, if the Holding Period for a M-ELO in symbol XYZ is set at 0.75 
milliseconds at 2:22:11 p.m., and at 2:22:41 p.m., the System 
determines to increase the duration of the Holding Period, it may do so 
only by 0.25 or 0.50 milliseconds during that event.
    When a Change Event occurs, and the System determines to adjust the 
duration of a Holding Period for a symbol, that adjustment will apply, 
not only to all M-ELOs and M-ELO+CBs for that symbol entered within the 
30 second period after the Change Event occurs, but also to M-ELOs and 
M-ELO+CBs entered prior to the Change Event with unexpired Holding 
Periods (with applicability retroactive to the time of Order 
acceptance). Thus, if a participant enters a M-ELO in symbol XYZ at 
1:14:299 p.m., and the prevailing Holding Period applicable to that M-
ELO is 2 milliseconds, and at 1:14:30 p.m., the System modifies the 
Holding Period to be 1.5 milliseconds, then the M-ELO will become 
eligible to execute at 1:14:3005 p.m. This is the case because the M-
ELO will have already expended 1 millisecond of its Holding Period as 
of the time of the Change Event; thereafter, the M-ELO will need to 
rest only another 0.5 milliseconds to become eligible to execute under 
the new 1.5 millisecond Holding Period (as measured from 1:14:299 
p.m.). This last feature ensures that the M-ELO Book maintains time 
priority among M-ELOs and M-ELO+CBs in a dynamic environment. That is, 
it ensures that no M-ELO or M-ELO+CB with an unexpired Holding Period 
at the time of a Change Event will end up becoming eligible to execute 
later than a M-ELO entered after the Change Event which has a shorter 
Holding Period applicable to it.
    If at any time, the System detects extraordinary instability in a 
symbol, then the System will activate a ``stability protection 
mechanism'' to provide an extra layer of protection to M-ELO and M-ELO 
users from the heightened risks of adverse selection that exists during 
such periods of instability.\26\ The stability protection mechanism 
will override the prevailing Holding Periods for M-ELOs and M-ELO+CBs 
in a symbol experiencing extraordinary instability and immediately 
increase the duration of those Holding Periods to 12 milliseconds for a 
period of 750 milliseconds. The System may activate the stability 
protection mechanism even between Change Events. The System will 
evaluate, at each NBBO update, whether market conditions remain 
extraordinarily unstable and, if so, it will restart the 750 
millisecond Stability Protected Period and maintain the 12 millisecond 
Holding Period until conditions stabilize. Once the System determines 
that market conditions have stabilized (i.e., all measurements for the 
symbol are at or below the threshold value throughout the duration of 
the prevailing Stability Protected Period), the System will revert the 
duration of the Holding Periods to that which prevailed as of the 
Change Event that

[[Page 62854]]

occurred immediately prior to the activation of the stability 
protection mechanism or, if the stability protection mechanism was 
active when a Change Event occurred, to the duration selected at the 
immediately preceding Change Event. The System will then proceed to 
reevaluate the duration of the Holding Periods as per the regular 
schedule of Change Events.
---------------------------------------------------------------------------

    \26\ For purposes of this Rule, the System determines that 
``extraordinary instability'' for a symbol exists through 
observations it makes following every change in the NBBO for that 
symbol that occurs during the trading day. When the NBBO changes, 
the System looks back at the prior three seconds of trading and 
measures the difference between the highest and the lowest NBBO 
midpoint values that occurred during that period, and then it 
compares that measurement to a threshold value for the symbol. The 
System concludes that extraordinary instability exists for a symbol 
if the measurement exceeds the threshold value. The threshold value 
for a symbol, in turn, is the difference between the highest and the 
lowest NBBO midpoint values for the symbol that, if applied to its 
trading activity during the prior trading day, would have caused the 
System to deem trading in the symbol to be extraordinarily unstable 
for as close to one percent of that day as possible.
---------------------------------------------------------------------------

    The following is an illustration of the operation of the stability 
protection mechanism. At 11:10:04 a.m., the prevailing Holding Period 
for M-ELOs in symbol XYZ is 1.5 milliseconds. At the same time, the 
NBBO for symbol XYZ updates. The System looks back at the prior three 
seconds of trading in symbol XYZ and finds that during that period, the 
highest observed NBBO midpoint was $10.05, and the lowest was $10.00, 
such that the difference between these two values is a range of $0.05. 
The System then looks back at trading behavior for symbol XYZ during 
the immediately preceding trading day. In doing so, the System 
calculates the value of the threshold that would have caused the symbol 
to be deemed extraordinarily unstable for one percent of the trading 
day; the System determines that this threshold value is a range of 
$0.03. The System then compares the $0.03 threshold to its measurement 
of the prior three seconds of NBBO changes ($0.05), and concludes that 
over these past three seconds, the symbol is extraordinarily unstable. 
Accordingly, the System activates the stability protection mechanism 
and the Holding Period for M-ELOs in symbol XYZ immediately increases 
to 12 milliseconds for a period of 750 milliseconds. However, 5 
milliseconds after the Stability Protection Period commences, the NBBO 
updates again, thus prompting the System to repeat its assessment of 
the stability of the symbol in light of the update. This reassessment 
reveals that the symbol remains unstable, such that a new Stability 
Protection Period of 750 milliseconds begins at that time (overriding 
the pre-existing Period). Over the course of this new Stability 
Protection Period, the NBBO shifts two more times, but each of the 
ensuing reassessments indicate that the NBBO ranges for the symbol have 
fallen below the $0.03 threshold. The Stability Protection Period 
elapses 750 milliseconds after it began with the symbol remaining 
stable. Thus, the Holding Period reverts to 1.5 milliseconds.
    If the Exchange halts trading in a symbol, then upon resumption of 
trading, any new M-ELO or M-ELO+CB in that symbol and any pending M-ELO 
or M-ELO+CB in that symbol with an unexpired Holding Period will be 
subject to a new 12 milliseconds Holding Period (running from the time 
when trading resumes) until the next scheduled Change Event, at which 
point the System may determine to adjust that Holding Period to a 
duration within the range applicable under normal market 
conditions.\27\ If, however, the System determines that extraordinary 
instability in the symbol exists, it will instead determine to activate 
the stability protection mechanism and maintain the duration of the 
Holding Period at 12 milliseconds for another 750 milliseconds. This 
design will help to ensure that M-ELOs and M-ELO+CBs receive added 
protection coming out of halt conditions.\28\
---------------------------------------------------------------------------

    \27\ Prior to commencement of a new 12 millisecond Holding 
Period for a new or pending M-ELO or M-ELO+CB following a Halt, the 
System will first determine whether the M-ELO or M-ELO+CB is or 
remains eligible for execution. That is, the Holding Period will 
commence only if, upon commencement of trading following the Halt, 
the midpoint price for the Order is within the limit set by the 
participant. If not, the System will hold the Order until the 
midpoint falls within the limit set by the participant, at which 
time the 12 millisecond Holding Period will commence.
    \28\ Also as a safeguard, the System will apply a default 
Holding Period of 12 milliseconds to a M-ELO or M-ELO+CB if ever it 
fails to receive a signal during a Change Event as to whether the 
System should adjust or maintain the duration of the prevailing 
Holding Period. The System will continue to apply the default 12 
millisecond Holding Period until the next Change Event where the 
signal is restored and the System is able to act dynamically again.
---------------------------------------------------------------------------

    The Exchange notes that same dynamic process described above will 
also apply to and govern the time periods during which Midpoint Orders 
on the Continuous Book must rest before they will become eligible to 
interact with M-ELO+CBs (provided that participants have opted for 
their Midpoint Orders to interact with M-ELO+CBs). Thus, the same 
Holding Period duration that the System sets for a M-ELO+CB in a symbol 
during Regular Market Hours will also be the length of time that a 
Midpoint Order must rest on the Continuous Book must rest before it may 
interact with a M-ELO+CB.
    Apart from these impacts of Dynamic Holding Periods, M-ELOs and M-
ELO+CBs will continue to behave as they do now in all respects, and as 
set forth in Rules 4702(b)(14) and (15).
    It is important to note that within the parameters discussed herein 
and in the White Paper, the Exchange will continue to re-train Dynamic 
M-ELO and M-ELO+CB on a weekly basis (outside of market hours) so that 
the model will continue to learn from and act upon the basis of more 
recent SIP and M-ELO book data sets, and further improve its 
performance over time. The retraining process should not result in 
dramatic or unpredictable changes to the behavior of Dynamic M-ELO. The 
retraining process will not retrain the model from scratch each week; 
rather, it will retain the model's existing data inputs, knowledge 
base, and objectives--all without alteration. Retraining will result in 
new behaviors only as needed to address new scenarios that the model 
did not confront previously, and even then, only in a manner designed 
to further optimize outcomes, i.e., reduce mark-outs or increase fill 
rates. If the System assesses that a retrained model would be worse 
than the existing model in achieving its objectives, then the System 
will continue to use the existing model and discard the retrained 
model. This retraining process is a standard and accepted practice for 
use of deep learning models; it helps to ensure that deep learning 
models not only work well, but that they continue to work well in 
dynamic circumstances.\29\
---------------------------------------------------------------------------

    \29\ During periods where the model is not undergoing 
retraining, the System will behave predictably from day to day, such 
that its decisions when presented with given set of facts and 
circumstances in a given security on day 1 should be the same as 
they would be on day 2.
---------------------------------------------------------------------------

    The Exchange will not modify the underlying structure of Dynamic M-
ELO and M-ELO+CB without first obtaining the Commission's approval to 
do so, including modifications to the data elements the model considers 
in making decisions about Holding Period durations, the conditions 
under which the model may adjust the duration of Holding Periods, the 
frequency with which the model my adjust the Holding Periods, the range 
of Holding Period durations available to M-ELOs and M-ELO+CBs, the 
increments by which Holding Periods may change at any given Change 
Event, and the procedures for triggering, maintaining, and ending 12 
millisecond Holding Periods during times of extraordinary 
instability.\30\ Although the Exchange will seek Commission approval 
prior to changing any of the data elements that the model considers, 
the Exchange will not seek Commission approval prior to retraining

[[Page 62855]]

the model to adjust the weighting it applies to those data elements.
---------------------------------------------------------------------------

    \30\ In addition to the proposed changes described above, the 
Exchange proposes to delete an extraneous reference in Rule 
4702(b)(15) to M-ELO+CB being eligible to execute against a Midpoint 
Order on the Continuous Book if the Continuous Book order has the 
``Midpoint'' Trade Now Attribute enabled. In a prior filing, the 
Exchange folded the concept of ``Midpoint Trade Now'' into the 
general ``Trade Now'' Attribute. See Securities Exchange Act Release 
No. 34-92180 (June 15, 2021), 86 FR 33420 (June 24, 2021)(SR-NASDAQ-
2021-044).
---------------------------------------------------------------------------

    To aid investors in understanding and evaluating Dynamic M-ELO, 
Nasdaq will continue to publish weekly and monthly transparency 
statistics on Nasdaqtrader.com, as it does now, about the performance 
of its M-ELOs and M-ELO+CBs, including statistics listing the weekly 
numbers of shares and trades in M-ELOs by symbol, weekly aggregated M-
ELO share and trade data, and monthly aggregated block data.\31\ Nasdaq 
also will continue to disclose monthly data on Nasdaq.com, as it does 
now (the M-ELO Monthly Report), about M-ELO and M-ELO+CB mark-outs 
(quote stability by time horizon) and fill rates.\32\ Moreover, Nasdaq 
will add statistics to the M-ELO Monthly Report about how frequently, 
on average, the System changes Holding Period durations for the top 
decile, median, and bottom decile of symbols, as measured by monthly M-
ELO and M-ELO+CB trading volumes. Nasdaq will retain copies of each 
historical iteration of its models as part of its books and records, 
and make them available to the Commission upon request, should it wish 
to examine them to understand how the model changes over time. 
Furthermore, Nasdaq will publish an equity trader alert in advance of 
deploying a retrained version of Dynamic M-ELO whenever Nasdaq has 
reason to anticipate that the retrained version will produce results 
that differ materially from the prior version, i.e., a projected change 
in mark-outs or fill-rates of 10% or more in either direction.
---------------------------------------------------------------------------

    \31\ See http://www.nasdaqtrader.com/Trader.aspx?id=MELOSymbolData.
    \32\ See, e.g., https://www.nasdaq.com/docs/M-ELO-Monthly-Report. Nasdaq understands that current users of M-ELO and M-ELO 
independently monitor the performance of these Order Types. Nasdaq 
often receives feedback from such users about M-ELO and M-ELO+CB 
performance, which Nasdaq then factors into decisions about 
improvements and enhancements. Nasdaq expects that this feedback 
loop will continue after implementation of Dynamic M-ELO.
---------------------------------------------------------------------------

    The Exchange acknowledges that systems necessary to implement 
Dynamic M-ELO, including the systems proposed that include model 
development and retraining processes, are ``SCI Systems'' within the 
meaning of Regulation Systems Compliance and Integrity (``Reg. 
SCI''),\33\ and that the Exchange, as an SCI Entity, remains 
responsible for compliance with all requirements of Reg. SCI, 
including, without limitation, to have policies and procedures 
reasonably designed to ensure that its SCI Systems operate in a manner 
that complies with the Act and the rules and regulations thereunder and 
Exchange's rules and governing documents, among them a plan for 
assessments of the functionality of SCI Systems designed to detect 
systems compliance issues, including by responsible SCI personnel and 
by personnel familiar with applicable provisions of the Act and the 
rules and regulations thereunder and Exchange's rules and governing 
documents.
---------------------------------------------------------------------------

    \33\ 17 CFR 242.1000 et seq. As set forth in Reg. SCI, the term 
``SCI Systems'' means ``means all computer, network, electronic, 
technical, automated, or similar systems of, or operated by or on 
behalf of, an SCI entity that, with respect to securities, directly 
support trading, clearance and settlement, order routing, market 
data, market regulation, or market surveillance.'' Id. at 242.1000. 
An ``SCI Entity'' means ``an SCI self-regulatory organization, SCI 
alternative trading system, plan processor, exempt clearing agency 
subject to ARP, or SCI competing consolidator.'' Id.
---------------------------------------------------------------------------

Implementation
    The Exchange intends to make the proposed change effective for M-
ELOs and M-ELO+CBs in the Second or Third Quarter of 2023, but that 
time frame is subject to change. The Exchange will publish a Trader 
Alert in advance of making the proposed change effective.
2. Statutory Basis
    The Exchange believes that its proposal is consistent with Section 
6(b) of the Act,\34\ in general, and furthers the objectives of Section 
6(b)(5) of the Act,\35\ in particular, in that it is designed to 
promote just and equitable principles of trade, to remove impediments 
to and perfect the mechanism of a free and open market and a national 
market system, and, in general to protect investors and the public 
interest, by allowing for more widespread use of M-ELOs and M-ELO+CBs.
---------------------------------------------------------------------------

    \34\ 15 U.S.C. 78f(b).
    \35\ 15 U.S.C. 78f(b)(5).
---------------------------------------------------------------------------

    When the Commission approved the M-ELO and the M-ELO+CB, it 
determined that these Order Types are consistent with the Act because 
they ``could create additional and more efficient trading opportunities 
on the Exchange for investors with longer investment time horizons, 
including institutional investors, and could provide these investors 
with an ability to limit the information leakage and the market impact 
that could result from their orders.'' \36\ Nothing about the 
Exchange's proposal should cause the Commission to revisit or rethink 
this determination. Indeed, the proposal will not alter the fundamental 
design of these Order Types, the manner in which they operate, or their 
effects.
---------------------------------------------------------------------------

    \36\ M-ELO Approval Order, supra 83 FR at 10938-39; M-ELO+CB 
Approval Order, supra, 84 FR at 48980.
---------------------------------------------------------------------------

    Even with Dynamic M-ELO Holding Periods, M-ELOs and M-ELO+CBs will 
continue to provide their users with protection against information 
leakage and adverse selection--and they will do so at levels which are 
substantially undiminished from that which they provide now.\37\
---------------------------------------------------------------------------

    \37\ See note 6, supra.
---------------------------------------------------------------------------

    At the same time, however, the proposal will benefit market 
participants and investors by reducing the opportunity costs of 
utilizing M-ELOs and M-ELO+CBs. The proposal, in other words, will re-
calibrate the lengths of the Holding Periods so that M-ELOs and M-
ELO+CBs will operate in the ``Goldilocks'' zone--their Holding Periods 
will not be so short as to render them unable to provide meaningful 
protections against information leakage and adverse selection, but the 
Holding Periods also will not be too long so as to cause participants 
and investors to miss out on favorable execution opportunities. Nasdaq 
believes the proposal will render M-ELOs and M-ELO+CBs more useful and 
attractive to market participants and investors, and this increased 
utility and attractiveness, in turn, will spur an increase in M-ELO and 
M-ELO+CB use cases on the Exchange, both from new and existing users of 
M-ELOs and M-ELO+CBs. Ultimately, the proposal should enhance market 
quality by increasing opportunities for midpoint executions on the 
Exchange.
    As Nasdaq explained above, the Proposal will operate within strict, 
well-defined, and transparent parameters. Although it will undergo 
weekly retraining (outside of market hours),\38\ such retraining will 
aim to improve the performance of the model in achieving its twin 
objectives; retraining will not alter the inputs, objectives, or basic 
design parameters of Dynamic M-ELO without prior Commission 
approval.\39\ Moreover, the Exchange will not deploy a retrained model 
if it fails to achieve performance improvements. To aid

[[Page 62856]]

investors in evaluating Dynamic M-ELO, the Exchange will publish 
statistics about its performance, including as to mark-outs and fill 
rates, as well as statistics about how frequently the System changes 
Holding Period durations. To further facilitate accountability, the 
Exchange will retain each historical iteration of its model as part of 
its books and records, and make such information available to the 
Commission, upon request. The Exchange will also publish equity trader 
alerts whenever retraining will result in a performance change of 10% 
or more.
---------------------------------------------------------------------------

    \38\ To be clear, performance statistics for Dynamic M-ELO cited 
herein and in the White Paper are based upon data derived from 
weekly, not daily retrainings.
    \39\ As discussed above, Nasdaq will not seek Commission 
approval prior to allowing the model, as part of its re-training 
process, to vary the weighting of the data elements it ingests. 
Nasdaq believes this is appropriate because such variance will only 
occur to the extent that it will improve the model's performance 
with respect to pre-defined objectives. Nasdaq will alert traders if 
the retraining process would result in substantial performance 
changes, and it will also publish statistics to help participants to 
assess performance themselves. Moreover, Nasdaq will retain 
historical iterations of its models for the Commission's review, 
should it wish to examine how these models have changed over time.
---------------------------------------------------------------------------

    Nasdaq notes that the twin objectives it prescribes for the model 
involve the absolute values of mark-outs and fill rates; they are not 
designed to further the performance of any participant or any category 
of participant. Furthermore, Nasdaq performed internal tests of its AI 
model to detect indications of harmful bias in its performance results, 
and such tests concluded that no such indications exist. That is, the 
Exchange reviewed the impact on fill rates and mark-outs of Dynamic M-
ELO, as compared to the ``static'' M-ELO, for those firms that 
accounted for more than 95% of M-ELO activity on the Exchange during Q1 
2022.\40\ The Exchange analyzed results both in an absolute and a 
relative sense. Testing revealed that all participants experienced at 
least some improvements in fill rates and mark-outs when using Dynamic 
M-ELO versus static M-ELO, with the volume-weighted average improvement 
being aligned with the results expressed in the White Paper. We 
detected no material variations that might suggest that a particular 
participant or category of participant (i.e., nature of firm; size of 
firm) benefitted from Dynamic M-ELO functionality to an extent that was 
unreasonably disproportionate to the benefits that other participants 
experienced. Thus, Nasdaq believes the model is objective, is designed 
to, and does avoid bias and discrimination.\41\
---------------------------------------------------------------------------

    \40\ Beyond this grouping of participants, the activity levels 
of other individual M-ELO participants were so small as to be 
insignificant. In many cases, these participants entered only a 
handful of M-ELOs during the study period. As such, the Exchange 
believes it is reasonable to exclude such participants from its 
analysis to avoid their data distorting the results.
    \41\ The Exchange will review its AI model periodically to 
affirm that it continues to perform in accordance with the 
Exchange's rules and has not introduced any harmful bias in favor of 
or against any participant or category of participants.
---------------------------------------------------------------------------

    The Exchange notes that use of Dynamic M-ELOs and M-ELO+CBs remains 
voluntary for all market participants. Accordingly, if any market 
participant feels that the dynamic Holding Periods are still too long 
or too short or because competing venues offer more attractive delay 
mechanisms, then the participants are free to pursue other trading 
strategies or utilize other trading venues. They need not utilize 
Dynamic M-ELOs or M-ELO+CBs.
    Furthermore, the design of Dynamic-MELO would constitute an 
``established, non-discretionary'' method that is consistent with the 
definition of an exchange, as set forth in SEC Rule 3b-16.\42\ The 
Commission stated as follows when it adopted Rule 3b-16:
---------------------------------------------------------------------------

    \42\ See 17 CFR 240.3b-16(a)(2) (``(a) An organization, 
association, or group of persons shall be considered to constitute, 
maintain, or provide `a market place or facilities for bringing 
together purchasers and sellers of securities or for otherwise 
performing with respect to securities the functions commonly 
performed by a stock exchange,' as those terms are used in section 
3(a)(1) of the Act, (15 U.S.C. 78c(a)(1)), if such organization, 
association, or group of persons: (1) Brings together the orders for 
securities of multiple buyers and sellers; and (2) Uses established, 
non-discretionary methods (whether by providing a trading facility 
or by setting rules) under which such orders interact with each 
other, and the buyers and sellers entering such orders agree to the 
terms of a trade.'').

    A system uses established non-discretionary methods either by 
providing a trading facility or by setting rules governing trading 
among subscribers. The Commission intends for ``established, non-
discretionary methods'' to include any methods that dictate the 
terms of trading among the multiple buyers and sellers entering 
orders into the system. Such methods include those that set 
procedures or priorities under which open terms of a trade may be 
determined. For example, traditional exchanges' rules of priority, 
parity, and precedence are ``established non-discretionary 
methods,'' as are the trading algorithms of electronic systems. 
Similarly, systems that determine the trading price at some 
designated future date on the basis of pre-established criteria 
(such as the weighted average trading price for the security on the 
specified date in a specified market or markets) are using 
established, non-discretionary methods.\43\
---------------------------------------------------------------------------

    \43\ See Securities Exchange Act Release No. 40760 (December 8, 
1998), 63 FR 70844, 70850 (December 22, 1998).

    Nothing in the Reg. ATS Adopting Release or in any of its 
illustrative examples suggests that Dynamic M-ELO would constitute an 
exercise of discretionary behavior. Dynamic M-ELO will handle and 
execute Orders according to published, pre-determined rules that are 
disclosed to the public and which provide reasonable notice of how the 
Order Type will behave.\44\ To the extent that the design of the System 
permits variation in the Holding Periods for such Orders, it does so by 
design. The range of potential variations, the objectives that such 
variations are intended to achieve, and the factors that determine when 
such variations may occur are also predetermined and set forth in the 
Exchange's Rules or otherwise disclosed to the public. The mere fact 
that the System may apply different weights over time to the factors it 
uses to determine whether and by how much to vary a Holding Period does 
not mean that the System will act with discretion in the same sense 
that a human being could be said to be exercise independent judgment 
when deciding whether and how to handle an order.\45\ Even when the 
System makes decisions about changing the Holding Periods, the System 
will operate pursuant to a mathematical algorithm from which it cannot 
deviate--an algorithm that is programmed to achieve pre-defined and 
pre-disclosed objectives.\46\
---------------------------------------------------------------------------

    \44\ See id. at 70900 (``an essential indication of the non-
discretionary status of rules and procedures is that those rules and 
procedures are communicated to the systems users'' and ``[t]hus, 
participants have an expectation regarding the manner of execution--
that is, if an order is entered, it will be executed in accordance 
with those procedures and not at the discretion of a counterparty or 
intermediary.'').
    \45\ Cf. id. at 70851 (explaining that a traditional block 
trading desk is an example of a system that does not use 
established, non-discretionary methods because the operators of such 
desks do not act according to fixed procedures known to their 
customers, but instead shop orders around for potential 
counterparties and make their own determinations as to whether and 
how to execute block orders, including by sometimes deciding to take 
a proprietary position in part of the block order).
    \46\ See id. at 80755 (describing an example of a system that 
would be non-discretionary in nature: ``System I permits 
participants to enter a range of ranked contingent buy and sell 
orders at which they are willing to trade securities. These orders 
are matched based on a mathematical algorithm whose priorities are 
designed to achieve the participants' objectives. System I does not 
display orders to any participants. System I is included under Rule 
3b-16.''); see also Securities Exchange Act Release No. 34-89686 
(August 20, 2020), 85 FR 54438, at 54445, n.92 (September 1, 2020) 
(Order approving SR-IEX-2019-15) (rejecting argument that IEX's D-
Limit order time is an exercise of discretion because ``D-Limit 
orders will not allow IEX to exercise any discretion on any 
particular order by deviating from the CQI and D-Limit 
functionality, which is hardcoded in the IEX rulebook.''; Securities 
Exchange Act Release No. 34-78101 (June 17, 2016), 81 FR 41141, at 
41153(June 17, 2016) (Order approving IEX Form 1 and D-Peg Order 
Type) (``the Commission does not believe that the hardcoded 
conditionality of the IEX proposed ``discretionary'' peg order type 
provides IEX with actual discretion or the ability to exercise 
individualized judgment when executing an order. Rather, if IEX's 
fixed formula determines the quote to be stable, the discretionary 
peg order can execute up to the midpoint; if it does not deem the 
quote to be stable, then it will hold the order to its pegged price. 
As such, IEX would not exercise discretion over the routing and 
execution of a resting order''). Nasdaq does not believe that it is 
necessary to codify its mathematical formula for Dynamic M-ELO in 
its Rules because Nasdaq has disclosed sufficient information in its 
Rules and in its filing to inform the public as to the possible and 
expected behaviors associated with Dynamic M-ELO, as well as a means 
for the Commission and/or investors to verify whether Dynamic M-ELO 
is performing appropriately. Much as the Commission does not require 
an exchange to codify the source code it uses to effectuate other 
behaviors or actions that it explains in its Rules, including the 
behaviors of other complex Order Types, there is no basis to require 
codification of the Dynamic M-ELO formula in this instance.

---------------------------------------------------------------------------

[[Page 62857]]

    The Exchange notes that it will continue to conduct real-time 
surveillance to monitor the use of M-ELOs and M-ELO+CBs to ensure that 
such usage remains appropriately tied to the intent of the Order Types. 
If, as a result of such surveillance, the Exchange determines that the 
Dynamic M-ELO Holding Periods do not serve their intended purposes, or 
adversely impact market quality, then the Exchange will seek to make 
further re-calibrations.
    Nasdaq does not believe that the design of Dynamic M-ELO lends 
itself to potential manipulation by a single participant or a small 
group of participants because the System makes determinations regarding 
Holding Periods based upon prevailing market-wide conditions for a 
given symbol, rather than the behaviors of particular participants with 
respect to that symbol, or the activity of participants in M-ELOs 
involving that symbol. Manipulation of the System also would be 
difficult to accomplish given the large number of variables that factor 
into the System's decisions to change Holding Periods during Change 
Events, as well as the different weights that apply to each such 
factor, which as described above, the System may vary over time. Any 
benefits that a participant might derive from manipulating the duration 
of Holding Periods would likely be small and outweighed significantly 
by the difficulty and cost of affecting such manipulation. 
Nevertheless, the Exchange will surveil for indications of manipulation 
and act accordingly if it detects such indications.

B. Self-Regulatory Organization's Statement on Burden on Competition

    The Exchange does not believe that the proposed rule change will 
impose any burden on competition not necessary or appropriate in 
furtherance of the purposes of the Act. To the contrary, the Exchange 
believes that this proposal will promote the competitiveness of the 
Exchange by rendering its M-ELO and M-ELO+CB Order Types more 
attractive to participants.
    The Exchange adopted the M-ELO and M-ELO+CB as pro-competitive 
measures intended to increase participation on the Exchange by allowing 
certain market participants that may currently be underserved on 
regulated exchanges to compete based on elements other than speed. The 
proposed change continues to achieve this purpose. With Dynamic M-ELO 
Holding Periods, both M-ELOs and M-ELO+CBs will afford their users with 
a level of protection from information leakage and adverse selection 
that is better from what is achievable at present.\47\ At the same 
time, the Dynamic Holding Periods will increase opportunities to 
interact with other like-minded investors with longer time horizons 
while also lowering the opportunity costs for participants that utilize 
M-ELOs and M-ELO+CBs, particularly for securities that trade within the 
``Goldilocks'' zone. In sum, the proposed changes will not burden 
competition, but instead may promote competition for liquidity in M-
ELOs and M-ELO+CBs by broadening the circumstances in which market 
participants may find such Orders to be useful. With the proposed 
changes, market participants will be more likely to determine that the 
benefits of entering M-ELOs and M-ELO+CBs outweigh the risks of doing 
so.
---------------------------------------------------------------------------

    \47\ See White Paper, supra.
---------------------------------------------------------------------------

    The proposed change will not place a burden on competition among 
market venues, as any market may adopt an order type that operates 
similarly to a M-ELO or a M-ELO+CB with Dynamic M-ELO Holding Periods.

C. Self-Regulatory Organization's Statement on Comments on the Proposed 
Rule Change Received From Members, Participants, or Others

    No written comments were either solicited or received.

IV. Discussion and Commission Findings

    After careful review, the Commission finds that the Exchange's 
proposal to adopt Dynamic M-ELO is consistent with the requirements of 
the Act and the rules and regulations thereunder applicable to a 
national securities exchange.\48\ In particular, the Commission finds 
that the proposed rule change, as modified by Amendment No. 2, is 
consistent with Sections 6(b)(5) and 6(b)(8) of the Act.\49\ Section 
6(b)(5) of the Act requires that the rules of a national securities 
exchange be designed, among other things, to prevent fraudulent and 
manipulative acts and practices, to promote just and equitable 
principles of trade, and, in general, to protect investors and the 
public interest, and not be designed to permit unfair discrimination 
between customers, issuers, brokers, or dealers.\50\ Section 6(b)(8) of 
the Act requires that the rules of a national securities exchange not 
impose any burden on competition that is not necessary or appropriate 
in furtherance of the purposes of the Act.\51\
---------------------------------------------------------------------------

    \48\ In approving this proposed rule change, the Commission has 
considered the proposed rule's impact on efficiency, competition, 
and capital formation. See 15 U.S.C. 78c(f). One commenter 
questioned whether Nasdaq's pending patent applications for the 
systems it will use to operate Dynamic M-ELO imposes an unnecessary 
or inappropriate burden on competition. See Letter from R.T. 
Leuchtkafer, dated January 21, 2023 (``Leuchtkafer Letter 1''), at 
3. The Commission does not believe that the sole fact that Nasdaq 
has a pending patent application for the technology it has developed 
to operate the Dynamic M-ELO is indicative that the operation of 
Dynamic M-ELO on the Exchange would place an inappropriate burden on 
competition. As explained below, Nasdaq has provided sufficient 
public disclosure and analysis to explain how Dynamic M-ELO will 
operate.
    \49\ In addition to providing a statutory analysis in its 
filing, Nasdaq also acknowledges, above in Amendment No. 2, that the 
systems it will use to implement Dynamic M-ELO, including the 
Exchange's model development and retraining processes, are SCI 
systems under Regulation SCI, see 17 CFR 242.1000 et seq., and thus, 
it will be responsible for compliance with Regulation SCI with 
respect to Dynamic M-ELO, including having appropriate policies and 
procedures. See supra note 33 and accompanying text.
    \50\ 15 U.S.C. 78f(b)(5).
    \51\ 15 U.S.C. 78f(b)(8).
---------------------------------------------------------------------------

Nasdaq's Obligation To Sufficiently Explain Its Proposed Rule Change

    The burden to demonstrate that a proposed rule change is consistent 
with the Act and rules and regulations thereunder is on the self-
regulatory organization (``SRO'') proposing a rule change.\52\ Each 
proposed SRO rule change must be ``accompanied by a concise general 
statement of the basis and purpose of such proposed rule change.'' \53\ 
As described in more detail below, several commenters argued that the 
proposal did not provide sufficient information with respect to the 
operation of Dynamic M-ELO, or that the information provided was not 
``clear and comprehensible,'' as required by Form 19b-4. For the 
reasons articulated below, the Commission believes that Nasdaq has 
provided clear and comprehensible information on the overall operation 
of Dynamic M-ELO and the role of the machine-learning model and 
demonstrated that the proposal is consistent with the Act. Several 
related comments addressed this issue; these comments and Nasdaq's 
responses are discussed below, followed by the Commission's analysis.
---------------------------------------------------------------------------

    \52\ See 17 CFR 201.700(b)(3).
    \53\ 15 U.S.C 78s(b)(1).
---------------------------------------------------------------------------

    One commenter stated that the initial filing would establish ``a 
dangerously vague standard for describing how exchange-hosted complex 
algorithmic

[[Page 62858]]

order types operate.'' \54\ In response to public comment, Nasdaq added 
more details describing the operation of Dynamic M-ELO to both the 
filing and public record since this proposed rule change was initially 
submitted to the Commission, and Nasdaq also provided additional legal 
analysis to support Dynamic M-ELO's consistency with the Act. Prior to 
the filing of Amendment No. 1, a commenter stated that although 
``Nasdaq shared some of the 142 features of their formula,'' Nasdaq 
should reveal all of these features so that prospective users may 
evaluate how the model works.\55\ Similarly, another commenter stated 
that the public cannot provide meaningful comment on the proposal 
without knowing all categories and parameters of the proposed Dynamic 
M-ELO.\56\ In its response to these comments, the Exchange, among other 
things, provided the specific 142 data elements that will be weighed by 
the machine-learning model as both an appendix to its first letter in 
response to comments,\57\ and as Exhibit 3B to its Amendment No. 1 
filing.
---------------------------------------------------------------------------

    \54\ See Leuchtkafer Letter 1, supra note 48, at 1-2. See also 
Letter from Joseph Saluzzi, Partner, Themis LLC, dated January 25, 
2023, at 3 (``Themis Letter'') (questioning whether the complexity 
of Dynamic M-ELO is necessary).
    \55\ See Themis Letter, supra note 54, at 2.
    \56\ See Leuchtkafer Letter 1, supra note 48, at 1.
    \57\ See Letter from Brett Kitt, Associate Vice President and 
Principal Associate General Counsel, Nasdaq, Inc., dated March 9, 
2023, at Appendix A (``Nasdaq First Response to Comments'').
---------------------------------------------------------------------------

    In response to the Exchange's disclosures in Exhibit 3B of 
Amendment No. 1, one of these commenters stated that the list of data 
elements was not ``clear and comprehensible'' as is required by the 
Form 19-4, but rather ``vague, confusing, and perfunctory.'' \58\ This 
commenter also stated that the disclosed data elements included 
unexplained terms (e.g., ``baseline simulated,'' ``action simulated,'' 
and ``synthetic mark-out'').\59\ In a subsequent comment letter, this 
commenter reiterated these points; the commenter specified that the 
commenter's concern is that Nasdaq's rule text does not disclose 
information about its methods for assessing market conditions and that 
``Nasdaq should carefully detail its methods in its rulebook, just like 
other exchanges have done, and Nasdaq should also thoroughly disclose 
its methods in its filing text.'' \60\ In its second response to 
comments \61\ and the revised Exhibit 3b to Amendment No. 2, Nasdaq 
expanded and ``simplified'' the explanation of these 142 data elements. 
The Exchange also added to the proposed rule text a definition of the 
term ``proprietary assessment of market conditions'' to explain how the 
machine-learning model will evaluate those 142 data elements.\62\
---------------------------------------------------------------------------

    \58\ See Letter from R.T. Leuchtkafer, dated May 2, 2023, at 8-9 
(``Leuchtkafer Letter 2'').
    \59\ See id.
    \60\ See Letter from R.T. Leuchtkafer dated May 30, 2023, at 3-
5; 8-9 (``Leuchtkafer Letter 3''). This commenter also cites to the 
rules governing the Crumbling Quote Indicator and D-Limit order type 
on the Investors Exchange (``IEX''), as well as language from the 
Commission's approval order for the D-Limit order type. See id. at 
4. The commenter notes the level of detail with regard to how and 
when the D-Limit order type exercises its discretionary price-
sliding that is set forth in the IEX Rulebook. See id. at 4; see 
also Themis Letter, supra note 54, at 2 (``Another exchange, IEX, 
operates a smart logic called CQI (Crumbling Quote Indicator) which 
aims to protect orders from being adversely selected. IEX has 
published detailed notes on how the CQI is calculated.''). Each 
proposal must be evaluated based on the specific facts and 
circumstances before the Commission. In this case, the Commission is 
only reviewing the proposed operation of Dynamic M-ELO and its 
machine-learning model. Accordingly, the level of detail provided in 
the IEX Rulebook for the D-Limit order type and Crumbling Quote 
Indicator--or the rulebooks for order types on other exchanges--does 
not determine whether Nasdaq has met its burden in this proposal.
    \61\ See Letter from Brett Kitt, Associate Vice President and 
Principal Associate General Counsel, Nasdaq, Inc., dated May 18, 
2023 at Appendix A (``Nasdaq Second Response to Comments'').
    \62\ See supra note 25.
---------------------------------------------------------------------------

    Furthermore, Nasdaq attached, as Exhibit 3A to its proposed rule 
change, the White Paper written by its AI Core Development Team that 
explains, among other things, how Dynamic M-ELO's machine-learning 
functions were developed and tested. The White Paper includes a general 
discussion of the type of model implemented in the proposed system, in 
this case a reinforcement learning model,\63\ as well as citations to 
academic research behind the Double Deep Q-Network algorithm that is 
the basis for the algorithm used in Nasdaq's model.\64\ The White Paper 
also describes the ways in which Nasdaq's implementation of the 
proposed model differs from the model and training in the academic 
research, providing both an English summary and a pseudocode 
description of differences in model training implemented by Nasdaq.\65\
---------------------------------------------------------------------------

    \63\ See White Paper Section 3.1.
    \64\ See White Paper Section 4.1.
    \65\ See White Paper, Section 7.2.
---------------------------------------------------------------------------

    One of the commenters stated that the White Paper is not easily 
understood by most market participants and that referencing the White 
Paper in the filing is an ``unacceptable substitute'' for a ``plain 
English'' explanation of the proposal in Form 19b-4.\66\ In response, 
Nasdaq explained that it drafted the filing to provide a general 
understanding of Dynamic M-ELO and how it will behave, and the more 
detailed information and explanation in the White Paper are meant to 
support the filing.\67\
---------------------------------------------------------------------------

    \66\ See Leuchtkafer Letter 2, supra note 58, at 3-4; 
Leuchtkafer Letter 3, supra note 60, at 4-5; Letter from R.T. 
Leuchtkafer, dated August 11, 2023, at 8-10 (``Leuchtkafer Letter 
4'').
    \67\ See Nasdaq Second Response to Comments, supra note 61, at 
3-5.
---------------------------------------------------------------------------

    The Commission agrees with comments and the Exchange that there is 
an extent to which the proposed changes will introduce an unavoidable 
degree of uncertainty with respect to the use of these order types. The 
deep reinforcement learning model that will determine the dynamic 
holding periods for each symbol for M-ELO and M-ELO+CB orders will be 
implemented through established, non-discretionary methods,\68\ but it 
is so complex that its complete details are, for most intents and 
purposes, not readily intelligible, and it would be immensely difficult 
for the Exchange or any market participant to precisely predict the 
holding periods that will be generated by the model for any given 
symbol at any particular time. Nevertheless, as further discussed 
below, the Commission believes that the Exchange has provided 
information sufficient for the Commission and public to understand the 
design, operation, and limits of the proposed changes to these order 
types, and the role of the machine-learning model therein.
---------------------------------------------------------------------------

    \68\ See infra notes 91-92 and accompanying text.
---------------------------------------------------------------------------

    While the holding periods under the proposal would be dynamic, 
Nasdaq has precisely articulated both the nature of changes that would 
be permissible under the proposal, and the limits to those changes. 
Nasdaq described when changes might occur (every thirty seconds 
throughout the trading day), the initial default holding period for all 
symbols (1.25 milliseconds), the permissible increments by which a 
holding period might change in each symbol (0.25 or 0.50 milliseconds), 
and the outer bounds of permissible holding period lengths (0.25 
milliseconds at the short end, and 2.50 milliseconds at the long end). 
Nasdaq also described the conditions of ``extraordinary instability'' 
in a symbol when these holding periods would not apply, and when the 
holding periods would be overridden by the proposed ``stability 
protection mechanism'' (with a holding period of 12 milliseconds for at 
least 750 milliseconds). The Commission believes that these details 
provide sufficient information to understand the range of potential 
holding periods that may be applied when M-ELO or M-ELO+CB orders are 
entered or resting on the order book, the changes that may occur, and 
the limits to those changes.

[[Page 62859]]

    Nasdaq has also described the role of the machine-learning model in 
its proposal. The model will determine whether, by which increment, and 
in which direction to adjust the holding period for each symbol 
throughout the trading day.\69\ In its Form 19b-4, White Paper, and 
response letters, Nasdaq described the goals towards which the model is 
optimized: reducing mark-outs and increasing fill rates.\70\ Nasdaq's 
White Paper includes a detailed discussion of model choice, 
development, and training, including citations to relevant other 
research.\71\ Nasdaq also provided several iterations of a list of data 
elements that the model will ingest and use, including a glossary 
defining terms used in the descriptions.\72\ Nasdaq affirmed that in 
operation during market hours, the data used would be calculated based 
on intraday market data.\73\ One version of these lists included 
Nasdaq's estimates of the tendencies of data elements to affect model 
outcomes.\74\ Nasdaq's White Paper also included an ``explainability 
study'' that assessed both the effects of individual data elements on 
model performance and the effects of interactions between individual 
data elements.\75\ Across its filing and incorporated exhibits, aspects 
of the model's operations and design are described in different formats 
and with different levels of specificity--for example, the filing and 
exhibits include ``plain English'' descriptions, mathematical 
definitions, and pseudocode. Together, this set of information allows 
the Commission to understand the type of decision the model will 
implement, the goals the model aims to achieve, which model type is 
implemented and how it was developed, the range of data types and data 
sources used by the model, and estimates of the manner in which those 
data may affect model outcomes.
---------------------------------------------------------------------------

    \69\ See supra notes 16-24 and accompanying text.
    \70\ See, e.g., Form 19b-4 at 9, White Paper Section 5, and 
Nasdaq First Response to Comments at 2. In its White Paper, Nasdaq 
provides mathematical definitions of fill rate for a period of time 
and mark-out by trade (White Paper at 5, Equations 1 and 2), as well 
as of the assessment made by the agent in the model's reinforcement 
learning process (White Paper at 11, Equation 3).
    \71\ See White Paper Sections 3-5.
    \72\ See Exhibit 3B. As described below, these data elements are 
also those used in training and retraining the model.
    \73\ See Letter from Brett Kitt, Associate Vice President and 
Principal Associate General Counsel, Nasdaq, Inc., dated September 
6, 2023 (``Nasdaq Third Response to Comments'').
    \74\ Nasdaq affirmed that, while this information was not 
included in all versions of the list of data elements, it remains 
accurate and valid. See id. at 3.
    \75\ See White Paper Section 5.3.
---------------------------------------------------------------------------

    Nasdaq also explains how and when the machine-learning model will 
be retrained. Nasdaq will retrain the model weekly, outside of market 
hours. Retraining will incorporate market data obtained during the week 
from the equity consolidated data feeds and M-ELO order book. A 
retrained model will only be promoted to production if it improves upon 
the model objectives compared to the prevailing model.\76\ Furthermore, 
the Exchange explained that the machine-learning model is consistent in 
its behavior from day-to-day during periods when it is not undergoing 
retraining, ``such that its decisions when presented with given set of 
facts and circumstances in a given security on day 1 should be the same 
as they would be on day 2.'' \77\ The Exchange also stated in its 
initial response to comments that ``[e]ven after the system undergoes 
retraining, which will occur on a weekly basis (and not during market 
hours), system behavior should not change dramatically or in unexpected 
ways from week-to-week.'' \78\ As noted above as well, the Exchange 
also represents that outside of set retraining periods, ``the System 
will operate pursuant to a mathematical algorithm from which it cannot 
deviate--an algorithm that is programmed to achieve pre-defined and 
pre-disclosed objectives.'' \79\ Nasdaq also will publish equity trader 
alerts when it anticipates that a model update may change mark-outs or 
fill rate by 10% or more in either direction.\80\ By including this set 
of information, Nasdaq has provided the Commission and public with 
information that allows them to understand how frequently the model 
will be retrained, the data used for retraining, and the criteria that 
will be used to determine whether to update the production model based 
on retraining. This information allows the Commission to understand 
when the proposed model may change and when it will remain constant, 
the circumstances under which a change would be implemented, and 
circumstances under which the public will receive notice of significant 
changes in the model's anticipated outcomes.
---------------------------------------------------------------------------

    \76\ For example, Nasdaq affirmatively states that if ``a 
retrained model would be worse than the existing model in achieving 
its objectives, then the System will continue to use the existing 
model and discard the retrained model.'' See Section III.A.1., 
supra.
    \77\ See Nasdaq First Response to Comments, supra note 57, at 2-
3. See also supra note 30.
    \78\ See Nasdaq First Response to Comments, supra note 57, at 2-
3. A commenter also noted that it was initially unclear when and how 
frequently the machine-learning model would retrain, stating that 
the White Paper set forth an analysis based on daily retraining, but 
the rule filing proposes weekly retraining. See Leuchtkafer Letter 
2, supra note 58, at 4; Leuchtkafer Letter 3, supra note 60, at 2; 
Leuchtkafer Letter 4, supra note 66, at 9. In Amendment No. 2, the 
Exchange affirmatively represents that ``the performance statistics 
for Dynamic M-ELO cited herein and in the White Paper are based upon 
data derived from weekly, not daily retrainings.'' See supra note 
38.
    \79\ See supra note 46 and accompanying text.
    \80\ See, e.g., Amendment No. 2 at 19-20.
---------------------------------------------------------------------------

    In addition, a commenter stated their belief that the Exchange's 
proposal for Dynamic M-ELO would result in the exercise of discretion 
by a national securities exchange because the machine-learning model's 
decisions would vary over time based on the following: (1) varying 
parameter values; and (2) results of retraining cycles.\81\ The 
commenter stated that by making it possible for Dynamic M-ELO to behave 
differently when confronted by the same market conditions before and 
after the model is retrained, Nasdaq's model would be exercising 
discretion that is more akin to a broker than an exchange.\82\
---------------------------------------------------------------------------

    \81\ See Leuchtkafer Letter 1, supra note 48, at 2; Leuchtkafer 
Letter 2, supra note 58, at 4-6; Leuchtkafer Letter 3, supra note 
60, at 5-7; Leuchtkafer Letter 4, supra note 66, at 4-8.
    \82\ See Leuchtkafer Letter 1, supra note 48, at 2. See also 
Leuchtkafer Letter 3, supra note 60, at 5-7. For example, the 
commenter believes that Dynamic M-ELO will ``exercise individualized 
judgment'' such that it can set a different time-in-force for the 
very same order presented in the very same market conditions on, for 
example, August 21 than it set on May 15, depending on the system's 
undisclosed individualized judgments of market conditions and 
participant behavior from even days or weeks in the past. See 
Leuchtkafer Letter 4, supra note 66, at 6.
---------------------------------------------------------------------------

    The commenter claimed that Dynamic M-ELO would operate outside of 
established non-discretionary methods, which require ``fully disclosed 
procedures operating in a strictly linear, invariant, and deterministic 
fashion.'' \83\ Additionally, the commenter stated that Nasdaq would be 
exercising discretion with Dynamic M-ELO to alter a participant's 
material order terms.\84\ The commenter claimed that Nasdaq would be 
using undisclosed data such as the

[[Page 62860]]

buyer and seller counts and recent trade sizes.\85\ Further, the 
commenter stated:
---------------------------------------------------------------------------

    \83\ See Leuchtkafer Letter 2, supra note 58, at 5-6. See also 
Leuchtkafer Letter 3, supra note 60, at 5-7. See also Leuchtkafer 
Letter 4, supra note 66, at 6-9 ('' . . . . Nasdaq's rulebook won't 
set out the `totality of the discretionary feature' (I believe it 
can't, because the totality changes week-to-week and even minute-to-
minute) and it won't define the `hardcoded conditionality' of its 
feature (again, I believe it can't), and a market participant won't 
be able to `recreate on its own' what Dynamic M-ELO has done 
(participants can't--it's not even clear anyone will be able to, as 
discussed below) . . . . Dynamic M-ELO departs from decades of this 
progress. Its behavior will not be deterministic or invariant over 
time, and purposefully so.'').
    \84\ See Leuchtkafer Letter 2, supra note 58, at 8. See also 
Leuchtkafer Letter 3, supra note 60, at 5-7.
    \85\ See Leuchtkafer Letter 2, supra note 58, at 8. See also 
Leuchtkafer Letter 3, supra note 60, at 5-7; Leuchtkafer Letter 4, 
supra note 66, at 5-7.

that (a) by determining the universe of data the system consumes, 
(b) by programming how the system thinks, (c) by controlling and 
supplying the information with which it thinks, and (d) by setting 
the goals and programming the nature and extent of its actions, and 
when it does all this to determine (e) when and in which prescribed 
intervals to set an ever variable time-in-force term for an order, a 
term which (f) dictates when to expose an order to the market to 
find contra-side interest, then without question Nasdaq is 
exercising control, judgment, and discretion over its customer 
---------------------------------------------------------------------------
orders.

    In Amendment No. 1, Nasdaq added language to address these 
concerns.\86\ Among other things, Nasdaq stated that to the extent that 
the design of Dynamic M-ELO permits variation in the Holding Periods 
for such orders, it does so by design, and the ``mere fact that the 
System may apply different weights over time to the factors it uses to 
determine whether and by how much to vary a Holding Period does not 
mean that the System will act with discretion in the same sense that a 
human being could be said to be exercise independent judgment when 
deciding whether and how to handle an order.'' \87\ Additionally, 
Nasdaq stated the following in its second response to comments:
---------------------------------------------------------------------------

    \86\ See supra notes 42-46.
    \87\ See supra note 45 and accompanying text.

    It is also worth noting that presently, exchanges like Nasdaq 
already employ non-linear, non-deterministic functionalities, like 
the randomized timers it uses to resolve certain unavoidable race 
conditions that arise in the order handling process. Nasdaq employs 
these functionalities with the knowledge of the SEC, and without any 
suggestion that they somehow transform Nasdaq into a broker.\88\
---------------------------------------------------------------------------

    \88\ See Nasdaq Second Response to Comments, supra note 61, at 
5-9. See also Nasdaq First Response to Comments, supra note 57, at 
5-7.

Furthermore, as noted above, the Exchange represents that outside of 
set retraining periods, ``the System will operate pursuant to a 
mathematical algorithm from which it cannot deviate--an algorithm that 
is programmed to achieve pre-defined and pre-disclosed objectives.'' 
\89\ The Exchange explains that outside of the set retraining periods 
``the System will behave predictably from day to day, such that its 
decisions when presented with given set of facts and circumstances in a 
given security on day 1 should be the same as they would be on day 2.'' 
\90\
---------------------------------------------------------------------------

    \89\ See supra note 46 and accompanying text.
    \90\ See supra note 29.
---------------------------------------------------------------------------

    Based on Nasdaq's representations described above, Dynamic M-ELO 
would operate pursuant to pre-determined, programmed procedures that 
would dictate order interaction and the terms for trading for each 
Dynamic M-ELO order entered on the Nasdaq trading facility. While the 
Exchange's procedures include conditions that, if satisfied under 
certain circumstances, might result in different outcomes for different 
M-ELO orders, such conditions and circumstances, if pre-determined, 
pre-defined, and programmed into the Exchange's trading facility, would 
be considered established and not discretionary. For example, according 
to the Exchange, Dynamic M-ELO may apply different pre-determined 
weights over time to pre-determined factors it uses to determine 
whether and by how much to vary a Holding Period.\91\ In such an event, 
Dynamic M-ELO will operate pursuant to pre-determined procedures and 
programmed mathematical algorithm from which it cannot deviate to 
``achieve pre-defined and pre-disclosed objectives.'' \92\ Further, the 
procedures governing Dynamic M-ELO and use of M-ELO orders will be 
established before the beginning of each trading day. For example, 
Dynamic M-ELO will use preset methods to evaluate and weigh specific 
data elements to determine the dynamic holding periods. Such pre-set 
methods will be established during the prior retraining period, and 
outside regular trading hours, and will not vary intra-day until 
adjusted at the next retraining period.
---------------------------------------------------------------------------

    \91\ See supra note 45 and accompanying text.
    \92\ See supra note 46 and accompanying text.
---------------------------------------------------------------------------

    Given the pre-determined, programmed procedures and rules that 
Nasdaq has proposed to dictate trading for Dynamic M-ELO, the 
Commission does not believe that Dynamic M-ELO is designed to provide 
Nasdaq with judgement and flexibility, and therefore, discretion over 
the handling or execution of a M-ELO order entered on the Exchange.

Unfair Discrimination

    Section 6(b)(5) of the Act requires that the rules of a national 
securities exchange not be designed to permit unfair discrimination 
between customers, issuers, brokers, or dealers. In several letters, 
one commenter stated that Nasdaq inadequately explains how it will 
monitor and, if necessary, adjust Dynamic M-ELO to ensure no unfair 
discrimination.\93\ Initially, this commenter emphasized what they 
perceived to be silence on the part of Nasdaq with regard to whether 
Dynamic M-ELO will discriminate among categories of participant 
types.\94\ In its response to these comments, Nasdaq initially added a 
new representation to the filing in Amendment No. 1, stating that that 
Dynamic M-ELO is not designed to further the performance of any 
participant or any category of participant, but instead has twin 
objectives--the absolute values of mark-outs and fill rates. In 
Amendment No. 2, Nasdaq expanded on this representation by adding the 
following:
---------------------------------------------------------------------------

    \93\ See Leuchtkafer Letter 3, supra note 60, at 7-8; 
Leuchtkafer Letter 2, supra note 58, at 5; Leuchtkafer Letter 1, 
supra note 48, at 3. Nasdaq's White Paper includes a ``firm-level 
analysis'' that ``tried to identify patterns and trends that could 
potentially signify a systematic bias towards specific firms.'' 
White Paper at 24. This analysis concluded that ``Dynamic M-ELO will 
not result in systematic-biased execution towards any one firm.'' 
Id. at 26.
    \94\ See Leuchtkafer Letter 3, supra note 60, at 7-8.

    Furthermore, Nasdaq performed internal tests of its AI model to 
detect indications of harmful bias in its performance results, and 
such tests concluded that no such indications exist. That is, the 
Exchange reviewed the impact on fill rates and mark-outs of Dynamic 
M-ELO, as compared to the ``static'' M-ELO, for those firms that 
accounted for more than 95% of M-ELO activity on the Exchange during 
Q1 2022 . . . . The Exchange analyzed results both in an absolute 
and a relative sense. Testing revealed that all participants 
experienced at least some improvements in fill rates and mark-outs 
when using Dynamic M-ELO versus static M-ELO, with the volume-
weighted average improvement being aligned with the results 
expressed in the White Paper. We detected no material variations 
that might suggest that a particular participant or category of 
participant (i.e., nature of firm; size of firm) benefitted from 
Dynamic M-ELO functionality to an extent that was unreasonably 
disproportionate to the benefits that other participants 
experienced. Thus, Nasdaq believes the model is objective, is 
---------------------------------------------------------------------------
designed to, and does avoid bias and discrimination.

    In Amendment No. 2, Nasdaq also affirmed that it will periodically 
review its model to ensure that it continues to perform in accordance 
with the Exchange's rules and that it has not introduced any harmful 
bias in favor of or against any participant or class of 
participants.\95\
---------------------------------------------------------------------------

    \95\ See Amendment No. 2 at fn. 34.
---------------------------------------------------------------------------

    In response to the above, the commenter submitted a fourth comment 
letter, in which they questioned the approach Nasdaq took to 
demonstrate that there is not bias against any one

[[Page 62861]]

participant or class of participants.\96\ The commenter, among other 
things, expressed concern about Nasdaq conducting its analysis using 
data for firms that accounted for 95% of M-ELO activity during Q1 of 
2022 rather than all M-ELO activity. The commenter states that Nasdaq 
did not describe how it determined the 5% of activity during that 
period to exclude from its analysis.\97\ For example, the commenter 
states that it is not clear whether Nasdaq excluded firms with large 
orders and trades, and the commenter opines that discarding any data 
could exclude activity that has qualitative or quantitative differences 
from the rest.\98\
---------------------------------------------------------------------------

    \96\ See Leuchtkafer Letter 4, supra note 66, at 1-4.
    \97\ See id. at 2-3 (``It seems Nasdaq trimmed its data before 
analyzing it for bias and constrained its analysis to `those firms 
that accounted for more than 95% of M-ELO activity on the Exchange 
during Q1 2022.' (I assume Nasdaq used the same data defined in the 
Filing as the `Training Period' for its analysis. Nasdaq doesn't say 
so, however.) Nasdaq doesn't describe the kind of M-ELO `activity' 
it filtered the data for, and specifically whether it filtered on 
order or trade counts or order or trade volume or some combination 
of two or more of these categories, or on some other factor, before 
removing firms from its analysis.'')
    \98\ See id. at 2-3.
---------------------------------------------------------------------------

    In response to this comment, Nasdaq represented that it conducted a 
supplemental analysis of the initially-excluded data--which were the 
activity of the least-active M-ELO firms from the control period of its 
initial analysis--to confirm whether its initial conclusions held for 
those participants.\99\ Nasdaq explains that the individual variations 
among the previously excluded participants was higher than that for the 
original batch of data, but that, based on simulated data, each of 
these participants would have experienced the same or better fill rates 
during the testing period if they had utilized Dynamic M-ELO.\100\ 
Based on this supplemental data analysis, Nasdaq concluded that there 
is no apparent biases for the Dynamic M-ELO, even among the least 
active M-ELO participants.\101\
---------------------------------------------------------------------------

    \99\ See id. at 5-6.
    \100\ See id.
    \101\ See id.
---------------------------------------------------------------------------

    The Commission concludes that Nasdaq has adequately demonstrated 
that the proposal is not designed to permit unfair discrimination 
consistent with Section 6(b)(5) of the Act. Through the White Paper, 
amendments, and response letters, Nasdaq has demonstrated that it has 
analyzed the anticipated or simulated effects of the proposed change on 
all current M-ELO users, and that this work did not indicate that 
particular firms or classes of firms are anticipated to unfairly 
benefit from or be harmed by the proposed Dynamic M-ELO functionality.

Prevention of Fraudulent and Manipulative Acts and Practices, Just and 
Equitable Principles of Trade, and the Protection of Investors and the 
Public Interest

    Section 6(b)(5) of the Act also requires that the rules of a 
national securities exchange be designed to prevent fraudulent and 
manipulative acts and practices, to promote just and equitable 
principles of trade, and, in general, to protect investors and the 
public interest. In Amendment No. 2, Nasdaq addressed whether the 
Dynamic M-ELO is designed to prevent fraudulent and manipulative acts 
and practices. Nasdaq states that the design of Dynamic M-ELO does not 
lend itself to potential manipulation by a single participant or a 
small group of participants because the machine-learning model makes 
determinations regarding Holding Periods based upon prevailing market-
wide conditions for a given symbol, rather than the behaviors of 
particular participants with respect to that symbol, or the activity of 
participants in M-ELOs involving that symbol. Nasdaq further states 
that manipulation of the machine-learning model would be difficult to 
accomplish given the large number of variables that factor into the 
machine-learning model's decisions to change Holding Periods during 
Change Events, as well as the different weights that apply to each such 
factor, which as described above, may vary over time. Furthermore, 
Nasdaq states that any benefits that a participant might derive from 
manipulating the duration of Holding Periods would likely be small and 
outweighed significantly by the difficulty and cost of effecting such 
manipulation.
    The Exchange, in Amendment No. 2, also sets forth representations 
regarding how it will surveil its market after Dynamic M-ELO is 
implemented. First, Nasdaq represents that it will review the machine-
learning functionality and operation periodically to affirm that it 
continues to perform in accordance with the Exchange's rules and has 
not introduced any harmful bias in favor of or against any participant 
or category of participants.\102\ Nasdaq also represents above that it 
will surveil for indications of manipulation and act accordingly if it 
detects such indications.
---------------------------------------------------------------------------

    \102\ See supra note 41.
---------------------------------------------------------------------------

    The Commission finds that the proposed rule change is designed to 
prevent fraudulent and manipulative acts and practices consistent with 
Section 6(b)(5) of the Act. The Commission agrees that aspects of the 
Dynamic M-ELO design reduce opportunities for manipulation or are 
likely to make manipulation costly or difficult. The model's operation 
depends on 142 data elements, which are each likely to have effects on 
model outcomes of differing magnitudes and in different directions. 
Many of these data elements are also based on market-wide data, in some 
cases spanning periods of days,\103\ which are likely themselves 
difficult for market participants to manipulate. Given these design 
features, it appears likely that manipulating the duration of Dynamic 
M-ELO holding periods in any given symbol or group of symbols would be 
an extremely complex undertaking. In light of this complexity, and the 
size of M-ELO activity relative to the market for NMS stocks,\104\ 
Nasdaq's assertion that the potential benefits of manipulating the 
dynamic holding periods for these order types would be outweighed by 
the cost and complexity of manipulation also appears reasonable. Nasdaq 
has also represented that it intends to surveil the proposed order 
types for manipulation. This ongoing surveillance, to ensure the 
appropriate use of Dynamic M-ELO by Exchange Members and behavior by 
the machine-learning model, is important to the successful 
implementation of Dynamic M-ELO and appears appropriately tailored to 
the accomplish the intent of the M-ELO and M-ELO+CB order types.
---------------------------------------------------------------------------

    \103\ See, e.g., Exhibit 3b.
    \104\ According to the ``M-ELO Monthly Report'' published by 
Nasdaq for July 2023 (available at: https://www.nasdaq.com/docs/M-ELO-Monthly-Report (accessed September 2, 2023)), the average daily 
notional volume executed in M-ELO was $624,556,748. The average 
daily notional volume executed in July 2023 across the market for 
NMS stocks was about $523,769,246,196. See, e.g., Cboe, Historical 
Market Volume Data, available at: https://www.cboe.com/us/equities/market_statistics/historical_market_volume/. The average daily 
notional volume in M-ELO for that month was approximately 0.12% 
(just over one-tenth of one percent) of the average daily notional 
volume across the entire NMS stock market.
---------------------------------------------------------------------------

    Furthermore, the Commission finds that overall structure of Dynamic 
M-ELO--particularly, the static numerical constraints set forth in the 
proposed rule text--is designed in general, to protect investors and 
the public interest and promote just and equitable principles of trade 
pursuant to Section 6(b)(5) of the Act. As described above, the model 
will continuously engage in dynamic analysis of current market 
conditions during trading hours, and outside of market hours, it will 
retrain with the goal of improving the overall performance of Dynamic 
M-ELO. These dynamic aspects of the proposal,

[[Page 62862]]

however, are constrained by the static numerical thresholds set forth 
in the proposed rule text. For example, the initial Holding Periods for 
each trading day will be 1.25 milliseconds, the overall range for any 
Holding Period must be between 0.25 and 2.50 milliseconds during normal 
market conditions, and the Holding Period can only change by either 
0.25 or 0.50 milliseconds at each Change Event during normal market 
conditions. Regardless of how the model analyzes the current market or 
changes the weighting of the data elements as a result of its 
retraining, Dynamic M-ELO cannot operate outside of the static 
numerical ranges and limitations or minimums set forth in the rule 
text. As such, the Commission finds that Nasdaq has designed Dynamic M-
ELO to operate in a manner that in general protects investors and the 
public interest and promotes just and equitable principles of trade in 
accordance with Section 6(b)(5) of the Act.

Compliance With SRO Recordkeeping and Reporting Obligations

    One commenter queried whether Nasdaq could maintain an adequate 
audit trail given the potential for frequently shifting Holding Periods 
for Dynamic M-ELO.\105\ In response, Nasdaq states that it will retain 
copies of each iteration of its system as part of its books and records 
and will disclose publicly statistics relating to Dynamic M-ELO 
performance.\106\ Nasdaq additionally represented that it will publish 
weekly and monthly Dynamic M-ELO performance statistics, which would 
include the weekly numbers of shares and trades in M-ELOs by symbol, 
weekly aggregated M-ELO share and trade data, and monthly aggregated 
block data, on Nasdaqtrader.com.\107\ Nasdaq also indicated it would 
add statistics to its existing M-ELO Monthly Report, which discloses 
quote stability by time horizon, about how frequently, on average, its 
system changes Holding Period durations for the top decile, median, and 
bottom decile of symbols, as measured by monthly M-ELO and M-ELO+CB 
trading volumes.\108\
---------------------------------------------------------------------------

    \105\ See Leuchtkafer Letter 1, supra note 48, at 1.
    \106\ See Nasdaq First Response to Comments, supra note 57, at 
3.
    \107\ See Section III.A.1., supra.
    \108\ See id.
---------------------------------------------------------------------------

    Nasdaq also added a representation to the filing, addressing how it 
would comply with its recordkeeping obligations.\109\ Nasdaq states 
that it will retain copies of each historical iteration of its models 
as part of its books and records, and make them available to the 
Commission upon request, should it wish to examine them to understand 
how the model changes over time.\110\ Nasdaq also states that it will 
publish an equity trader alert in advance of deploying a retrained 
version of Dynamic M-ELO when Nasdaq anticipates the retrained version 
will produce results that differ materially from the prior 
version.\111\ Based on these representations, the Commission finds that 
Nasdaq has met its burden to demonstrate that it will comply with all 
relevant exchange recordkeeping requirements and obligations when it 
implements Dynamic M-ELO. In addition, the Commission notes that Nasdaq 
must comply with its reporting obligations under Rule 613 of Regulation 
NMS \112\ and the National Market System Plan Governing the 
Consolidated Audited Trail (``CAT NMS Plan'') \113\ with respect to 
Dynamic M-ELO, which requires it to record and electronically report to 
the central repository the material terms of each order and each 
reportable event.\114\
---------------------------------------------------------------------------

    \109\ See id.
    \110\ See id.
    \111\ See id.
    \112\ See 17 CFR 242.613.
    \113\ The CAT NMS Plan was approved by the Commission, as 
modified, on November 15, 2016. See Securities Exchange Act Release 
No. 79318 (November 15, 2016), 81 FR 84696 (November 23, 2016).
    \114\ See 17 CFR 242.613(c)(7).
---------------------------------------------------------------------------

Nasdaq's Obligation To File Proposed Rule Changes Relating to Dynamic 
M-ELO

    Prior to the filing of Amendment No. 1, a commenter stated that it 
was unclear what types of changes to the model would lead Nasdaq to 
seek approval from the Commission via an SRO rule filing.\115\ As 
explained above,\116\ Nasdaq represents that it will not modify the 
underlying structure of Dynamic M-ELO without first obtaining the 
Commission's approval to do so, including modifications to the data 
elements the model considers in making decisions about Holding Period 
durations, the conditions under which the model may adjust the duration 
of Holding Periods, the frequency with which the model may adjust the 
Holding Periods, the range of Holding Period durations available to M-
ELOs and M-ELO+CBs, the increments by which Holding Periods may change 
at any given Change Event, and the procedures for triggering, 
maintaining, and ending 12 millisecond Holding Periods during times of 
extraordinary instability. In contrast, the Exchange states that it 
will not seek Commission approval prior to retraining the model to 
adjust the weighting it applies to those data elements pursuant to the 
weekly retraining process.
---------------------------------------------------------------------------

    \115\ See Leuchtkafer Letter 1, supra note 48, at 2.
    \116\ See Section II.A.1., supra.
---------------------------------------------------------------------------

    Section 19(b)(1) of the Act \117\ and Rule 19b-4 thereunder \118\ 
require an SRO to file a proposed rule change with the Commission 
whenever it seeks any proposed change in, addition to, or deletion from 
the rules governing the SRO and its members' activities on the SRO. As 
discussed above, the proposal sets forth the specific data elements 
that Dynamic M-ELO will use during the trading day. Furthermore, the 
proposed rule change sets forth when the machine-learning model will 
retrain and the extent to which the retraining can and cannot cause the 
machine-learning model to update Dynamic M-ELO's operation during 
subsequent trading days.\119\ In addition, the proposal sets forth the 
operation of Dynamic M-ELO, such as the potential range for a Holding 
Period, how often Dynamic M-ELO reevaluates market conditions for a 
given security to adjust a Holding Period, and the increment by which a 
Holding Period may be changed. Nasdaq represents that it will not 
change any of these aspects of the proposal or any other function of 
Dynamic M-ELO without first filing a proposed rule change.\120\ Nasdaq 
does, however, state that it would not file a proposed rule change in 
connection with the operation of the machine-learning model's weekly 
retraining and the results of that process.
---------------------------------------------------------------------------

    \117\ 15 U.S.C. 78s(b)(1).
    \118\ 17 CFR 240.19b-4.
    \119\ See Section III.A.1., supra.
    \120\ See supra note 30 and accompanying text.
---------------------------------------------------------------------------

    Based on the foregoing, the Commission believes that Nasdaq has 
adequately responded to the commenter's concern. Nasdaq will need to 
file a proposed rule to make any changes, additions, or deletions to 
the operation of Dynamic M-ELO as approved herein. Nasdaq has 
delineated when it would file a proposed rule change to alter the 
operation of Dynamic M-ELO, and when the machine-learning model would 
retrain and adjust the weighting it applies to the data elements 
without it filing a proposed rule change. Specifically, Nasdaq's 
proposed rule change and rule text reflect the 142 data elements 
Dynamic M-ELO will consider when determining the Holding Period for a 
security and the goals Nasdaq will consider when weighing those data 
elements (i.e., reducing mark-outs and increasing fill rates) but does 
not set forth the relative weighting of each those individual data 
elements. Though

[[Page 62863]]

the structure of the proposal does not disclose of the exact weighting 
for each of the 142 data elements, it does set forth the two goals 
Nasdaq will consider when weighing those data elements initially and 
during each weekly retraining, which provides information as to how 
those 142 factors will be used in determining the Holding Period for a 
security. Based on how the proposed rule sets forth the goals that will 
govern each retraining, the Commission believes that Nasdaq's 
delineation of when it would and would not file a proposed rule change 
to alter the operation of Dynamic M-ELO is consistent with Nasdaq's 
rule filing obligation. The Commission agrees that the weekly 
retraining to optimize the weighting of the 142 data elements 
considered by Dynamic M-ELO to best achieve those goals within the 
rule's parameters would not necessitate the filing of a proposed rule 
change with the Commission because those adjustments would be 
reasonably and fairly implied by the proposed rule. However, to the 
extent Nasdaq seeks to change, add to, or delete from the rule's 
construct in connection with the weekly retraining, it would first be 
required to file a proposed rule change with the Commission.

V. Solicitation of Comments on Amendment No. 2 to the Proposed Rule 
Change

    Interested persons are invited to submit written data, views, and 
arguments concerning whether Amendment No. 2 is consistent with the 
Act. Comments may be submitted by any of the following methods:

Electronic Comments

    <bullet> Use the Commission's internet comment form (https://www.sec.gov/rules/sro.shtml); or
    <bullet> Send an email to <a href="/cdn-cgi/l/email-protection#f684839a93db95999b9b93988285b6859395d8919980"><span class="__cf_email__" data-cfemail="3a484f565f17595557575f544e497a495f59145d554c">[email protected]</span></a>. Please include 
file number SR-NASDAQ-2022-079 on the subject line.

Paper Comments

    <bullet> Send paper comments in triplicate to Secretary, Securities 
and Exchange Commission, 100 F Street NE, Washington, DC 20549-1090.

All submissions should refer to file number SR-NASDAQ-2022-079. This 
file number should be included on the subject line if email is used. To 
help the Commission process and review your comments more efficiently, 
please use only one method. The Commission will post all comments on 
the Commission's internet website (https://www.sec.gov/rules/sro.shtml). Copies of the submission, all subsequent amendments, all 
written statements with respect to the proposed rule change that are 
filed with the Commission, and all written communications relating to 
the proposed rule change between the Commission and any person, other 
than those that may be withheld from the public in accordance with the 
provisions of 5 U.S.C. 552, will be available for website viewing and 
printing in the Commission's Public Reference Room, 100 F Street NE, 
Washington, DC 20549 on official business days between the hours of 
10:00 a.m. and 3:00 p.m. Copies of the filing also will be available 
for inspection and copying at the principal office of the Exchange. Do 
not include personal identifiable information in submissions; you 
should submit only information that you wish to make available 
publicly. We may redact in part or withhold entirely from publication 
submitted material that is obscene or subject to copyright protection. 
All submissions should refer to File Number SR-NASDAQ-2022-079, and 
should be submitted on or before October 4, 2023.

VI. Accelerated Approval of Proposed Rule Change, as Modified by 
Amendment No. 2

    The Commission finds good cause to approve the proposed rule change 
prior to the 30th day after the date of publication of Amendment No. 2 
in the Federal Register. Amendment No. 2 does not include any material 
changes to the operation of the proposed Dynamic M-ELO and its machine-
learning model. In Amendment No. 2, the Exchange: (1) adds the defined 
term ``proprietary assessment of market conditions'' to the proposed 
rule text, which consolidates certain details and explanations about 
how the machine-learning model would operate from prior versions into a 
single defined term; (2) revises the list of factors provided in 
Exhibit 3b to include expanded and ``simplified'' explanations of the 
terminology used therein; (3) adds a representation that the systems 
used to operate Dynamic M-ELO and machine-learning model are ``SCI 
Systems'' and thus subject to compliance with Regulation SCI; and (4) 
expands the legal analysis to address comments regarding unfair 
discrimination and the exercise of impermissible discretion by the 
Exchange.
    The Commission finds that Amendment No. 2 raises no novel 
regulatory issues that have not previously been subject to comment and 
is reasonably designed to prevent fraudulent and manipulative acts and 
practices, to promote just and equitable principles of trade, and, in 
general, to protect investors and the public interest, and not be 
unfairly discriminatory, or impose an unnecessary or inappropriate 
burden on competition. Amendment No. 2 does not alter the proposed 
operation or any material features of Dynamic M-ELO, which operation 
and features have been subject to two rounds of public comment. In 
response to public comment, the revisions to the proposal contained 
within Amendment No. 2 provide additional clarification and details 
regarding how Dynamic M-ELO and the machine-learning model will 
operate, as well as additional legal analysis to support the Exchange's 
position that the proposal is consistent with the Act. Accordingly, 
pursuant to Section 19(b)(2) of the Act,\121\ the Commission finds good 
cause to approve the proposed rule change on an accelerated basis prior 
to the 30th day after publication of notice of the filing of Amendment 
No. 2 in the Federal Register.
---------------------------------------------------------------------------

    \121\ 15 U.S.C. 78s(b)(2).
---------------------------------------------------------------------------

VII. Conclusion

    It is therefore ordered, pursuant to Section 19(b)(2) of the 
Act,\122\ that the proposed rule change (SR-NASDAQ-2022-079), as 
modified by Amendment No. 2, be, and it hereby is, approved on an 
accelerated basis.
---------------------------------------------------------------------------

    \122\ 15 U.S.C. 78s(b)(2).

    For the Commission, by the Division of Trading and Markets, 
pursuant to delegated authority.\123\
---------------------------------------------------------------------------

    \123\ 17 CFR 200.30-3(a)(12).
---------------------------------------------------------------------------

Sherry R. Haywood,
Assistant Secretary.
[FR Doc. 2023-19728 Filed 9-12-23; 8:45 am]
BILLING CODE 8011-01-P


</pre><script data-cfasync="false" src="/cdn-cgi/scripts/5c5dd728/cloudflare-static/email-decode.min.js"></script></body>
</html>