最適化問題としての
機械翻訳
渡辺太郎

Machine Translation
data
learner
黑山头口岸联检部门将原来要二至三天办完的出入境手续改为一天办
完。

model
decoder

The United Inspection Department
of Heishantou Port has shortened
the procedures for leaving andentering the territory from originally2 - 3 days to 1 day.

• A data-driven approach to MT• We learn parameters from data assuming a “model”
2Channel Model+ noiseencoderchannelYŷX= argmax P r(y|x)
yf = sourcee = target

P r(x|y)P r(y)= argmax
P r(x)y= argmax P r(x|y)P r(y)yê = argmax P r(f |e)P r(e)e• Employed in: ASR, OCR, MT...
3

decoderResearch Questions• How to model the process of translation?
• How to learn the parameters (given data + model)?
• How to decode (given model + parameters + input
sentence)?

4

Bilingual Data
1.The development of Shanghai's Pudong is in step with theestablishment of its legal system
2.Xinhua News Agency, Shanghai, February 10, by wire
(reporters Jinhu Xie and Chijian Zhang)
1.上海浦东开发与法制建设同步
3.In recent years Shanghai's Pudong has promulgated andimplemented 71 regulatory documents relating to areas
2.新华社上海二月十日电（记者谢金虎、张持坚）such as economics, trade, construction, planning, science and3.上海浦东近年来颁布实行了涉及经济、贸易、建设、 technology, culture and education, etc., ensuring the orderly
规划、科技、文教等领域的七十一件法规性文件，确保 advancement of Pudong's development.
4.Pudong's development and opening up is a century了浦东开发的有序进行。
spanning undertaking for vigorously promoting Shanghai andconstructing a modern economic, trade, and financial center.
4.浦东开发开放是一项振兴上海，建设现代化经济、贸 Because of this, new situations and new questions that have
易、金融中心的跨世纪工程，因此大量出现的是以前不 not been encountered before are emerging in great
numbers.
曾遇到过的新情况、新问题。
5.In response to this, Pudong is not simply adopting an
5.对此，浦东不是简单的采取“干一段时间，等积累了经 approach of "work for a short time and then draw up laws
and regulations only after waiting until experience has been
验以后再制定法规条例”的做法，而是借鉴发达国家和深 accumulated." Instead, Pudong is taking advantage of thelessons from experience of developed countries and special
圳等特区的经验教训，聘请国内外有关专家学者，积
regions such as Shenzhen by hiring appropriate domestic
极、及时地制定和推出法规性文件，使这些经济活动一 and foreign specialists and scholars, by actively and promptly
formulating and issuing regulatory documents, and by出现就被纳入法制轨道。
ensuring that these economic activities are incorporated
6.去年初浦东新区诞生的中国第一家医疗机构药品采购 into the sphere of influence of the legal system as soon as
they appear.
服务中心，正因为一开始就比较规范，运转至今，成交 6.Precisely because as soon as it opened it was relatively
standardized, China's first drug purchase service center for药品一亿多元，没有发现一例回扣。
medical treatment institutions, which came into being at the
beginning of last year in the Pudong new region, in operating
up to now, has concluded transactions for drugs of over 100
million yuan and hasn't had one case of kickback.

(part of LDC2007T02, English translation of Chinese treebank)5

Alignment + Extraction
?B

b
m
#

"mb?

; M;
M
M

Q
B
H2 mBivm ?H Dmt
?
b

?2H/

iHF
rBi?
a?`QM• Annotate word alignment• Exhaustively extract phrases
X

6(Koehn et al., 2003)

Features• Collect all the phrase pairs from the data,
and ML estimates + other features

不 熟悉 ||| 'm not familiar ||| -1.4859937213 -7.2301988107 -0.3036824138 -3.0311892056
不 熟悉 ||| do n't know ||| -1.2064088591 -5.3571402084 -3.4402617349 -6.8870595804
不 熟悉 ||| i 'm not familiar ||| -2.522085653 -9.1804032749 -1.06784063 -3.0311892056
不 熟悉 ||| it will be great ||| -2.522085653 -20.871716142 0.0 -11.4593095552
不 熟悉 ||| not accustomed ||| -2.522085653 -5.5628513514 -0.6931471806 -2.2177906617
不 熟悉 ||| not accustomed to ||| -2.522085653 -8.5631752395 0.0 -2.2177906617
不 熟悉 ||| not familiar ||| -1.8754584881 -3.4150084505 -0.4212134651 -2.4210642434
7

Decoding--●---

Sharonh(&YWL, FYWLM
held)--●●●-

talks--●●●●

w------ +h(LIPHEXEPO, NY\MRKPILYMXER) Sharon+h([MXL, ]Y)Bush●-●●●●

held
a
talk
with
Bush
●----●--●●●●●-●●●
andbushi yu shalong juxing le huitan(Koehn et al., 2003)
Node: bit-vector representing covered source words•• Edge: phrasal translations, strictly left-to-right + score• DP-based Search space: O(2 ),Time: O(2 n )
n

8

n 2

Non-local featuresp(LIPH|7LEVSR)

p(7LEVSR| W ) --●---:Sharon

Sharon------:
Bushp(&YWL| W )--●●●-:heldheldp(LIPH|&YWL)●-----:Bushheldheld a talk●--●●-:held●--●●●:talkp(LIPH|&YWL)p(E|LIPH)p(XEPO|E)• Features that requires scoring out of phrases: bigram
language model• We usually employ 5-gram LM (4th order Markov)• Space: O(2 V ), Time: O(2 V n ) for m-gram LMnm-1

n m-1 29

synchronous-CFG
X X1
s

Xs 1

Xs 2

Xs 3

Xを⅏ Xs 4

X⃍嫛

X₝ 㼨爨

Xs 2
Xs 5

Xs 3

X"mb? X?2H/ Xs 5

Xs 4

X iHF XrBi? a?`QM

Xℕ ↩店(Chiang, 2007)• paired-CFG: two RHS, but single LHS• Reordering represented by swapped non-terminals
10synchronous-TSG
XS
X
XAXAS
XX
XL
XLS"
S"
XX
mb?B
XXX#X#mb?B

XX

XXxX1X1x
XXxX2X2
x

Xo
XoXS
XS

XXxX3X3x
XXxX5X5
x

X"mb?
mb?
XX
XXX"

XS
XS
XSXS

Xo
XoS"
S"
XX

XS
XS
XX XL
XLS"
S"
XX

XXaXa XL
Xo
XoXaXa X
XLS"
S"
XX

mtBM;
XX XDXDmtBM;
mBiM
XX X
XXvm
v
Xm
X XbXb?HQM;
?HQM;XX XXXXXHXH2X2X XXX?X?mBiM
XX XX
XX
X
XoXa

XX
XrBi?
Bi?
2H/
XX  XxXxX6X6 XXXr
XXX?X?2H/

X X7
x

XiXiHF
XX X
XaXa?HQM;
?`QM
X
HF
XX XoS"
X
X
X
X (GalleyX et al., 2004)
X
XXa X Xx1 ,LS"X

• Bilingual version of Tree XDmtBM;
XH2X
X

XX
X?2H/
X  XxX1
11 X X X

Substitution Grammar X

X X44
x

Evaluation: ngram precision
Well , I 'd like to stay five nights beginning
October twenty-fifth to thirty .

• I 'd like to stay there for five nights , from October twenty
fifth to the thirtieth .

• I want to stay for five nights , from October twenty fifth tothe thirtieth .

• I 'd like to stay for five nights , from October twenty fifth tothe thirtieth .

• I would like to reserve a room for five nights , from October
twenty fifth to the thirtieth .
12Evaluation: ngram precision
Well , I 'd like to stay five nights beginning
October twenty-fifth to thirty .
11
p1 =
15

• I 'd like to stay there for five nights , from October twenty
fifth to the thirtieth .

• I want to stay for five nights , from October twenty fifth tothe thirtieth .

• I 'd like to stay for five nights , from October twenty fifth tothe thirtieth .

• I would like to reserve a room for five nights , from October
twenty fifth to the thirtieth .

Evaluation: ngram precision
Well , I 'd like to stay five nights beginning
October twenty-fifth to thirty .
11
p1 =
15

5
p2 =
14

• I 'd like to stay there for five nights , from October twenty
fifth to the thirtieth .

• I want to stay for five nights , from October twenty fifth tothe thirtieth .

• I 'd like to stay for five nights , from October twenty fifth tothe thirtieth .

• I would like to reserve a room for five nights , from October
twenty fifth to the thirtieth .

Evaluation: ngram precision
Well , I 'd like to stay five nights beginning
October twenty-fifth to thirty .
11
p1 =
15

5
p2 =
14

3
p3 =
13

• I 'd like to stay there for five nights , from October twenty
fifth to the thirtieth .

• I want to stay for five nights , from October twenty fifth tothe thirtieth .

• I 'd like to stay for five nights , from October twenty fifth tothe thirtieth .

• I would like to reserve a room for five nights , from October
twenty fifth to the thirtieth .

Evaluation: ngram precision
Well , I 'd like to stay five nights beginning
October twenty-fifth to thirty .
11
p1 =
15

5
p2 =
14

3
p3 =
13

2
p4 =
12

• I 'd like to stay there for five nights , from October twenty
fifth to the thirtieth .

• I want to stay for five nights , from October twenty fifth tothe thirtieth .

• I 'd like to stay for five nights , from October twenty fifth tothe thirtieth .

• I would like to reserve a room for five nights , from October
twenty fifth to the thirtieth .

Evaluation: BLEU
• (Uniformly) weighted combination of precision(Papineni et al., 2002)• brevity penalty: penalize too short sentences• r = reference length, c = candidate length• If we have multiple “r”, choose the closest-shortest
reference to “c”• Both factors are computed over the whole document
17

Why BLEU?
• Used as a standard metric for more than 10 years:
Progress of MT is due by BLEU!

• Good indication of you “progress” and probably notfor comparing systems with different philosophy• However, non-linear decomposition into sentences:
corpus-wise metric, thus, harder to optimize• BP-problem(Chiang et al., 2009):You can generatespuriously long translations together with a highly
confident short translations
18

A Bad Example“we come from the land of the ice and snow”“from the midnight sun where the hot springs flow”
system 1xxx xxx xxx xxx land xxx xxx ice xxx snow
xxx xxx midnight xxx xxx xxx hot xxx flow
system 2
x come x x land x x ice x snow x x x x x
from xxx sun xxx• Both shared the same # of words, and thesame # of matches

19

MT Research @• Better model, search, and optimization• Sophisticated models using “syntax” (or, better“parsing” with complex models)• Unsupervised learning with structured hidden
variables (via Bayesian models)• Better optimization (via discriminative training)• Today’s focus: Optimization20

Nightmares of
Optimization

Optimization• How to learn w?
• Assume a loss function ℓ𝓁  and minimize the “risk”

over bilingual data, F and E (usually small... WHY?)! !

• Since true distribution is unknown, minimize aregularized empirical risk

• NOTE: ℓ𝓁  ≠  error (i.e. 1 - BLEU)
22

Nightmares• Problem 1: Many alternative translations (e) possible
with many alternative derivations (d)• We cannot enumerate all possible hidden variables• Problem 2: ℓ𝓁  and/or error are corpus-wise, notsentence-wise (i.e. BLEU)• We cannot assume a convex function, or gradientsare available• Problem 3: Enormous search space and high risk ofsearch errors

23

Learning Strategies

(k-best) Batch Learning• Approximate the document-wise candidate space by kbest merging using decoding step + optimization step
(Och and Ney, 2002)25

k-bestthis is the kyoto kanko hotel , front desk . ||| -48.68790464
this is the kyoto kanko hotel , front desk . ||| -48.85902546
this is the kyoto kanko hotel , front desk . ||| -49.90369084
this is the kyoto kanko hotel , front desk . ||| -50.07481166
this is the kyoto kanko hotel , front desk . ||| -50.32856858
kyoto kanko hotel , front desk . ||| -51.13501382
kyoto kanko hotel , front desk . ||| -51.30613464
this is the kyoto kanko hotel , front desk . ||| -51.54435478
kyoto kanko hotel , front desk . ||| -52.35080002
kyoto kanko hotel , front desk . ||| -52.52192084
kyoto kanko hotel , front desk . ||| -52.71186262
kyoto kanko hotel , front desk . ||| -52.77567776
kyoto kanko hotel , front desk . ||| -52.88298344
hello , this is the kyoto kanko hotel . front desk . may i help you as soon as possible . ||| -53.77178844
hello , this is the kyoto kanko hotel , front desk . may i help you as soon as possible . ||| -53.90754257
hello , this is the kyoto kanko hotel . front desk . may i help you as soon as possible . ||| -53.92571267
kyoto kanko hotel , front desk . ||| -53.92764882
kyoto kanko hotel , front desk . ||| -53.99146396
hello , this is the kyoto kanko hotel , front desk . may i help you as soon as possible . ||| -54.06146681
kyoto kanko hotel , front desk . ||| -54.09876964
kyoto kanko hotel , front desk . ||| -54.35252656
hello , this is the kyoto kanko hotel . front desk . may i help you as soon as possible . ||| -54.98757464

Batch Candidate Space• Perform optimization over the merged k-bests• Eventually converge, but merely anapproximation of all the translation candidates
27

Objectives• Minimum Error Rate (MERT) (Och, 2003)• Log linear (softmax) (Och and Ney, 2002; Blunsom etal., 2008)• Pair-wise Rank Optimization (PRO) (Hopkins andMay, 2011)• Expected BLEU (xBLEU) (Pauls et al., 2009; Rosti etal., 2010; Rosti et al., 2011)
28

MERT

• Due to non-linear error function, non-convex andimpossible to compute derivatives• Gradient-free optimization: Powell’s method or
Downhill-simplex method• An efficient line search by exploiting piece-wise linear
function of k-best list (Och, 2002)29

softmax• A set of oracle candidates (o) are softly separated(Blunsom et al., 2008), thus not strictly-convex• Due to non-linear error function, oracles arecomputed from k-bests by hill-climbing (Watanabe,
2012)

PRO

• We suffer hinge-loss through exhaustive pair-wisecomparisons• Sample small # of pairs (Hopkins and May, 2011) or
sample from a set of good translations and a set ofbad translations (Watanabe, 2012; Semianer et al.,
2012)
31

xBLEU• Minimize BLEU risk
• by the expected ngram count (Pauls et al., 2009;
Rosti et al., 2010; Rosti et al., 2011)• not by the expected sentence BLEU (Li andEisner, 2009)

32

BP?

1.2

(ex - 1) /(1 + e1000x) + 1
ex

1

y

0.8

x
e
−1
x
min {e , 1} ≈+1
1 + e1000x

0.6
0.4
0.2
0
-10

-5

0
x

5

10

• They tried many alternatives by matlab (Rosti etal., 2010; Rosti et al., 2011)• Ignore BP(Tromble et al., 2008)• Ignore min(Pauls et al., 2009)33

Online Learning

(k-best) Online Learning• Introduce online approximation for objectives by
optimizating over a “batch” with decoding step +35
optimization stepOptimize Updates• Perceptron for MT (Liang et al., 2006)• MIRA (or PA-1) for MT (Watanabe et al., 2007;
Chiang et al., 2008)• SGD + projection for MT(Watanabe, 2012;
Semianer et al., 2012)

36

MIRA

• Large margin principle: a good translation is separated froma bad translation by a margin• Minimize the progress by regularizer (Crammer et al., 2006)• Easily overfit: stop iterations, averaging etc. (Watanabe etal., 2007; Chiang et al., 2008)

37

SGD + Projection• Solve a “batch local” objective in each update• Set learning rate (η) + update by a sub-gradient(L2 and Δℓ𝓁) + projection into a L2-ball (ShalevShwartz et al., 2007) 38

Intricacy of BLEU
• Optimization for a sentence-wise BLEU
≠optimal for a document-wise BLEU

• BLEU on a larger batch: better document-wise
BLEU estimates• However, requiring more iterations• Previous work: Pseudo-document, Decayed

BLEU (Watanabe et al., 2007, Chiang et al., 2008)39

Optimized Update• 2-step update: suffer sub-gradient from L+ solve a QP (Watanabe, 2012)• Similar to MIRA: global L + directly use
2the learning rate as a hyperparameter
40

2

Experiments(Watanabe, 2012)• NIST Chinese-to-English translation task
• Tune on MT02, development testing on
MT06, testing on MT0841

Learning Curves
40
35
30

BLEU

25
20
15
MIRA-L MT02
MT08
ORO-L MT02
MT08
O-ORO-L MT02
MT08

10
5
0
0

5

10

15
iteration
42

20

25

30

(Watanabe, 2012)Experiments: Batch Size

(Watanabe, 2012)43

Conclusion

Summary• Some concepts from MT• Optimization for MT• Intricacy of an evaluation metric• k-best batch approximation for candidate space• Online approximation of objectives
45

Outlook• Feature selection (Simianer et al., 2012)• Larger data for optimization (Xiao et al., 2011)• Bayesian models• Phrasal model (DeNero et al., 2008; Neubig et al.,2011)

• Syntactic models (Blunsom et al., 2009; Cohn andBlunsom, 2009; Levenberg et al., 2012)• Deep learning (Le et al., 2012)46

6IJIVIRGIW&PYRWSQ 4 'SLR 8 (]IV '  3WFSVRI 1  %YKYWX  % KMFFWWEQTPIVJSVTLVEWEPW]RGLVSRSYWKVEQQEVMRHYGXMSR -R 4VSG SJEGPMNGRPT TT   7YRXIG 7MRKETSVI %WWSGMEXMSRJSV'SQTYXEXMSREP0MRKYMWXMGW 6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#XQ`;fMi?QHQ;vfSf
SyNfSyN@Ry33&PYRWSQ 4 'SLR 8 3WFSVRI 1  .YRI  % HMWGVMQMREXMZIPEXIRXZEVMEFPIQSHIPJSVWXEXMWXMGEPQEGLMRI XVERWPEXMSR -R 4VSG SJ EGP ,PX TT   'SPYQFYW 3LMS 6IXVMIZIH JVSQ?iiT,ffrrrX+Hr2#XQ`;fMi?QHQ;vfSfSy3fSy3@Ryk9'LMERK (  .YRI  ,MIVEVGLMGEPTLVEWIFEWIHXVERWPEXMSR 'SQTYXEXMSREP0MRKYMWXMGW     6IXVMIZIHJVSQ ?iiT,ff/tX/QBXQ`;fRyXRRekf+QHBXkyydXjjXkXkyR HSM GSPM

'LMERK ( (I2IIJI 7 'LER = 7  2K , 8  3GXSFIV  (IGSQTSWEFMPMX]SJXVERWPEXMSRQIXVMGWJSVMQTVSZIHIZEPYEXMSRERHIJ½GMIRXEPKSVMXLQW -R 4VSG SJ IQRPT  TT  ,SRSPYPY ,E[EMM 6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#XQ`;fMi?QHQ;vf.y3@Rye9'LMERK ( 1EVXSR =  6IWRMO 4  3GXSFIV  3RPMRIPEVKIQEVKMRXVEMRMRKSJW]RXEGXMGERH
WXVYGXYVEPXVERWPEXMSRJIEXYVIW -R 4VSGSJIQRPT TT   ,SRSPYPY ,E[EMM 6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#XQ`;fMi?QHQ;vf.y3@Ryk9'SLR 8 &PYRWSQ 4  %YKYWX  % &E]IWMERQSHIPSJW]RXE\HMVIGXIHXVIIXSWXVMRKKVEQQEV
MRHYGXMSR -R 4VSG SJIQRPT TT   7MRKETSVI 6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#
XQ`;fMi?QHQ;vf.f.yNf.yN@Ryjd'VEQQIV / (IOIP 3 /IWLIX . 7LEPIZ7L[EVX^ 7  7MRKIV =  1EVGL  3RPMRITEWWMZIEKKVIWWMZIEPKSVMXLQW .SYVREPSJ1EGLMRI0IEVRMRK6IWIEVGL  
(I2IVS . &SYGLEVH'|Xq % /PIMR (  3GXSFIV  7EQTPMRKEPMKRQIRXWXVYGXYVIYRHIVE&E]IWMERXVERWPEXMSRQSHIP -R 4VSG SJIQRPT TT   ,SRSPYPY ,E[EMM %WWSGMEXMSR
JSV'SQTYXEXMSREP0MRKYMWXMGW 6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#XQ`;fMi?QHQ;vf.y3@Ryjj+EPPI] 1 ,STOMRW 1 /RMKLX /  1EVGY (  1E]   1E]   ;LEX W MR E XVERWPEXMSR
VYPI# -R( 1 7YWER (YQEMW 7 6SYOSW )HW  4VSG SJLPXREEGP TT   &SWXSR1EWWEGLYWIXXW 97%%WWSGMEXMSRJSV'SQTYXEXMSREP0MRKYMWXMGW,STOMRW 1  1E] .  .YP]  8YRMRK EW VEROMRK -R 4VSG SJ IQRPT  TT  )HMRFYVKL 7GSXPERH 9/ %WWSGMEXMSR JSV 'SQTYXEXMSREP 0MRKYMWXMGW 6IXVMIZIH JVSQ ?iiT,ff
rrrX+Hr2#XQ`;fMi?QHQ;vf.RR@RRk8/SILR 4 3GL * .  1EVGY (  1E].YRI  7XEXMWXMGEPTLEVWIFEWIHXVERWPEXMSR -R 4VSG SJ,082%%'0  TT   )HQSRXSR
0I ,7 %PPEY^IR % =ZSR *  .YRI  'SRXMRYSYWWTEGIXVERWPEXMSRQSHIPW[MXLRIYVEPRIX[SVOW -R 4VSGIIHMRKWSJXLIGSRJIVIRGISJXLIRSVXLEQIVMGERGLETXIVSJXLIEWWSGMEXMSRJSV
GSQTYXEXMSREPPMRKYMWXMGW ,YQERPERKYEKIXIGLRSPSKMIW TT   1SRXVqEP 'EREHE %WWSGMEXMSR
JSV'SQTYXEXMSREP0MRKYMWXMGW 6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#XQ`;fMi?QHQ;vfLRk@Ryy8
0IZIRFIVK % (]IV '  &PYRWSQ 4  .YP]  % FE]IWMERQSHIPJSVPIEVRMRKWGJKW[MXLHMW
GSRXMKYSYWVYPIW -R 4VSGIIHMRKWSJXLINSMRXGSRJIVIRGISRIQTMVMGEPQIXLSHWMRREXYVEPPERKYEKI
TVSGIWWMRKERHGSQTYXEXMSREPREXYVEPPERKYEKIPIEVRMRK TT   .INY-WPERH /SVIE %WWSGMEXMSR
JSV'SQTYXEXMSREP0MRKYMWXMGW 6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#XQ`;fMi?QHQ;vf.Rk@RykR
0M >  )MWRIV .  %YKYWX  *MVWXERHWIGSRHSVHIVI\TIGXEXMSRWIQMVMRKW[MXLETTPMGEXMSRWXSQMRMQYQVMWOXVEMRMRKSRXVERWPEXMSRJSVIWXW -R 4VSG SJ IQRPT  TT   7MRKETSVI
6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#XQ`;fMi?QHQ;vf.f.yNf.yN@Ryy8
2IYFMK + ;EXEREFI 8 7YQMXE ) 1SVM 7 /E[ELEVE 8  .YRI  %RYRWYTIVZMWIHQSHIPJSVNSMRXTLVEWIEPMKRQIRXERHI\XVEGXMSR -R 4VSG SJEGPLPX TT   4SVXPERH 3VIKSR 97%%WWSGMEXMSRJSV'SQTYXEXMSREP0MRKYMWXMGW 6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#XQ`;f
Mi?QHQ;vfSRR@Rye9
3GL * .  .YP]  1MRMQYQIVVSVVEXIXVEMRMRKMRWXEXMWXMGEPQEGLMRIXVERWPEXMSR -R 4VSGIIHMRKWSJXLIWXERRYEPQIIXMRKSJXLIEWWSGMEXMSRJSVGSQTYXEXMSREPPMRKYMWXMGW TT   7ETTSVS .ETER %WWSGMEXMSRJSV'SQTYXEXMSREP0MRKYMWXMGW 6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#XQ`;fMi?QHQ;vf
Syj@RykR HSM 
3GL * . 2I] ,  .YP]  (MWGVMQMREXMZIXVEMRMRKERHQE\MQYQIRXVST]QSHIPWJSVWXEXMWXMGEPQEGLMRIXVERWPEXMSR -R 4VSGSJEGP TT   4LMPEHIPTLME 4IRRW]PZERME 97% 6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#XQ`;fMi?QHQ;vfSyk@Ryj3 HSM 

4ETMRIRM / 6SYOSW 7 ;EVH 8 >LY ;.  .YP]  &PIY EQIXLSHJSVEYXSQEXMGIZEPYEXMSRSJQEGLMRIXVERWPEXMSR -R 4VSGSJEGP TT   4LMPEHIPTLME 4IRRW]PZERME 97%%WWSGMEXMSR
JSV'SQTYXEXMSREP0MRKYMWXMGW 6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#XQ`;fMi?QHQ;vfSyk@Ry9y
HSM 
4EYPW % (IRIVS .  /PIMR (  %YKYWX  'SRWIRWYW XVEMRMRK JSV GSRWIRWYW HIGSHMRK MR
QEGLMRIXVERWPEXMSR -R 4VSG SJIQRPT TT   7MRKETSVI 6IXVMIZIHJVSQ ?iiT,ff
rrrX+Hr2#XQ`;fMi?QHQ;vf.f.yNf.yN@RR9d
6SWXM %: >LERK & 1EXWSYOEW 7 7GL[EVX^ 6  .YP]  &FRW]WXIQHIWGVMTXMSRJSV[QXW]WXIQGSQFMREXMSRXEWO -R 4VSGIIHMRKWSJXLINSMRX½JXL[SVOWLSTSRWXEXMWXMGEPQEGLMRIXVERWPEXMSRERH
QIXVMGWQEXV TT   9TTWEPE 7[IHIR %WWSGMEXMSRJSV'SQTYXEXMSREP0MRKYMWXMGW 6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#XQ`;fMi?QHQ;vfqRy@Rd93
6SWXM %: >LERK & 1EXWSYOEW 7 7GL[EVX^ 6  .YP]  )\TIGXIHFPIYXVEMRMRKJSVKVETLW&FRW]WXIQHIWGVMTXMSRJSV[QXW]WXIQGSQFMREXMSRXEWO -R 4VSGIIHMRKWSJXLIWM\XL[SVOWLSTSR
WXEXMWXMGEPQEGLMRIXVERWPEXMSR TT   )HMRFYVKL 7GSXPERH %WWSGMEXMSRJSV'SQTYXEXMSREP
0MRKYMWXMGW 6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#XQ`;fMi?QHQ;vfqRR@kRRN
7LEPIZ7L[EVX^ 7 7MRKIV = 7VIFVS 2   4IKEWSW 4VMQEPIWXMQEXIHWYFKVEHMIRXWSPZIVJSVWZQ -R 4VSGSJMGQP  TT   'SVZEPMW 3VIKSR 6IXVMIZIHJVSQ ?iiT,ff/QBX+KXQ`;f
RyXRR98fRkdj9NeXRkdj8N3 HSM LXXTHSMEGQSVK
7MQMERIV 4 6MI^PIV 7 (]IV '  .YP]  .SMRXJIEXYVIWIPIGXMSRMRHMWXVMFYXIHWXSGLEWXMGPIEVRMRKJSVPEVKIWGEPIHMWGVMQMREXMZIXVEMRMRKMRWQX -R 4VSGIIHMRKWSJXLIXLERRYEPQIIXMRKSJXLIEWWSGMEXMSR
JSVGSQTYXEXMSREPPMRKYMWXMGW ZSPYQI 0SRKTETIVW TT   .INY-WPERH /SVIE %WWSGMEXMSRJSV'SQTYXEXMSREP0MRKYMWXMGW 6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#XQ`;fMi?QHQ;vfSRk@Ryyk
8VSQFPI 6 /YQEV 7 3GL * 1EGLIVI] ;  3GXSFIV  0EXXMGI1MRMQYQ&E]IW6MWOHIGSHMRKJSVWXEXMWXMGEPQEGLMRIXVERWPEXMSR -R 4VSGSJIQRPT TT   ,SRSPYPY ,E[EMM 6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#XQ`;fMi?QHQ;vf.y3@Rye8;EXEREFI 8  .YRI  3TXMQM^IHSRPMRIVEROPIEVRMRKJSVQEGLMRIXVERWPEXMSR -R 4VSGIIHMRKWSJXLIGSRJIVIRGISJXLIRSVXLEQIVMGERGLETXIVSJXLIEWWSGMEXMSRJSVGSQTYXEXMSREPPMRKYMWXMGW ,YQER
PERKYEKIXIGLRSPSKMIW TT   1SRXVqEP 'EREHE %WWSGMEXMSRJSV'SQTYXEXMSREP0MRKYMWXMGW6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#XQ`;fMi?QHQ;vfLRk@Ryke;EXEREFI 8 7Y^YOM . 8WYOEHE ,  -WS^EOM ,  .YRI  3RPMRI 0EVKI1EVKMR8VEMRMRK JSV
7XEXMWXMGEP1EGLMRI8VERWPEXMSR -R 4VSGSJIQRPTGSRPP TT   4VEKYI '^IGL6ITYFPMG<MES < 0MY = 0MY 5 0MR 7  .YP]  *EWXKIRIVEXMSRSJXVERWPEXMSRJSVIWXJSVPEVKIWGEPIWQX
HMWGVMQMREXMZIXVEMRMRK -R 4VSGIIHMRKWSJXLIGSRJIVIRGISRIQTMVMGEPQIXLSHWMRREXYVEPPERKYEKI
TVSGIWWMRK TT   )HMRFYVKL 7GSXPERH 9/ %WWSGMEXMSR JSV 'SQTYXEXMSREP 0MRKYMWXMGW6IXVMIZIHJVSQ ?iiT,ffrrrX+Hr2#XQ`;fMi?QHQ;vf.RR@Ry3R