{¡ ¢£¤)¥7 ¦§.¨©ª"¢¬«:®­:¯_­: "¢¬«:¯7¢°¯_­K­:¯7{¤§¬¨±¢p«P³²{´¡_¤§¬¨Qµ¦ª¶7·,¸³¹ º »=¼

    !"# $%'& )(  *,+.-0/213547683:983;=<?>A@B<?CD<FEG;IHKJL;AM8>BCDNPOQOAJRNPS8@
T,U.VQW 3P9X?476'Y +Z-K6 U.V\[ -K6?]_^`+ZaK+ U<8ObN5c8CEG;IHKJL;AM8>BCDNPOQOAJRNPS8@
deV\f=g 1%4 VQh
i ;Ij=k:EG;AHbJl;IM8>BCDNPOQO=JlNPS?@
T,U.VQWnm [ +po V\[<8qsrtIkPEG;IHKJL;AM8>BCDNPOQOAJRNPS8@uv w=xy{z{|}v ~Qz#sK|}wP5z_v y5 lv~: vbAsKy_~PvlD~Plbv ylzK5z_:xKxPP bQ5
½¿¾ÀbÁ5ÂPÃÄAÁUv y_v®Ósv®ÉKv~Pv yx7z_v°Ç5lbv yv®{v z_FKA|}5Ç5vL Ë È:{L~PÉ"|Ðx~\ÇPRÏIv y_v ~Qz#xLÉKby{lz{P|ÐÍ?Å©v°:{v°5PwPwAKy{)z Vv´z_Ky#ÎÔxbPl~:v
ÅÆv®wPy_v{v ~Qzx|}v z_P5ÇÈpbyF K~:z{y_: z{L~PÉv~:v|GÊéöW VÎ±_í =xy{z{là= LxKB~Pv PyxB~:v z_%é.í=|Gv|}Ky_bÊ Ë xbv´ÇË lv´,py{b| L Ë y_xKy{LvÈz_PK={xK~:ÇP,%|}5Ç5v ÌÍ
LvxKy{~:l~PÉÔé*X"ßí=ÇPv Ì{lb~z{y_v v%é.uZYí Ë xÉbÉKvÇÇ5v´ ÌLK~Î 5Ç5v l Ë yxy_lv´xy_vÈÉbv ~Pvy_xz{vÇ=L~PÉÇ5lÏIv y_v ~Qz_zy{vvÑéZò [Ê£Zu Yí Ë \bz{vÇÇ5v´ ÌLK~êz{y_v v´Ñé.ò°- YÊ£8u Yí lv´xy_~Pl~:É±xLÉKby{lz{P|ÐÈxK~:ÇÑw:xKy_xK|}v z{vy%{v zz_l~:ÉbÍxK~:Ç Ë \bz{v´ÇÈz_P|}w:®é.ò°-YÊ-Y°ÎÔûíÍ7ÒPby8v´xK%xKlÉbKy_Rz_P|ÒPKy_Ó®xyÇez{vw\Ó°L{v{v Lv z{LK~ÔÌ=v´Çz{exKÇPÇz_ÓvÐz_y_xKl~Ñ|}5Ç5v ÌÈ=L~PÉÔ|Ðx~\ÞÇ5lÏAvy{v~bzGw:xyx|}v z{v yÈ{v z{Ê
z{Pv v~:v| Ë Lvez{Pv |}5Ç5vL}z_:x7z)|ÐxÕ\L|}lÖvRz
z_l~:ÉbÍêÒPby)v ÕPxK|Gw:lvbÓvz{yxL~ ´ëP - VÎÔ Ë n7xy_QL~PÉ
w=vypby{|Ðx~= vKÍÔ×Ø~:{v | Ë lv{v Lv z{LK~ÙxKlL7Ó"v ~PÊ
z_Pv|Ðxy_ÉKL~w:xKy_xK|Gv z{v]y \Gsz{:veúbv y_~Pv ösxK~:Çz{PvúKvy{~:v v| Ë LvFz_ Ë vKw5z_l|}LÖ vÇz_Èw=vypby{|ÐxK~: v®|}v z{Êw=xyx|}v z_v yßé.vKÍ É:Í7xy_\l~:ÉÉQx|}|ÐxÓ°lz{ Üò®ÒÑúbv y_~Pv ÌÍ íy{ÌÚ=ÛxbÑxK  PyxK b) y_b_©v ~Qz{y_Kw\KÐ|}v´x~
wPy{v´ ÌLK~?KyÜÝ"Ûy{v´xPÍÞ×Õ5w=vy{L|}v ~Qz_Ó°lz{Å©vz{yxL~±x Ë K5zÈëìbìKì}|}5Ç5v ÌpKyßvxbÔwPy_ Ë Lv |Í"\K|}v
vKv ~z_vzßw:y{ Ë lv|Ðx~=Çz{v~±|}v z_y{Ì Ç5v |}K~PÊ
|}5Ç5vL®:x0bvv ÕP v LLv ~Qz°w=vypby{|Ðx~= vK\0v ^b=xIz{}by Ë v zz{vyz_y_xz{vßz{Pv Ë v ~:v àPzKv~:v| Ë Lv"vlv´z_lb~DÍ
z_:x~}z_Pv Ë vzØ|}5Ç5v Ì#y_v wAKy{z{v´ÇL~}z{:vllz{vy_xz{Py_vKÍ#Ýz{:v y|}5Ç5vLDP7Óv bv y´A=x0Kv%|}v´Ç5l5 y{v}Ky³vKv ~Æw=\byßwAv y{pKy{Ê|ÐxK~: vbÍFÜx7z{:v yz{:xK~, K| Ë l~PvÉK\5Ç)x~=Ç Ë xbÇ)|}5Ç5v Ìl~
á#âßãQä Á\Â:å®æçÄAÁPèå ä
xK~Ùv ~=v| Ë LvKÓv:{v)pby{Ó®xyÇÑz{v w\Ó°Ìvevlv´z_lb~Þpy{b|~nv~:{v | Ë lveL}x© blLv z{LK~K|}5Ç5v ÌGÓ°Pb{vewPy{v´Ç5ÌÊ z_Ó°P:v}v ~Úl Ë x0yKxvy_y_ xKÉKvÇ©|Gz_5KÇ5Ébvv Lz{ß:z_v eyÐà:\~:lvÇÞLÇnxv ÕP  Ë v Llv vz³~bzÐKswA|}v y{5pKÇ5y_v |ÐÌßxz{~:= x7vKz Íz_lb~:Gxy_v b| Ë L~PvÇ Ë ÞÓv LÉKQz{v´ÇÙx0Kvy_xKÉKL~PÉKyKz_l~:É:Í Y®:v Ë xK{Ì°v ~:{v | Ë Lvvlv´z{LK~Ðw:y{5 v´Ç5Py_v°LØKvy{GL|}wPlDv _uLv zz_v y_LêéZëìbìKìQí"zx7z_v"z_:x7z©î{ï~Pv v_{xKy{©x~:ÇÑ{5ð}Ê
 lv~Qz b~:Ç5lz{LK~"pbyxK~Èv ~=v| Ë LvØK: ÌxK_là:v yBz_ Ë vs|}by{v bÍQz_xKyz°Ó°lz{z{:v"v |}w5z)v~:v| Ë LvKÍxb  Py_xz{vz{:xK~©xK~\ÔRz³l~:ÇPl\ÌÇ5:x|}v| Ë v y³L³lsz_Pv ëPÍ°ÇPÇz{ßz_Pv°v ~:{v | Ë Lvsz_Pv®|}\ÇPv PL~%z_Pv°L Ë yxy_³z{=x7z
 Lxb{{là:v y®xy_v³xb  Py_xz{v³x~:ÇeÇ5LKvy_{vKÍ ñ|}xÕ5l|}LÖ vz{Pvv ~:{v | Ë L%v ` wAv y{pKy_|}xK~: v³z_Ðz{:v%v y{ÊÎ±x~\|Gv z{P5ÇP:x0bv Ë vv ~ÆwPy{bw=Qv´Çez{,ÉKv ~:v yx7z{vxK  5Êyx7z_vKKv zÇ5LKv y{vKFv z_ÈK°|G5Ç5vLÍÔò®xÉbÉKL~PÉÙé.òy_v L|}xK~D :Í°y{Ü°bv ywA|}vxv zz{y_QL³z{vbw±~eëÈxGpbPyLlÌx LàPL| Õ5vË Çeép~\7PxL| ÌÇPË x7v z_ylb~=Fí®Rz_vv yzx7Í z{LK~=®Ky´óKóbôbí:z{yxL~:B|}5Ç5v ÌBK\b~PvFz\w=vépvKÍ É:ÍLsÇ5v L{lb~ßz{y_v víb~ Ë \zz_y_xKwÐ{xK|Gw:lv´Iz{:v°z{yxL~PL~PÉÈv zÍØÝwPlz{ÖÈéóKóbóbí =Í°PÜ°~Qv z{z_LP?y{x~nLIz{z{:Pvev"v|}~:5{v Ç5| vLË °lv,=x0pKy{vb| Ë vvz{P~v:~P{vv´ÇBz_Í vÇøv zÐv~5ÊË xÉbÆpv´x7z{:y{v´ÆL~:z{v´xKÇ`Kz_y_xKl~PL~PÉêw=bl~Qz_Íõò\bz{L~PÉ
v| Ë LvGz_:x7z):xb}|Ðx7Õ5l|P| wAv y{pKy_|Ðx~: v,b~nz_Pvéö5:xKwPly_vK?ëKìKì:´í°Ébv ~Pvy_xz{v´x,w=Kz{v~bz_LxKlL |}Ky_vGÇPlbv yvPlLÌ Ll| Ë é.7xLLÇPxz{LK~=í®{v z´Í{v z|G5Ç5vL,z_:x~ Ë xÉbÉKL~PÉ Ë ÚÓv LÉKQz{L~PÉÙz_Pv©z{yxL~5Ê
L~PÉ¿v zz{npKy v©~Pv Ó÷|G5Ç5vL xzz_v ~:Ç'z{øz{PQv©wAKL~bz
z_:x7z}xy_v)Ç5lðÐ Plzz_± ÌxK_lpÞ Ky_y_vz_lbÍÔ\PLlL7x~Ùv zxöÍ w:ÎÔy{\v´Ç5ÇPLv Ìz{%LKx~:y_³v)Ó°xKÇ:lz{Ç5ÆvÇ©z{:z_vÐ±|Gx~Ñ5Ç5vv~:L"vxK| ly_Ë vLxbv Ç5Ë Þl~©x0Kz_vPyv}xÉbv l~~::{Év | z_PË v lvbLy ÍéöëìbìKìbí Ë \bz?pv´x7z_Py{v´?L~:z{v´xKÇ"K5z{yxL~Pl~:Éw=bl~Qz ÍF×#y_y{byÊ Ky_y{v´z_l~PÉKÊöb5z{w:5zÊ£ 5Ç5v´Øé.×®°Ý"®í#éZuLv z{z{v y_ÌGùÚò®xú\Ly{ö Y®xKl:L7LÓ°®L~|ÐPÉÔxúbv v~=xKvÇP| ÇPË l~PLvÉ}xÓ°|GRz_Ù5Ç5v vÕPI z{v Llz{vP~bv³zGvw=~:vyvpb| y{Ë |ÐLv xK~:¥0 v a z{ .xbË z´v ´óKóQKíB y{v´x7z{v´?|}5Ç5vL8Ó°Rz_GÇPv by{y_v Ìx7z_vÇ³vy{y_Ky Ë ³z{yxL~5Ê
L~PÉø|}\ÇPv Ìpby|%Plz{lÊ Lxb{,wPy_ Ë Lv |ÐeK~Ç5lÏAvy{v~bz Ç5lÊ pÎÔbPK~:y_v Ç 7Kl~vÛyF|}z_Pl~\vÔ5z_{vv ÞLvpz_y_lKb| ~nw:L y{Ë 5y_ xKv´y{Ç5LvP©y_veÓ°xKlz{lL 7ÓëGìbìK:ì'Gz{|}Þ5bÇ5w5vz{LlÊ Í
:z{b|}lv´ ÍÙÎ :~Py{Þx~=Çûxy_|Ðx~Qz{néóbóKôQí% y{v´x7z{v´ÇÙÇ5lÊ |}LÖ vz_PvØv~:{v | Ë lvz_x~\ßvxbLl³ b|}wP5z{v´Ç³w=vypby{|ÐxK~: v
bv yvß~PvPy_xKB~Pv z°\Lx} K|}w=v z{lz{LK~ x|}b~PÉ~:\ÇPvÍ|}v z{y_LKÍßÅ©vGv 7xL:x7z_vÈz_PvwAv y{pKy_|Ðx~: v%K#v ~:{v | Ë lv}{v Êüs+-ýQý56þÿ 879,: !;=<>,?@A: BFÿ !D#C2"%A:$EF&ÿ ')Q(0ÿ FQ*ÿ +A@-G,H0.H.IDJK-,0EM/2L´13@ý N40+O@+5P Lv z{LK~nK~Ñz_v ~wAv y{pKy_|Ðx~: v|}v z_y{Ì ÍÞÅÆv Ë v LlvKv,z{PÌÌsz{Pvà:yzz{L|}v"xÈLvxy_~PL~PÉ%|}v z{P5Ç,:xK Ë vv ~,v 7xL:x7z_vÇGH0HIRQ@N PObþ°ÿ S%POLT4J
xb y_b_®:xGÓ°LÇPv"0xKy{Lv zÐKwAv y{pKy_|Ðx~: v³|Gv z{y_L Íhwdziwf%kuf%ordps+dkuxqv d@fkulnm%kZdf%g4hf{%mDk+t|s+h;osume%dc;dk+~fs+d %%%zorm--d{?xqf%k+dR-d0x+gk+tyjd0t|csuh;d¡i;idcj-t2¢m%s+d
suhwf@s8v d-mec;msq-ds+dk+ortc;dvhwfsqiwfk4fords+dk4xWt|d{ydxsidk+lnm%kuo]fcwgdvh;dc£sukuf%t|c;tc;¤om--d{x¥¡{{}om--d{xf%k+d
fD;-d0s+mf¦{t|ywkuf%k+c;mo]fss+dkzhwm@v¥%mWm-§m%kywf%§s+h;d
f%k+dD9¨m--d{6i;kud;tgps+tm%cwxRmDcs+h;d©s+k4ftcf%cwh;t{|{?gp{to y;tc;
x+dps4xzf%k+d9gf%g4hwd2h;t?xzx+t|ori;{t|wdxzv mDk+Wtc;¦vt|s+hs+h;d9{|t|~ywkuf%k+ªfcwªo]fDdx]orm--d{Zxd{|d0gsut|mDc«l¬f%xs+dkeyjd0gfjxdsuh;dorm--d{x;mcwmshwf:%dZsumydRdp-d0gp-sud-;kut|cwrx+d{dgps+tm%c=­®Z¯°²±q³;´Kµ¶·¸º¹·q»¼3°¾½Z¿¼¾À¼3¿¼KÁÂ-¶´6·
hwd¤x+t|ori;{dÃlnmDk+vfk4Äom--d{x+d{dgps+tm%c¾i;kum-gpd0-;kudÃi;kudp~
x+dcsud«tcÅs+h;dÆcs+kumW;wgsut|mDcÅtxel¬f%xs9fcjÅdp>dgps+t%d%yw-s
x+m%ords+tod0x)m@%dk;s4xKs+mZs+h;dhwt|{{g{|to ywt|c;Rx+dps0%kud-wgt|cw8dc-~x+dozy;{dZidk+lnm%kuo]fcwgd%)ÇdRorfD-dZs+h;kuddRfD;-t|s+tm%cwx#sumrs+h;t?xx+d{dgps+tm%cªi;kumWgd-wk+ds+mkud-jgpdm@%dk;s+s+tc;wÅh;d0xdf%k+d
;txugpwxux+d9t|csuh;dRc;dpWssuh;k+ddxwy-~x+dgps+tm%cwx h;dx+dRordpsuh-~m-;x6o]f:zydqwx+dplnw{jtc©msuh;dk#fi;iw{|t?gfs+tm%cwx6vh;dkudlnm%kuvfk4
xs+diWvtx+d¦xd{|d0gsut|mDcªt?x©iwk+mDc;ds+mÃm@Ddk+;ssut|c;j#x+wg4hÅf%x©t|c
lnd0f@su;k+dRx+d{dgps+tm%cÈ¬É}m%hjf:t=Êº%mDh;c=2Ë0Ì%ÌÍ%Î
Ï=Ð¬ÑÐ)Ò3ÓWÔ¬ÓWÕÖ0×¬ØÙªÚr×¬ÖÛÅÜ9ÓDÝ)Ô¬ÞwÕ%ÓWßÃÓWÙ>Ö
ÇÅt|s+hàorm--d{¦x+d{dgps+tm%câá ã.ä.åwæ@ç-äkudi;{?f%gdordcsidk+lnm%k+~o]f%cwgpdrtoiwk+m@DdxZf%xZs+hwdydxsorm--d{?xRfkudfD;-d0¦sum9suh;ddcwx+do yw{|dD#idf%-xfcw¤suh;dc£è;t?g4W{|ª-dg{|tc;dxêé)dklnmDk~o]f%cwgpd-kum%iwxydgfwx+desuh;dydxsrorm--d{xrtcsuh;d{|ty;k4fku
hjf:%dyjddc«wx+d«fcjªx+d{dgps+tm%cªo wxs©c;m@vf%;ªorm--d{?xsuhwf@s8hwks8s+h;ddcwx+do yw{|dDëKt%wk+d]Ë x+h;m@vxsuh;t?xydhwf:Wtm%k
lnmDk}k+mWms+~ord0fc-~x+èwf%k+d0W~dkuk+mDkZìc-lnmDksu;cwf@sud{%=ormxsRdk+~
kum%k)ordpsuk+t?gxtd{?o wg4hy;;ori;tdk6%k4fi;hwxMsuhwfcsuh;txvh;dchwt|{{g{|to ywt|c;Zt?xK-mDc;d#vtsuhzx+orf%{|{W;fsufZxdsux@orf%Wt|c;8ts -t|l.~gp;{|ss+mrkud{tf%y;{ei;t?g4efr%mWm-xsum%i;i;tc;]im%tcs6h;dR{mDxuxtcrijdklnmDk+o]f%cwgpdgf%czydxt%cwtjgfcs)tls+h;didf%t?x)ot?xuxd02ëMt%;kud}Ëf{?xmRx+h;m@vxKs+hwfs6xd{|d0gs+tc;}orm--d{?xá ã.ä.å©í+îïð|ñDò4îpóô îõwä=Dk+d0f@s+{ekud;wgpd0xs+h;t?x8i;k+mDy;{doö-d{dgsut|mDcvtsuh¦kudp~iw{fDgpdordcDsf%{|{m@vxormW;d{?x#s+m]ydf%;-d0©sums+h;dRdcwxdo y;{doz;{|s+ti;{|ds+tordx÷b8cwgpdijd0f«ijdklnmDk+o]f%cwgpdtx9k+d0f%g4h;d02
t|ls+hwde;cWwx+d§orm--d{xf{{6hW;k+szdcwxdo y;{d©idk+lnm%kuo]fcwgd%
x+d{dgps+tm%cfD;;x om--d{x s+hjf@szv#dkud9fD;-d0yjdlnm%kudek4f@s+hwdksuhwfc9hW;ks#ijdklnmDk+o]f%cwgpdD)hwtx6øjf@s+s+dcwx s+h;dZijdklnmDk+o]f%cwgpd

g;ku%diwfDxsqs+h;dzijd0f>f%cwf{{m@vxxd{|d0gsut|mDcs+m©jc;dsu;c;ddcwx+do yw{|d0xqyWv dt|Dhs+tc;©orm--d{?xùqom--d{xZf%w-dsumesuh;ddcwx+do yw{|d}o w{sut|i;{d}s+tordxkudgdt%d}orm%kud8v#dt%hs0
0.36

hillclimb: selection without replacement
hillclimb: selection with replacement
test: selection without replacement
test: selection with replacement

0.355
0.35
RMS Error

b8cedf%g4heijdklnmDk+o]fcjgpdqords+kutgqv dZgpm%oriwf%k+dqdcwx+dozy;{|d}x+dp~{dgps+tm%cs+mrsuh;dRorm--d{Kt|cs+hwdR{|ty;k4fku]s+hwfsijdklnmDk+o]xydxs
mDc suhwf@sords+kutg%)#d0gfjxd#v dDdc;dkufs+dxm}o]fcW ;t>dkudcs
orm--d{x{t|y;k4fkutdxKwxjf{{| gm%csuf%t|cflndvorm--d{?x)vt|s+hrdpW~gd{{|dcs8idk+lnm%kuorf%cwgpdzm%cf%cW9idk+lnm%kuo]fcwgdordps+kut?g}%wxsx+d{dgps+tc;8s+h;dydxsxtc;%{d#om--d{Wlnk+mDofZ{|ty;k4fkuZWtd{?;x)kudp~o]f%k+@fyw{|Z%mWm-}ijdklnmDk+o]f%cwgpdDKcjxdo y;{d6x+d{dgsut|mDc=:h;m@v~d%dkqwcw;xdcwxdo y;{dx9suhwf@sm%;s+idk+lnm%kuos+h;dydxsx+tc-~
D{|deom--d{xh;t?x x+;D%dxsuxs+hwfs wx+t|cw-t|dk+dcs {|d0fkuc-~tc;8ods+h;m-;x)f%cwRiwf%kuf%ods+dkKx+dpssut|cwDxK%dcwdk4f@s+d0x={t|ywkuf%k+tdxgm%csuf%t|cwt|c;]fr-t%dkux+dZx+dpsm%lKDmm-W~idk+lnm%kuotc;ormW;d{?x

0.345
0.34
0.335
0.33
0.325
0.32

0

50

100

150

200

Number of Models in Ensemble

ú>ûýüWþ0ÿ    ! #"$ % & '(&)+*ö-d{dgsut|mDcevt|s+hk+di;{?f%gpdordcs6øjfssudcwx#suh;dZg;ku%d}xmzo wg4h
suhwf@sfs+d0xsx+dpset?xecwms9cwdd;d«sum-dpsudkuort|c;dvh;dc£s+m
xs+mDi¤fD;-tc;orm--d{xzs+ms+hwddcwx+do yw{|dDh;dh;t|{{?gp{t|ozy-~tc;x+dpsgf%cydjxd0¦sumxs+mDihwt|{{g{|to ywt|c;jhwtx}ordf%cwx
dcwx+do yw{|drxd{|d0gsut|mDc-mWd0x8cwmsRc;dd0ormDk+d s+dxsxdsuxZsuhwfcsuh;dZywf%x+dp~{|d%d{worm--d{?x6v#m%w{]hjf:%dqjxd0rs+mrxd{|d0gs#om--d{
ijfk4fordpsudk4x36cwx+do yw{|dx+d{dgsut|mDczjxd0xMs+h;d@f{twf@s+tm%crx+dpssum]-m]yjm%s+hiwf%kuf%ods+dkRñõ ,Zorm--d{Kxd{|d0gs+tm%c3
Ï=Ð?Ï=Ð)Ò3Ø/.0Ö0Ó0218Ù43ÓWß65MÔ*Ó87ÙK×¬Ö×¬ÞwÔ¬×:9DÞ-Ö×*ØjÙ
ëwm%kuv#f%kurx+d{dgps+tm%c©x+m%ordpsut|ordx m@%dkwsux d0fku{|zt|cxd{|d0gs+tm%cvhwdcdcwx+do yw{|d0xzf%k+dxo]f{{Ãb8cwd9vf:sumiwk+d%dcszs+h;t?xt?x s+mtc;t|s+t?f{t<;dedcwxdo y;{dxzvt|s+hormDk+d©orm--d{xÆcwxs+d0f%
m%lxs4fk+s+tc;vtsuhÃfc§dori-sdcwx+dozy;{d%)xmDksRsuh;d©orm--d{?xtcsuh;d {t|ywkuf%k+yWesuh;dtk8idk+lnm%kuo]fcwgd%fcwiw-sqsuh;dzydxs
orm--d{?xrtc¤suh;ddcjxdo y;{d% =txeg4h;mDx+dcªyWÃ{|mWmDtc;
=fsijdklnmDk+o]f%cwgpdZm%cs+hwdRh;t|{{?gp{t|ozy;t|cw©xds6h;t?xsWi;t?gf%{|{fD;;?
x >@~@ >ml}s+hwd¦yjd0xs9om--d{x©s+mf%c«dcwx+dozy;{|Bd AuDî Cpæ@íuîDk+dd-xsudiWvt?xd x+d{dgps+tm%cyd%tcwxqöWtcwgpdd0f%g4hm%lMsuh;d =ydxsRormW;d{?x}ijdklnmDk+o]xZv d{|{2suh;dlnmDk+oâf9xs+kum%c;tc;tsutf%{dcwx+do yw{|dfcj t|stx)ormDk+d-<t E©gp;{|s lnm%k Dk+dd-x+d{dgps+tm%czs+m
jcwom--d{xs+hwfsm@Ddk+;svh;dcf%w-d9s+msuh;dRdcwx+dozy;{d%Ï=ÐGF3ÐIHÞKJ-JjÓ021ZÙI30ÓWßL5)Ô¬ÓÒ3ÓÔ*ÓWÕÖ×*ØjÙ
¡8xs+h;dcW;ozyjdk]ml8orm--d{xrtcªf{t|y;k4fku§tcwgpkudfDxd0x suh;dg4hjfcwgdx ml=wcj-t|cwgm%o ywt|cwfs+tm%cwx ml=orm--d{x s+hjf@sm@%dk+;s
suh;d©h;t{{g{|to y;tc;¦x+dps tcwgk+d0f%x+dxef%Dt|cwgfc§ort|cwt|orGt ;dsuh;t?xi;kum%y;{do#Çdzk+d0-wgd}s+h;dzcwo ydkm%l)orm--d{x8xd{|d0g~
sut|mDcgf%c¦g4h;mWmDx+d}lnk+mDoày-k4f:vt|cw©f]kuf%cw-mDoxufori;{dmlorm--d{xlnk+mDo suh;d{t|ywkuf%k+fcwx+d{dgps+tc;]lnkum%o s+hwfsZxufo~
iw{|dD#Ælf]ijfk+s+t?gp;{?fkqgm%ozy;t|cjf@s+tm%cm%l ¨ orm--d{xqm@%dk+;s4x
suh;d iwk+mDywfy;t{tsml suh;mDx+dz¨ orm--d{xqyjdt|c;9t|cf]k4fcw-mDo
yjf©m%l)orm--d{xqtxq{|d0x+xs+hwf%cÃÈN
Ë MPO>RÎ Q lnm%S
k O¦s+h;d lnkufDgs+tm%c

T UWVXTZYZ[\^]S_G`8abK[cedgf/hIij[ke]R[ml8n2oep qKrZdg`eYc/dgfs[`e][Vtce\<[u][\G[vwa_GT `xgo8a_GVX[]ya Tz_<`e]kK{ [Xabedga(abK[c-[]RaVXTZYZ[\^]²`KT | `WhÙ¢N`Åd)aa{&d vwa_G} [zU¯[d)ake{[jTgU[`e]R[VscK\G[È]R[\<[v+a_GT `_^]ab/d)au_<a6ï ðgñT Za _<VX_G¶[aTVX[a{ _Gv]]Rkev&b»d ]uK¢ ª h#ij[| G_ \G\Sbed~}@[VXd `6T@K/T@{Ra kK`K_<a_G[]ya Tc-[8][\G[v+a [YhzbK[uKt`/dg\[`/]R[VscK\G[_^]mabe[yd~}@[{&dgf@[TgUa bK[sxgos[`e][VcK\G[]hd f@]

U¯[{[` aSVX[wa{ _^v]#d {[d KK{ T K{ _Gdga[_G`uYZ_ ò®[{ [` a$][waa_G`Kf@]Sdg`eY

T U4[`e][Vsce\<[]]R[[VvwT VXK\G[wrKcekZaN[d@v&b8[`e][Vsce\<[(_^]ke]Rac-[vdgke][L\G[d {`e_<`Kf¼V[abKTZYK]?ab/d)a8-[{U¯T { Vd| [_<f@b a[YLd~} [{&dgf@[(TgUSVXTZYZ[\G]r]RT?abK[Xd~} [{ d f [(TgUmd?][wa
T U$[`e]R[VscK\G[]yd \G]T8_^]ydP]R_GVXK\<[ | [_Gf b a[YLd~}@[{&dgf [sTgUSa bK[

VX[a{ _Gv(YZTX`KT adg\ | d~Z]m/[{RU¯T@{Vc/d ][wt\<[} [\eVTZYZ[\G]hdgf f@_<`ef(_G]$YZ_G] vwk/]][YX_G`8[v+a_GT `qZh KhóôzØÙõÞRÕeÞRÛ®ö(÷»WøöÎKyIZK° `ja bK_G]]R[v+a _<T@`vT VXed {[P-[{U¯T { VXd `evw[ke]_<`efÈa[`»VX[wa {_^v]Xc-[vdgke][YZ_<UÏt| [uvwT@VXedg{ [X[`e][VcK\<[u][\G[vwa_GT `jaTPa bK[

c-[]RaLVXTZYZ[\^]za { d _<`e[Y
ij[[wZ/[{_GVX[` aK{ T ce\<[VX]¡¢£¤¥Zyr| _ a b][}@[`¦§¨©$ª m«¬ © r/¥ ©   ©$ª h W­ r®dg`eY¥ ©  ©$ª h -xU¯{T@V
a bK[¤ ¦$°Xª [/T ]R_<aT@{±D$\^dg²@[z³µ´z[{¶@rN­·@· ¸ ¹+r$´ © £ ° ®rdºK`K[kKVXT `K_^d»YKd)a&dº][warZ¥W¢ ¦ r YKdga d¼U¯{T@V½vwT@\<\^dgc-T {&d)ta T {&]¼dga¼abe[¾ a&dg`ZU¯T@{ Y¿¥W_<`e[dg{B¢Nvvw[\G[{&d)a T {r8d `eY¿À«t¬ ©Sª Z¬ ©m¦ yrsabK[ ° `eYZ_^dg`Á¬S_<`K[]Âb/[{ ]-[v+a { d \8YKdga d±ÄÃkedg\<a_G[{ _#[asdg\Äh<r$­· ·@·@¹whu¢N£¤N¥ZyrI´ © £ ° 6d `eYZ¥W¢ ¦d {[scK_<`/dg{ ?K{ T cK\G[V]h ¦§¨©Sª m«y¬ © r/¥ ©  ©Sª r/dg`eYÀ«¬ ©Sª

Z¬ ©m¦ Ådg{ [Ær/x ÇKred `eYj­ÇXvw\^d ] ]K{ T cK\G[V]rK{ [wt]-[v+a _<}@[\G hIi6[#vT `} [{Ra [YNabK[]R[a TcK_G`edg{ Nc-[vdgke][Vdg`
T Uabe[8VX[wa {_^v][ ]Rak/YZÂdg{ [8YZ[e`K[YÂT `ºcK_G`edg{ ÈK{T@cZt| \G[V]r(dg`eY¾c-[vd ke][j\G[d {`e_<`Kf»VX[wa bKTZYK]]kev&bÉd ] ¨ ´]d `eY6c/TT@]Ra_G`Kfzdg{ [X[d@]R_G[{aTzd KK\GzaTPcK_G`edg{ K{ T ce\<[VX]h¦§¨©$ª m«¬ © | d ]XvwT@`} [{a[YÈaTÈcK_G`ed {ca {[d)a_G`Kf

a bK[u\^dg{ f []asv\Gd@]]d@]sv\Gd@]]­@h¥ ©  ©$ª | d@]vwT@` }@[{a[Y
a | T | d~Z]hÊ¥ ©  ©$ª h W­Pa{ [d)a&]a bK[jvT `ZU¯k/]d cK\<[L\<[aRa[{Ë§Ë d@]?v\Gd@]]L­Ldg`eYBa bK[L{ [Vd _<`K_G`Kfºx@qÈ\G[waRa [{&]?d@]?vw\^d ] ]oerW_G[\^YZ_<`efPd}@[{ kK`cedg\^dg`/vw[Y6cK_G`edg{ zK{ T cK\G[VPhu¥ © mt ©$ªh /x8ke]R[](\G[waRa [{&]­wt&­d ](v\Gd@]](ozd `eYj\<[aRa [{&]X­ÌgtxgÇd@]vw\^d ] ]­@re_G[\^YZ_<`ef?dYZ_<ÍvkK\<ar®cKkZacedg\^dg`ev[Yr/K{ T cK\G[VPhÀ«¬ ©Sª Z¬ ©m¦  | d ]NvT `} [{Ra [Ya T?cK_G`edg{ c a {[d)a_G`Kf
a bK[(\^dg{ f [yvwT `KU¯ke]d cK\G[yv\Gd@]]ZT) c-[d `ZtÎ´z_<` a _<\d@]vw\^d ] ]­@hbe[][uYKd)a&d]R[a ]| [{ [u][\G[v+a [YÈc-[vdgke][XabK[jd {[\^dg{ f [[`KT kef bÅaT2dg\G\<T |VTZYZ[{ dga[È]R_G¶[ja { d _<`Édg`eYÅ})dg\G_GYed)a_GT `][wa&]r)d `eY]Ra_G\<\ bed~} [$YKd)a&d\G[wUÏaWU¯T@{I\^dg{ f [/`edg\@a[]a]R[a ]hIÐeT {T@kK{[wZ-[{ _<VX[` a&]r | [k/]R[Yua { d _<`K_G`KfX]R[a ]mTgUqgo@o o-T _G` a ]h© d v&bXa { d _<`K_G`Kf]d Ve\<[ | d@]S]K\<_<a_<` aTd(a { d _<`u][waTgUÌ o o o-T _G` a ]d `eYsdbK_G\<\^vw\G_<VcK_G`KfÑ)}~d \<_^YKdga_GT `][waT UW­o@o o-T _G` a ]hbe[(e`ed \a[]a]R[a ]U¯T@{VXT@]RaNTgUa bK[][K{ T cK\G[V]vT ` a d _<`x oKr o o@ou-T _G`@a&]Ò[`KT@kKf bLaTPVdg² [YZ_^] vw[{ `K_G`KfP]RVdg\G\#YZ_<UÏtU¯[{[`evw[]_<`P-[{U¯T { Vdg`ev[{ [\G_Gd cK\G[ h| [\<\T `2T@`K[| [\<\T@`8T abK[{VX[wa{ _^v]h| _ a bádg`àT UXabK[Â\G[d {`e_<`Kfd \<f@Tgt{ _<abKV]redg`eYdg\^]Ta T]R[} [{&dg\®T abK[{[`e][VcK\<[yVX[wa bKTZYK]h  @ýeþ ùú:û-úIüzý/þ)ÿ

¬[{RU¯T@{Vd `evw[VX[wa {_^v]$]Rkev&bud@]#¢ ¦¦ T { ª ´LX{&dg`Kf@[mU¯{T@VozaT¼­@r | bK_G\<[8Tgabe[{&]?±:¥IÐyr4´zã © ¹ { d `Kf [U¯{ T V
oPa T?lb [{ [mlYK[-[`eYK]T `abK[Yed)a deh#ÐKT@{]RT@V[yVX[wa {_^v]\<T | [{| e})d \<kK[]_G`eYZ_^vd)a [c-[waRa [{/[{RU¯T@{Vdg`/vw[ hÐKT@{Tga bK[{&]bK_Gf be[{})d \<kK[]dg{ [$c-[waa[{hbK[$c/d ][\G_<`K[m{&d)a[]4TgUeVX[wa {_^v]]kev&bd ]ª§(¦ dg{ [$_G`eYZ[/[`eYZ[` a#T Ueabe[Yed)a der | bK_G\<[T abK[{ ]]kev&bXd ]¢ ¦¦ b/d~} [ced@]R[\<_G`K[{ dga[]Nab/d)aYK[-[`eYLT `za bK[Yed)a deh ° U¦ ¦ n oKh · ¸Kr®¢ ¦¦ nÙoKh · ¸e­(K{ T ced cK\<_^]N-TT {c/d ][\G_<`K[¢ -[{U¯T { Vdg`ev[ rcKkKa_<UNa bK[d~@[]T@Za_GVdg\¢ ¦¦ noeh Ç@oKr
d@v&bK_G[}_<`efN¢ ¦¦ITdg\G\<T |nBoKhqg·VX_Gf b aIc-[S[Kvw[\G\G[` aI-[{U¯T { Vdg`ev[ hd~} [{&dgf@_<`efLd v{T ]]K{ T cK\G[V]dg`/Y¼VX[wa {_^v]r | [o ­ I]vdg\G[ h § `  vT `} [{Ra-[{U¯T { Vdg`ev[]a T8d`eT { VXd \<_G¶[Ya bK_^]u]vdg\G[ roj{[K{[]R[` a ]ced ][\G_G`K[z/[{RU¯T@{Vd `evw[8U¯T {a bK[
VX[a{ _Gvdg`eYÂK{T@cK\G[V±¯[@h f/h<rSoeh qzU¯T { ª§(¦ rIT@{sa bK[8c/d ][wt\G_G`K[¢ ¦¦U¯T@{$[d v&bK{ T cK\G[Vu¹wrd `eYz­{ [e{[]R[`@a&]a bK[c-[]Ra-[{U¯T { Vdg`ev[][[`?U¯T@{dg`VXTZYZ[\T@`?a bK[e`ed \®a[]RaN][wah¦ ¢¥#r/abK[sV[a{ _Gvske][YaT?V[d ]kK{ [(K{T@cedgce_<\G_ aPvd \<_GcK{&d)ta _<T@`r_G]skK`ke]kedg\$_<`a bed)asabK[8ced ][\G_<`e[uVXTYK[\#a bed)aK{ [wt

YK_Gvwa ]IlU¯T {dg\G\/vd ][]r | bK[{[ _^]abe[-[{&vw[`@a$TgU-T@]_ a _<}@[]_G`a bK[Na []Ram][war b/d ]#[Kvw[\G\G[` amvdg\G_<cK{&d)a _<T@`h ?bK_^]$v{[d)a []d8K{ T cK\G[V| bK[`L`KT@{Vd \<_G¶_G`Kf ¦ ¢¥º] vwT@{[]c-[vdgke][sa bK[c/d ][\G_<`K[(VXTZYZ[\Wdg`eYPd~ []mT Za _<Vdg\WVTZYZ[\bed~}@[y]_GV_G\^dg{¦ ¢¥] vwT { []hI¤N`K\G_<²@[abK[NTgabe[{#VX[d@]Rke{[]r ¦ ¢¥z_^]S] vdg\G[Y]TPab/d)a(abe[uVX_<`K_GVskeV
T@ce]R[{}@[Y ¦ ¢N¥] vwT@{[X_^]oKh oPdg`eY¦
a bK[(VX[d `8T@ce]R[{}@[Y¢N¥] vwT { [_G](­@h oehbe[c/T aRa T VTgUId cK\G[Â­]RbeT | ]`KT { Vdg\G_<¶[Y»] vwT { []U¯T {[d v&bB\<[dg{ `K_G`Kfjd \<f@T { _ a bKV| bK[`º_<a ]/dg{&dgVX[wa [{&]d {[z][wtdg`eYÈVX[a{ _Gvk/]R_G`Kfza bK[u})dg\G_GYed)ta _<T@`]R[a ][`e][Vsce\<[u][\G[vwa_GT `ÈbK_G\<\^vw\G_<Vce](T@`h © `@a {_G[]_<`\G[vwa[YjU¯T {s[d v&bÈK{ T cK\G[VÓNÔPÕeÖ ×SÕeØÙÚÛ®ÅÜÝÕ/ÞRÛij[Âke][Èa[`ß/[{RU¯T@{Vd `evw[ÈVX[wa {_^v]¡àd vvwkK{&d vá±:¢ ¦¦ ¹wr{ TTgatÄVX[d `ZtÎ]â ked {[YtÄ[{{ T {z± ª ´LK¹wrV[dg`¾vw{ T@] ]t[` a{ T ±D´zã © ¹+r\G_ UÏaz±:¥IÐ4¹wrK{[vw_^]R_GT `®Ñ~{ [vd \<\ScK{[dg² t[} [`È/T@_<` a±D © ¬$¹wrK{ [vw_^]_<T@`-Ñ~{ [vdg\G\ÐItÎ]vT { [B±DÐ ¦ ¹+rNd~}@[{&dgf [6K{ [wtv_G]_GT `á±:¢¬ ª ¹wr ª§(¦¢{ [d2± ª§(¦ ¹wr$d `eY2dVX[d@]RkK{ [zTgU
e{T@cedgcK_G\G_ aávdg\G_<cK{&d)a _<T@`ä± ¦ ¢N¥I¹whsbK[Ba[` abÙV[a{ _GvB_^]
å#æNç

nè± æé(éÊê¾ç(ë(éáê{ T ceke]aVX[wa {_^v8aTke][±R­sìç(íßå ¹R¹ î)Kh2Z¢ ª_^]d| bK[`ºabe[vwT { { [v+aXVX[wa {_^vP_G]Xke`Zt

a bK[a&dgcK\G[Ndg{ [d~} [{ d f []IT)}@[{a bK[N][} [`XK{T@cK\G[V]hZvT { []!#"%$'&)(+*-,.0/1/32)#/3$'4-$5/32)6,.798:(6/3$;/32)6<=8?>?,@$'4 /3A+BC8(
4D0E.5F$E3BC8:G)70H5F$:EI987J2K4LE3$'ML(?BN8:G)*OB-0/.E3A7HAP5Q"=;RSGL0"AP/9T
U <%?7?8:&),.V$:56/32LA+,?WXB-98:,.& E3?,Y(ARZO[=\^]_/`> 4)A798(+(P>!8E3CG)$/&),.9*a8(+$:G)'W@M)& /b8cE3O&L,.9*aAGa7?$'Gd&)GL70/3A$'Ga"6AP/32e$:/32)JEbB-98cf,.&LE3?,;,.&L7J2g8:,;hIi;[j/3$KAGL,.&LE3k/32)8/$'GL(P>lB-$S* ?(,H"6AP/32gm'$$S**LA,.70E3AB-AG 8/3A$:Gon:p q@m'$$S*Y798:(AMLEr8c/3A+$:Gb8cE3=,.?(?70/39*Ts=2)A,* $?,G)$:/OB-98G[=\^]tA,K8!4D$$:EoB-J/.E3A+7Ku_AP/OA,O0v070/3AwZx8/K*LA,f/3A+GLm'&LA+,.2LAG)mK4D$$E3(P>l7?8:(AMLEr8/3?*gB-$S*L?(,H5yE3$'Bz"%?((@798(A+M Er8/39*
B-$S*L?(,?T

{ |L}~FY:):3C:?9k ?3?XF3)#%0@ ):+IKS ?:1:rbLD^D:.3't: 3LJ':)¡:@1L.?kL# 0+003'¢Z=??.+:
KSL?£I¤Z0r::)LLr:r¥S+LH¦6P3V§I?3?..'¢L£I¤Z0r::)¡£KS ??) -¨J¥S);3)%?#KSL?D¢£-LDF3:-:3)yJ©
ª6«¬¢­ ®
¯ ° ° ±²?° ®:±Z³ ´L«¢° ¯µL´ ¶L­Lµ ´Dª6² ª#·­ °L¯¢® ²J¯¢´ ª6­S¯¸­ ¸²Z¹#²?­ ®)¹ º¢»y¼)½ ¾ º¢»y¼ ¿ ¿ º¢»y¼ ¼)À º¢»y¼ ¼ Á º¢»y¼ Â ½ º¢»y¼DÁ)¼ º¢»y¼ Â)º º¢»y¼ ÂÃ Ä¹ Å ÄSÆ º¢»y¼ ¼)¾ º¢»y¼ ¾ ¼¶ ¯ÇÈ­ ²0¯'ÉDÊ ÄD¹PÅ ËÆ ÄD¹PÌSÅDÍ ÄD¹PÅ ÎSÅ ÄD¹PÅSÌLÏ Ä¹ ÅLÎSÎ Ä¹ ÅLÏÆ ÄD¹PÅ ÏÄ ÄD¹PÅ ÏSÅ Ä¹ Å Ä Î ÄD¹PÅÐÍ ÄD¹PÅÐ Ì¶L­L²J³
ÄD¹PÅ ËÌ ÄD¹PÅDÍÅ ÄD¹PÅ Î Ï ÄD¹PÅSÌ Ì Ä¹ ÅLÏÅ Ä¹ ÅLÏÌ ÄD¹PÅDÍÅ ÄD¹PÅÐ Ð º¢»y¼ ÀL¿ ÄD¹PÅ ËÐ ÄD¹PÅÐ Æ¯:ÉÊ ¯®'® ÄD¹PÌSÑSÆ ÄD¹PÌSÄDÍ ÄD¹PÅSÌLË ÄD¹PÅSÌ Ì Ä¹ ÅLÎSË Ä¹ Å ÆDÍ ÄD¹PÌ ËSÎ ÄD¹PÌSÄ Å Ä¹ Ì Ñ Ë ÄD¹PÅDÍ:Æ ÄD¹PÌSÅSÄ
²J³Z¯D° Ò ®'´ ÄD¹ËSÎSÏ ÄD¹ÎSÎSÎ ÄD¹PÌSÑLÏ ÄD¹ÎÅ Å Ä¹PÎSÌSÆ Ä¹ ÌSÐLÎ ÄD¹PÑSÑ Ë Ó.ÄD¹PÅSÅSÄ ÓÄD¹PÄDÍ Í ÄD¹ÎÄLÏ ÄD¹ Ð ÄSÆ²rÉÈªÄD¹PÌDÍÑ º¢»y¼LºD¼ ÄD¹PÅÐLÌ ÄD¹PÅSÆLË Ä¹ Å ÑSÑ Ä¹ Å ÑSÌ º¢»yÂDÁ Á ÄD¹PÌ ÎSÌ Ä¹ Ì ÌSÅ º¢»y¼)º ½ º¢»y¼)ºD½¯¸¸ÄD¹PÌ ÎSÎ ÄD¹PÌ ÎSÏ ÄD¹PÅÐLÅ ÄD¹PÅ Ï Ï Ä¹ ÅÍ:Î Ä¹ ÅÍcÐ ÄD¹PÌ ÏÑ ÄD¹PÌSÆ Ñ º¢»y¼Ã ¾ ÄD¹PÌSÅ Æ ÄD¹PÅSÄ Ë¶ ¯DÊ¢Ó`¬³
ÄD¹PÌDÍSÍ ÄD¹PÌSÆDÍ ÄD¹PÅÐ)Î ÄD¹PÅSÆLÎ Ä¹ ÅSÐLË Ä¹ ÅLËSË ÄD¹PÌ ÏÅ º¢»yÂ ¼ ¿ Ä¹PÎSÌSÆ ÄD¹PÅSÄSÐ ÄD¹PÌSÌSÌÒ¸¸ÄD¹ÎSÏÆ ÄD¹PÌÐ Æ ÄD¹PÅSÄ Å ÄD¹PÅSÑLÎ Ä¹ Ì Ì Ï Ä¹ Ì ÌSÅ ÄD¹ÎÆDÍ ÄD¹ÎÑLÏ Ä¹ ÌLÎÆ ÄD¹PÌÐ)Î ÄD¹PÌÐSÐ
¶L²0³ Ó`¬³º¢»yÂ)¼)º ÄD¹PÌSÅSÅ º¢»y¼ ½ Á º¢»y¼DÁLÂ º¢»y¼ ¾)º º¢»y¼L¿Ô ÄD¹PÆSÄ Î ÄD¹PÆDÍ Í Ä¹ ÐÍÑ ÄD¹PÌ ÎDÍ ÄD¹PÌSÄSÆ
¬³
ÄD¹ÏSËÆ ÄD¹ÎÌSÅ ÄD¹PÌ ÏSÄ ÄD¹PÌSÆ Ì Ä¹PÎSÆ Î Ä¹PÎSÅ Ï ÄD¹ÏSÏÆ ÄD¹PÆ ËÐ Ä¹PÎ ËÄ ÄD¹ÎZÐ)Ï ÄD¹ÎSËSË
¶L²0³ Ó`²0³Dª6µ ÄD¹ÎÑ Ë ÄD¹ÎÅSÄ ÄD¹PÅSÄ Æ ÄD¹PÅDÍ:Å Ä¹ Ì ÆDÍ Ä¹ Ì ÑÐ ÄD¹PÑSÄÐ ÄD¹ËÌ Æ Ä¹ Ñ ÌSÅ ÄD¹PÆ ÏSÅ ÄD¹PÆSÆSÅ
Õ Ö:×SØHÙHÚÕÛDÚFÜ×Ý0Ö^ÝJÞ×SÝI×YßoàDÛDÖ:áDâ¢Ö:ØrãäàLØJßæå#ÜcáàLåJÖÝJàkÝJÞÖ;çÈÖ:årÝâÈÖØJãäà Ø0ßæ×SÕÜÖ!à çåJÖØ0è Ö'Ûà Õ × áyáåJÖèLÖÕtâØJàLçáÖßæåéëê¡Öì)×íÝ0ÚyèLÖ;ÖÕ)ÝJØ0ÚÖ:å#ßoÖ:× ÕKÝJÞÖçÈÖ:årÝIßoàDÛDÖáFå#âÈÖØJãäà Ø0ßîçÈÖáàïjç¢× åJÖcí
áÚÕÖ é#ð^à áFÛVÖ:Õ)ÝJØ0ÚyÖ'å#ÚyÕKÝ0ÞÖ;ç¢à ÝrÝ0à ßzàSãQñ@× çáyÖbÙåJÞàïòï;ÞÚÜ?Þ
áÖ:× ØJÕÚyÕìY×Sáì à Ø0ÚyÝJÞßæåó ÚÖáFÛbÝJÞÖçÈÖ:årÝ%âÈÖØJãäà Ø0ßæ×SÕÜÖ6à ÕVÖ:× Ü?Þ
ßoÖÝJØ0ÚÜ é=ñHÞÖYáF× årÝHÜà áôßoÕxÚFåÝJÞÖkßoÖ:×SÕlÕà Ø0ßæ×SáÚyõ:Ö:Ûæå0ÜcàLØJÖ
×LÜcØ0àLå0åÝJÞÖbÝ0ÖÕ!ßoÖcÝJØ0ÚFÜå:éö ìLÕà Ø0ÚÕì!ÝJÞÖlÖÕåJÖßOçáÖ÷ßoÖcÝJÞà ÛåVÚÕÝJÞÖ÷ÝJàLâàSã;Ý0ÞÖ÷Ý?×íçáyÖLøoÝJÞÖ çÈÖ:årÝ×Sáì à Ø0ÚyÝJÞßæåùàè Ö:Ø0× áyáo×SØ0ÖûúDüYýeå:ø÷þ¡êYê¡å:ø× ÕÛlç×SìLì Ö:Û÷ÝJØ0ÖÖ'åøDï;ÚyÝJÞgßoÖ:×SÕgÕàLØJßæ×SáÚõÖ:ÛlåJÜà Ø0Ö:åHÿé ÿ Døÿé Lÿø=× ÕÛÿé ø=Ø0Ö:åJâ¢Ö'Ü9Ý0ÚyèLÖáó éð^à)à)å3Ý0Ö:Û Ý0ØJÖ:Ö:åKÛDà_ÕàSÝâÈÖØJãäà Ø0ß ï^Öáá1àLÕ!ÝJÞÖVâØJàLç×SçÚyáÚ+Ý3ólßoÖcÝJØ0ÚFÜå  ý_úXøXý ¡ø
× ÕÛ ^þ 9øÈçôDÝY× ØJÖVÖ ÜcÖ:áyáÖÕ)Ýà Õ!Ý0ÞÖCÝ0ÞØ0Ö:åJÞà áFÛgßKÖÝJØ0ÚÜ:åþ Yø #ú YøI× ÕÛ =ñ o×SÕÛjà Ø?ÛDÖØ0ÚÕì ßoÖÝJØ0ÚÜ:å  Yø
þ !-øQ× ÕÛe"ð !#9é ö ãIßoÖ:×SÕ â¢Ö:ØrãäàLØJßæ× ÕÜcÖCïH× åYßoÖ:×LårôØ0Ö:ÛàèLÖØ $3ô¢å3ÝCÝ0ÞÖ:åJÖlåJ%Ú ùßoÖcÝ0ØJÚFÜå:ø=ç¢à à)å3Ý0Ö:ÛùÝJØ0ÖÖ'åCï^à ôáFÛùà ôDÝJí
âÈÖØJãäà Ø0ß ç× ì ìLÖ:ÛæÝ0ØJÖ:Ö:å:ø¢ú ükýeåø¢×SÕÛgþ¡êYê¡å:éñHÞÖHÝJà âoà ãXñ%×SçáÖ-ÙåJÞàïå%Õà Ø0ßæ×SáÚyõ:Ö:ÛoåJÜà Ø0Ö:å@ãäà Ø#Ö:ÕåJÖßKíçáyÖkårÖ:áyÖ'Ü9ÝJÚà Õ1ø)ðH×Zó Ö'årÚF×SÕKßoàDÛDÖáX×Zè Ö:Ø0× ì ÚÕìøZÝ0ÞÖ¡çÈÖ:årÝÚÕÛDÚyíè ÚFÛDô× áDßKàDÛDÖ:áå=à ãÈ× Õ)óCÝ3ó)âÈÖ "ð úñ 9ø ×kårÚßoâáÖ×Zè ÖØ?×SìLÖ6àSã
× áyá@ßoà ÛÖáF&å þ1ü ' þ (cøÈ× ÕÛaårÝ0× *Ü ) ÚÕìæï;ÚyÝJÞeáà ìLÚårÝJÚFÜ-Ø0ÖcíìLØJÖ'åJåJÚà +Õ úñ@þ #, 9éú)Ý?× *Ü ) ÚÕ-ì . àLáyâÈÖØJÝ:øQ/Ù #ï;ÚyÝJÞ
áà ìLÚårÝJÚFÜ-Ø0ÖìLØJÖ'åJåJÚyàLÕ!â¢Ö:ØrãäàLØJßæåâÈà à Ø0áyó!çÈÖ:Ü:×SôåJÖCØ0Öì Ø0Ö:å0åJÚyàLÕ
àèLÖ1Ø 0Ý0åkÛDØ?×Sßæ×Ý0ÚÜ:×Sááyó!ï;ÞÖ:Õ_Ý0ÞÖØ0Öo×SØ02Ö SÿLÿ ÿxÞÚyìLÞáóeÜcà ØJíØ0ÖáF×Ý0Ö:ÛÚÕâôÝKßoàDÛDÖáFåo×SÕÛà Õáóû3Ù ) â¢àLÚyÕ)Ý?åOÚÕÝ0ÞÖxè× áyÚyí
Û×ÝJÚà ÕåJÖcÝ'éxþÕôÕ ï^ÖÚì Þ)ÝJÖ'Û ×ZèLÖØ?×SìLÖOà ãH× áyá#Ý0ÞÖæßoàDÛDÖáFåþ1
ü ' þ (CÚFåKç¢ÖÝrÝJÖ:ØVÝ0Þ×SÕjïÖ:ÚyìLÞ)ÝJÚÕìaÝJÞÖlßoàDÛDÖáFåKè Ú×Ø0ÖìLØJÖ'åJåJÚà Õ÷çÈÖ:Ü:×SôåJÖ-ôÕ ïÖ:ÚyìLÞ)ÝJÖ:Ûg×Zè Ö:Ø0× ì ÚÕìOÜ:×SÕÕàSÝ;àè Ö:Ørí0ÝKÝ0ÞÖ!è×SáÚÛ×SÝJÚà ÕtårÖÝ:éòþ¡è ÖØ?×SìLÚyÕìø#Þàï^ÖèLÖØ'ø#â¢Ö:ØrãäàLØJßæå
ï^à Ø?åJÖÝ0Þ×S4Õ $3ôårÝIâÚF*Ü ) ÚyÕìbÝJÞÖçÈÖ:årÝIåJÚyÕìLáyÖ;ßoàDÛDÖ:áå#çÈÖ:Ü:×SôåJÖåJà ßoÖæà ãÝ0ÞÖ÷ßoàDÛDÖáFåCÞ×ZèLÖoâ¢à à ØVâ¢Ö:ØrãäàLØJßæ×SÕ¢ÜcÖ÷×SÕ¢Û_Ý0Þ ôåÞ ôØJÝ!ÝJÞÖÖ:ÕåJÖßVçáyÖ'åé þ¡å!3Ö  âÈÖ:ÜcÝJÖ'ÛQøbâÚ*Ü ) ÚÕìtÝJÞÖçÈÖ:årÝåJÚÕì áÖxßoàDÛDÖá^ãäØ0à ß ÝJÞÖgáÚçØ0× ØJ5
ó "ð ^úDñ CâÈÖØJãäà Ø0ßæåVçÈÖcÝJí
Ý0ÖØYÝJÞ× Õ_×SÕ ógà ÕÖOáyÖ'×SØ0ÕÚÕì÷×Sáì àLØJÚyÝJÞß ÚÕeÝJÞÖVçÈàSÝrÝ0à ß àSãÝ0ÞÖkÝ0× çáyÖkçÈÖ:Ü× ôåJÖYÚyÝÜ×SÕlâÚFÜ*)oÝJÞÖbçÈÖ:årÝHßoàDÛDÖ:áXãäØJàLß ×SÕ óáÖ:× ØJÕÚyÕìKßoÖcÝ0ÞàDÛgãäà Ø;Ö:×LÜ?ÞxßoÖcÝ0ØJÚFÜ-×SÕ¢Û÷Ý0Ö:årÝâØJàLçáÖß!é
ðH×ZóLÖ:åJÚ× ÕîßoàDÛDÖáx×ZèLÖØ?×SìLÚyÕ6
ì 7Yà ßoÚÕì à)å8ø  ÿ ÿ ÿ eÞ¢× å×ßoÖ'×SÕûåJÜà Ø0ÖxàSãbÿé :9ø6åJÚyìLÕ;Ú ö 0ÈÜ×SÕ)Ý0áyóçÈÖcÝJÝJÖØoÝ0Þ×SÕ ú ükýeå
äÝJÞÖ_ç¢Ö'å3ÝlåJÚÕì áÖ!ßKàDÛDÖ:<á cé ÕëðH×Zó Ö'årÚF×SÕjßoà ÛÖá¡×ZèLÖØ?×SìSíÚÕì¢ø Ö:×LÜ?ÞoßKàDÛDÖ:áÚÕoÝJÞÖáÚçØ0× ØJóVÚå#ï^ÖÚì Þ)Ý0Ö:ÛKç óCÝJÞÖáy=Ú ) Ö:áyÚyíÞà)àDÛVàSãÝ0ÞÖ^Û×Ý02× äè× áyÚFÛ×Ý0ÚyàLÕOåJÖc>Ý ìLÚyèLÖÕ-Ý0ÞÖHßKàDÛDÖ:á ï;ÞÖÕßoàDÛDÖ:áâØ0Ö:ÛDÚFÜ9Ý0ÚyàLÕå;×SØ0Ö¡Ý0ØJÖ'×Ý0Ö:Û!× åHâØ0à ç¢×SçÚáyÚyÝJÚÖ:å@?
ACBEDGFHI%JKMLNO/P IQSR OUTWV IYX 3[
Ù Z LNO/P I  Q 3Ù Z R OUTWV I ï;ÞÖØ0Ö A B ÚFå6Ý0ÞÖ¡ï^ÖÚì R Þ)ÝIOUTWà VãI ßoà ÛÖMá \zø LNO/P I ÚFå#Ý0ÞÖY^ÿ ] Ù
Ý?×SØ0ì ÖÝ^ãäàLØÜ×Lår`Ö _?ø×SÕÛÚå^Ý0ÞÖCâØ0Ö:ÛDÚFÜ9Ý0Ö:ÛlâØ0à ç× çÚyáyí
ÚyÝ3óùÝJÞ×SÝKÜ:× åJaÖ _YÚFåKÜcáF× å0å÷Ù é êàSÝ0Ö÷Ý0Þ×Ýoð^×ZóLÖ:åJÚ× Õ ßKàDÛDÖ:á
×ZèLÖØ?×SìLÚyÕìbàLôDÝJâÈÖØJãäà Ø0ßæå"ð úñûà ÕxàLÕábó CàSãQÝ0ÞÖOÙ'ÿCßoÖcÝJí
Ø0ÚFÜå:é. ÚyÝJÞ ×÷ßoÖ'×SÕ_Õà Ø0ßo× áyÚõÖ'Ûeå0Ücà Ø0ÖCà ãÿé cøÈÖÕåJÖßOçáyÖoåJÖcí
áÖ:ÜcÝJÚà ÕÚFå-Ý0ÞÖxÜcáÖ:× ØCï;ÚÕÕÖØ'é ö ÝVà ôÝJâÈÖØJãäà Ø0ßoå-ÝJÞÖlàSÝJÞÖØÖ:ÕåJÖßVçáyÖ^ßoÖcÝJÞà Ûå ×SÕ¢ÛCÝ0ÞÖHç¢Ö'å3Ý#ÚÕÛDÚè)ÚFÛDôD × áDßoà ÛÖáFdå 1à Õ× áyáIÙ:ÿKàSã%ÝJÞÖæÙ'ÿoßoÖcÝJØ0ÚFÜå åJÚì Õ%Ú 0¢Ü× Õ)Ý¡×[Ý e gÿ f ÿ ÿ/Ù 9hé .ùÖ
çÈÖáÚÖè Ö ÖÕ¢årÖ:ßVçáÖ åJÖáÖ:ÜcÝJÚà Õ Üà ÕåJÚFå3Ý0ÖÕ)ÝJáó à ôDÝ0â¢Ö:ØrãäàLØJßæå
Ý0ÞÖ-àSÝ0ÞÖØ;Ö:ÕåJÖßVçáyÖbßoÖcÝ0ÞàDÛåHãäà ØHÝ3ï^àoØJÖ'× åJà Õ@å ?
iÖÕåJÖßOçáyÖCåJÖáÖ:ÜcÝJÚà Õ!ÚFå×SçáÖbÝJàæàLâDÝJÚßoÚyõ:ÖbÝJÞÖ-Ö:ÕåJÖßKíçáyÖCÛ;Ú jXÖØ0ÖÕ)ÝJáóæãäàLØ;Ö:× Ü?ÞgâÈÖØJãäà Ø0ßo× ÕÜcÖbßoÖcÝ0ØJÚFÜ
iàè Ö1Ø 0ÝJÝJÚÕìòÚFå!×jåJÖØ0ÚyàLôålâØJàLçáÖß ï;ÞÖ:ÕëÝJÞÖØ0Öù× ØJÖSÿ ÿLÿVßoàDÛDÖ:áåHÝ0àæÜcà ßOçÚÕÖ éIñHÞÖ-ßKÖÝJÞàDÛåâØJÖ'årÖ:Õ)ÝJÖ:ÛÚyÕ ú Ö:ÜcÝJÚà 5
Õ aÝ0àùÜà ßVç¢×ÝOàèLÖ1Ø 0ÝrÝ0ÚyÕì ÜcàLÕ)ÝJØ0ÚyçôÝJÖlÝJà
ÖÕåJÖßOçáyÖ-åJÖáÖ:ÜcÝJÚà lÕ k åHÖ ÜcÖ:áyáÖÕ)ÝâÈÖØJãäà Ø0ßæ×SÕÜÖ éñHÞÖÜcàLÕåJÚårÝJÖ:ÕLÝ0áyóbårÝJØ0à ÕìâÈÖØJãäà Ø0ßæ×SÕÜÖ6àSãÖ:ÕåJÖßVçáyÖ^årÖ:áyÖ'Ü9íÝ0ÚyàLÕ÷årôì ì Ö'å3Ý?å%ÝJÞ×SÝIôåJÚyÕìCßæ×SÕ óOÛD%Ú jXÖØ0ÖÕ)ÝIáÖ:×SØ0ÕÚÕì-ßoÖcÝ0ÞDíàDÛå¡×SÕÛlâ×SØ?×SßoÖÝJÖØ;åJÖcÝJÝJÚÕìLå;ÚFå;×SÕ!3Ö jÈÖ'Ü9ÝJÚè Ö-ïH×ZóæàSã=ì Ö:ÕDí
Ö:Ø0×SÝJÚÕìx×÷ÛÚyèLÖØ?årÖCÜà ááyÖ'Ü9ÝJÚà Õaà ã#ßKàDÛDÖ:áå:é¡ñHÞÖKÜcàLÕårÚFårÝJÖÕ)Ý
âÈÖØJãäà Ø0ßæ×SÕÜÖ×SáFåràCåJôìLì Ö:årÝ0å#ÝJÞ×SÝ6ãäà Ø0ï^× Ø0ÛKårÝJÖâ ï;ÚFårÖYårÖ:áyÖ'Ü9íÝ0ÚyàLÕ ÚFåb×SÕ Ö jXÖ:Ü9Ý0ÚyèLÖoï^×Zó!àSãHårÖ:áyÖ'Ü9Ý0ÚyÕìgÞÚì ÞDíâÈÖØJãäà Ø0ßo× ÕÜcÖm@no1m3prqs%m/otvudwpyxdzgm@o1m8p{wY|Ym3s<o&tvwu{}~U}u1=mxCw:tm3u1tvwu1
pb}nm`p{mxdu1<3o%tSwU~m3u1gx1x1=ng&<o3}u1m3tvgs=s%83wnx1udws=s=m@|M

APR
ROC
ALL
MXE
LFT
BEP
RMS
FSC
ACC

12

l<lW^88^@55/WU:¡ nCxdzg=o&om/>xd%wnC¢#mawp{}:udm{m3no1m3p&qgs%mao1m3s=m@x1=wnCx1wxdzgm
qm@oxao=ngs%m-p2wY|Ym@s=o£ ¤#¥#¦Y§¨=n©§}:qs%m«ª/¬ro=ngCm@ud3m3nx
udm@|g>xd%wn­%n®m@u1udwu/«¯ngm8}|Y~U}:nxd}mbwtm@ud3m3nx&udm@|Y>
xd%wn°=n°m3udu1wu8wp{}:udm@|x1wngwudpb}:s=%±@m@|²odwu1m/ob=o8x1z}Uxnwudp{}s%=±3m/|«odwu1m/o`*z}:ngmb}o`qmxxdm3u&p2wY|Ym@s=or}:udm&tvwgn|}n|+o1zg%t³xxdzgmxdw+wthx1zgm{o1@}:s=mgl`§"zgm&m@ud3m3nxudm@|Y>
xd%wn+=n+m3udu1wutvudwp´p2wY|Ym@sµ¶x1w·¤G|gm3m3n|gowns%·wn xdzgm
m3u1tvwudpb}:n3m@o4w:t"p{wY|Ym3s<oµ¨}:n|C¤4¸Mngwxwn¹owp{mbw:x1zm3up{wY|Ym@sMx1z}:x|Ymngm@o#xdzgm`x1w·wt(xdzgm m@utvwu1pb}nm od3}:s=m§w@}:s<gs<}Uxdmhm@ud3m3nx(udm@|Yx1=wn =nm3ududwu/¸3¢#mhu*ox3wn^~m@uxm/}*z+p{m3x1ud=x1w8s=wodo¢zgm3udm&º8u1m@gu1m/om@nxdom3u1tvm@x4m3u1tvwu1
pb}nm+}n|»ª udm3u1m/om@nx*o2¢¼wudoxbm@utvwu1pb}:nm°½Sm@utvm/>x
u1m/|Y=x1=wn©^%m@s=|gob¾4¿ ¿ÁÀÃÂrÄÅÆÀÈÇÉ[¿ÁÀÃÊ Ë ¿yÀ
¾Å ÊÌÀÍÉh¾ÊÎÀ6ªr}n| Ê ÏÍÉhÄ6ÀÍÏGÐ«Ä6ÀÍ¿`¾Ñ5ÀÒº¸}s%sw:t¢zg<*z²z}W~m s%wo1o-ºg¶µn»mÓg}:p{gs=m+¢=s%s4zgm3s=l ¡ tµÔÔÎ=p{gudwU~m@o tvu1wpÕº×ÖØ-x1w«º×ÖÖ^¸x1zgm-s%wo1o1m@o&}:udmaºg×ÙØ}n|²ºg×Ù:Ú¸¼udm@o1m@>xd%~m3s=¸"}:n|©xdzgm+m3u*m@nx-u1m/|Y>xd%wn5%nm@u1udwu8=o·ÛÜ86¯ngmwx1m3nxd=}s4|Y<od}|Y~U}:nx*}:m wtm@ud3m3nx
udm@|g>xd%wn %n s=wodo<ox1z}Ux%x%~m@o"p{wudmm3p{gz}o1=o"xdwaudm
|g>xd%wnor}:xs%wU¢6s%wo1o@Ýudm@|Y=ng s%wo1o tvu1wpÕº ºÙaxdw+ºg ºgª
<o}aØºÜÞudm@|g>xd%wnl¸g¢zgm@ud}ou1m/|Y3%ng{s=wodo"tvudwpEº Ùº&x1wº%ª/ß«£ o1}p{m{}qo1ws=Yx1mb*z}ngmW¬>¸M<o wngs=+} ØÜ´udm@|g>xd%wnl
¡ n o1wp{m|Ywp{}%no#x1zg<oqg<}o<o}ggudwgud=}:x1m ¡ nwx1zgm@udo%x<o"ngwx@[àwu1pb}s%=±3m/|-odwu1m/o#|gw2nw:xz}W~mxdzg=oqg=}o3

ä§}:qs%m`Ù ozwU¢ohx1zmm3u*m3nx#u1m/|Y>xd%wnb%n8s%wo1o[tvwum@no1m3p2
qs%mom@s%m/>xd%wnáwn®x1zgmÖ·xdm@ox2u1wqgs%m@pbo2}n|²ª/º+p{mxdu1<3o@¸
3wp{}u1m/|`x1w4x1zm¼qm@oxhp2wY|Ym@s=oho1m3s=m@x1m/| tvwuSm/}*zru1wqgs%m@p}n|­p{mxdu1<:Cµo&¢;xdz®ngwu1pb}:s==±3m@|­odwu1m/o3¸n}s¼m3u1tvwu1
pb}nm/o}:udm`m@ox1=p{}:x1m/|-wns=}u1mn}:sMxdm@oxom3xdonw:xo1m@|
tvwu(xdud}%n%ng}:n|r=o(x1zgm"}W~m@ud}mSwU~m@ulx¢#wx1ud<}:s<o(¢%x1z&m@}*zu1wqgs%m@p#½Swo%x1=~m`m3nxdu1=m@o=nx1zm xd}qgs%mrp2m/}:nm3ududwu¢"}oudm@|gm/|M¸Y  mhm@no1m3prqs%mo1m3s=m@x1=wn·m@utvwu1p{m@|-qmxxdm3u/¥no1m3prqs%mo1m3s=m@x1=wn ¢=noâãbw:t¼ÖUº2xd%p{m@o2£o1=ng%3}nx}UxÀÈºgå ººª/¬5¯n}W~m3u*}:m¸hm3nom@prqgs=m·o1m3s=m@x1=wn©u1m/|Y3m@o
s=wodoÛgæÖUâ^ÜÎwU~m@u¼xdzgm`qm/oxp{wY|Ym@s=o"tvwum@}*z8gudwqgs=m3pE}:n|p{m3x1ud=r§(w·pb}:çmrxdzg=o 3wn3u1m3x1m¸%txdzgm{qm/ox p{wY|Ym3shz}o}33gud}¹ßº ººÜ8¸Ss%wo1o&=o2ºg=ª@ººº¸S}n|®}nÛgæÖÜÈudm@|Y>
xd%wn«=nCs%wo1o`wu1udm@o1wn|gox1wudm@|Y=ng-s=wodo4x1wºg ºßª@Ú8wu
=n3u1m/}o1%ng+}33gud} xdwßº Û^ÖÜ8·¦Y%p{=s=}u1s=¸S%t#xdzgm8qm@oxp{wY|Ym@sz}o#èé¦aº Ùºººg¸}n-ÛgæÖÜÌudm@|Y>x1=wn8%n-s%wo1o¼wu1
udm@o1wn|go(x1w`udm@|Y=ngèé«¦ xdw4º%ª/ÛÙ:âl§(m/ox[o1mx*o3wnxd}%nÙºg¸ ººº«@}o1m@o@ ¡ n3u1m/}o1%n¹}33gud}áº Û^ÖÜÕp{m/}:no ªUÖWãp{wu1m 3}om/o"}:udm4u1m/|Y=x1m/|-3wudu1m/>x1s=aq^bxdzgm m3no1m3p&qgs=m@o@§"zm%p{gudwU~m@p{m3nxh=ongw:x¼|gud}p{}:x1<:¸qgYx¼3wno1=ox1m@nx1s=r=nY
3u1m/}o1%n}@gu*}3 ºg ÛÖÜêwuSu1m/|Y=ng`èé¦&ºg ºgªUÖWã3wp2
}:udm@|«x1wxdzgmaqm@oxrw:tÙ:ººº·p{wY|Ym3s<o <o %p{gudm@odo1%~mb¢zgm3n
xdzgm qm@oxp2wY|Ym@s=oz}W~m`o*z-wwY|·m3u1tvwudp{}nm¤#Yx¢#m

Mean Percent Advantage Over Best Single Model

10

8

6

4

2

0

0.2

0.4

0.6

0.8

1

Fraction of Models in Bagged Samplesëìæíî/ï1ð"ñ:ò^óô*õôdö÷`ø@ùóú@û/û@ô>ü­ý:ô*þæô*ÿd÷æø/õ«ø/õ«÷ô2ôd÷ æÿô æûU÷ ;üô[ø3ù÷ ôhû úrú3
÷ ôô ô*õU÷ (õø ^ú3û/û ;õû|gwn x¢"}:nxxdwawU~m@udoxd}:x1m`x1zmr=p2u1wU~m3p{m3nxw:thm@nom@prqgs=mo1m3s=m@x1=wnl gwurwp{}:ud=o1wn(¸lq}:=ng wu qw^wox1=ng xdu1m@m@o^=m3s<|go}aÙºÜ¨u1m/|Y>xd%wn%n s=wodowp{}:udm@|-x1wbu*}W¢êx1udm3m@o@¸Ùg Ø&xd%p{m/ox1zmqm@ngmxw:thm3nom@prqgs=mom@s%m/>xd%wnl#¤#Yxx1zgm@nl¸
=p{gudwU~=ng xdzgm4m@utvwu1pb}:nmwtlp2m/|Y=w^3u1m¸z%za~U}:ud=}nm
p{wY|Ym@s=oo1*z}o"|Ym@3=o1=wnaxdu1m@m@o"<o"m@}o=m3u¼x1z}n-=p{gudwU~=ng
xdzgm m3u1tvwudp{}nm`wt(x1zm ~m3udbqm@oxp{w^|gm3s<o3l!(#"%$'&(& #&C/*)¶,+®.-0/21433/5&=gudmÙ o1zgwU¢o[x1zgmm@ud3m3nx¼udm@|Y>x1=wnb=nas=wodo[wtm@no1m3p2
qs%m o1m3s=m@>xd%wn8¢%x1z·q}:=ng&}o ä ¸^x1zgm4tvu*}x1=wn8wt(p{wY|Ym3s<o=nxdzgmq}:o@¸W~U}:ud%m/oltvudwpÞª|YwU¢nx1wºå ºÙØ#tvwu(xdzgm4ª@ºp{mx1
ud<3o}W~m3u*}:m/|}udwodoxdzgm·Ö-gudwqs%m@p{o@-µ#x ä À ª¸q}o3wnxd}%n·}:s=sMp{w^|gm3s<o¼¢z=*z·=o"m 6g%~U}s%m@nx#x1w{ngwxq}:=ng¯na}W~m3u*}:m¸Uq}m/|2m3no1m3p&qgs%mom@s%m/>x1=wnbudm@|gm/o[s=wodo[}:n}|g|Y%x1=wn}s^Ùg ØÜ°}:x ä À°ºgå×ØY§"zgmudm@o1gs%xdo%n2§}qgs%m/o[ª#}:n|Ùr}:udmtvwu ä À²ºå Øg¸}r~U}:s=gm¢#mo1m3s=m@x1m@|aqm3tvwudms=wwç^%ng&}Ux
xdzgm@o1mudm@o1gs%xdo@¸}:n|{¢zg=*z7=gudmÙ`o1zgwU¢oh<oo1gqwYx1=pb}:s §"z=oÙg ØÜÒ<o[}:qwgx}4x1zg=ud|2wtxdzgmx1wxd}sÛgæÖÜGqm3nmgx¢#m
o1m3m ¢%x1zm3no1m3p&qgs%mo1m3s=m@x1=wnl8
 %gudmrÙ£v¢zg<*z o1m@oxdzgm
n}:sxdm@oxo1mx*od¬"o1gm@oxdo ä %n·xdzgm`ud}ngm`ºgå=:ª 9Cºgå Ú2¢¼wgs=|^=m3s<|®tvuxdzgm3ub=p{gu1wU~m3p{m3nx/<
 ;Cm+}:udm·3gu1udm3nxds%­m3Ó^m3u1
=p{m3nx1=ng©¢;xdzêo=ng3u1wo1o-~U}:s==|}Ux1=wn»x1w©g<*ç5}ngm/}:uwYx1=pb}:sM~U}:s=gm wt ä tvwum/}*z·gudwqgs=m3p }n|-p2m3x1ud=
=>?A@BDCFE:G'HJIKMLN?AOP@BQBQRSTKURPVXWYEHUZ[]\*G_^a`bRPc
d@e_G'RPcfOPCgB

m3udm¢#m`gudm@o1m3nx"}23}om`ox1|Y8¢zgm3udmwYx1=p{%±@%n{}n8m@nY
o1m3p&qgs=m"x1w4x1zgmwududm@>xSp{mxdu1<"z}o[%p{}>x[wn{}:nb}ggs==@}UhiQjkmlon8prqQsutTvwTtx]yz:t|{}DwTy~x%~x gxD tJDtUtTtTwTy~x74tvyDt5tTy%r{tv:x]tmvxD~xM:mv~yD2
¡¢

£_¤D¤

¥]¦¤

¥4§

D¨¤

£

D¡

_¢a¦

¢©¨¡

¤D£¨

¦T£¨

¢¡£¨ª£«¨¬'§¤QD¸¡ §D¹¡¡r§Q§_¡f® °¡r§Q§_¡f® Q³¢a¡«¨Å!¦È_¹¡ ¦¡¤D§¦£_¤­_®¯r­¼º »4½ ¾¿°·Q® µ·­_®³4µ¶_®­r¶´Q® µ¶³Q®o°m±°® ±²º À½o»Á³]¯'®·³µ¨®³·ºÆ½ÃÄ]Ç´_®³r±´_®´´¯'®³r²µ¨® ¯­¶'®!°¶°m¶r¶'® ¶¶¶'® ²­´_® ±²²'® ²´´_® µ¶º À½¿Âµ'°® ²²¶'® ±¯³_®´r­³]¶'® ±¯­'®´r¯´_® ²³­'®´³³]²'®·´°® ±­µ¨® ¶¶³r·_®³r²³_® ²³­_®¯r¶¼º »4½ÃÁÂ°´Q®¶³°m¶_®o°m­¯_®´]­°m­_®´r´³Q®o°m²¶_®·Q°­_®·]±³4µ'®²³µ'® µ¯¶_®¶]µ·Q®·Q°¶_®­r¯°r®­r¯¯ ®¯r¯_°´Q®­]µ±_®¯´¶_®o°³°²_®·r´¶_®·Q°±_®­r¶°r® µ'°³Q®¯_°º!¾Q»4½o»Ä±_® µ²³]¶_®²r±´Q®o°³µ'®¶³µ'®¶´­]µ'®·4µ¯_®³r·°r®´r·°r°r®±_°°r®¯·µ'®³Q°³ ®¶]µ³Q³]¶_®·4µ´°m¶_®¶r²r²°r®±r¶_°°m­_®·r·r³­_®­r¶_°¢a¡r£¨ªµ¨®´]µrµ´Q®o°m±'°°m²_®¶r±¯°m¶_®´r·]µ°m¶_®´]²±·Q®o°m¯r­´Q®±r²]µ·_®´_°m­°r®³r´4µ±_®o°·]¯²_®¶r¶r²ÉTÊÃËÌ2ËrÍuÎ2ÏÐÑ'Ê!Ì'ÒÓÃÒÏrÔTÌ'Ê!Ì'ÕÖÉËÏÖ×¨ÏÔPÉTÊ!ÐmÓÃÒ×'ÑQØ_ÙÊ!ÐÙ×'ÔTËÚ'Ó!ÒmÎÛ
Ü ÉÉÑ'ÒNÝDÉTÏÌ_ÍÞËÔßáàÊ!Ì'ÒÏrÔ Ü ÐmÐmÒmÓ!ÒmÔÏ]ÉËÔâ:ÒmÌDÉÒÔäã¼Ý_à Ü â8åçæÉTÔTÏÊÃÌ¨Òß%ËÌ¨ÐmÒ×¨ÏrÔÏrÎ2ÒmÉÒmÔÙÏrÔTÒUÙÒmÓ!ÒÐ|ÉTÒßfÛ 8Ñ'ÊoÙaÐÏrÌÏrÓoÙPËMÚ(Òß'ËÌ'ÒôÊÉTÑéÒmÌÙPÒÎÚ'Ó!ÒÙÒmÓ!ÒÐçÉÊ!ËÌÛ Ü ÌQØ ÙÉTÔTÏrÉÒÕØÍÞËÔXìÙPÊ!Ì'ÕÑ¨ÊÃÕÑèÒmÌ¨ÒmÔTÕØ×¨ÏrÔÉÊoÐçÓ!ÒéÚ(ÒÏÎÙêÏÔÒAÐçËÓÃÓ!Ê!ß'ÒßáÉTËëÕÒmÌ'ÒÔTÏrÉÒÙì'Ú¨ÏrÉËÎ%ÊoÐ×¨ÏrÔÉÊoÐçÓ!ÒÙÛ ÜÎÏ]íËÔ2ÐÑ¨ÏÓÃÓ!ÒmÌ'ÕÒîÊ!Ì<ÉTÑ'ÒÙÒÒçïQðÏÌ¨ßÔTÒmì¨ÙÊ!Ì'Õÿ]ÏrÓ!Êoß'Ï]ÉTÊÃËÌ7ÙPÒmÉTÙæDÊ!Ì¨ÐmÓÃì¨ß'ÊÃÌ'Õ2ÐçÔTËÙTÙÿ]ÏrÓ!Êoß'Ï]ÉTÊÃËÌæ

×(ÒmÔTÊ!Î%ÒÌDÉTÙ2ÊoÙ%ÉË*ÐçËÔÔTÒÐçÉÓ!ØñÐçÓoÏÙTÙPÊÃÍÞØNÉÑ'Ò0×ÏrÔÉÊoÐçÓ!ÒîÉTÔTÏÐòQÙÛó ÒÔPÍÞËÔÎÏÌ¨ÐçÒJÊ!ÙÎ2ÒÏÙì'ÔÒß7ôÊÃÉÑÉTÑ'ÒÝ_à Ü âöõXðÙTÐçËÔÒ÷ÔTì'Ì'Ì¨ÊÃÌ'ÕMÒmï_×ÒÔÊ!Î2ÒmÌDÉTÙôÊÃÉÑôÑ¨ÒmÔTÒùÊoÙuÉÑ¨Ò×ÒÔTÐmÒmÌDÉËrÍ_ÒÿÒmÌDÉÙÏÐÐçÒm×'ÉÒßJÍÞËÔ×'ÔTÒß_ÊoÐ|ÉTÊÃËÌæ
ÏÌ¨ß



ýÊoÙ#ÉÑ'Ò8×'ÔTËÚ¨ÏÚ'Ê!ÓÃÊÃÉØMËrÍ¨Î2ÊoÙÐmÓ!ÏÙÙÊÏ×'×'Ó!Ê!ÐÏ]ÉTÊÃËÌ_ðÙ×(ÒÐçÊ





ÐmÏrÉÊ!ËÌÛÝ_à5õÊoÙÏrÌ

Ð×(ÒmÔÍÞËÔTÎ2ÏÌ¨ÐçÒ8Î2ÒçÉTÔÊoÐ:ÉÑ¨ÏrÉÒÙPÉÊ!ÎÏ]ÉÒÙÉTÑ'ÒäÙPÉTÏrÉÊoÙÉTÊ!ÐÏrÓÖ×Ë]ôÒmÔËrÍMÉTÑ'ÒéÎ2Ë_ß_ÒmÓ¼Û
ÚQØ

Ê!Ù2Ò ì¨ÊÃÿ]ÏrÓ!ÒmÌDÉ%ÉËAÑ¨Ï4ÿQÊÃÌ¨ÕÌÐçÔTÒÏÙÊ!Ì'Õ<Ý_à5õÎ2ËÔÒß'Ï]ÉÏ'æôÑ¨Ê!ÐÑ×(ËrÉTÒmÌDÉÊoÏrÓ!Ó!ØäÙTÏ4ÿÒÙÖÑQì'Ì¨ß'ÔÒß'ÙMËrÍÉÑ¨Ëì¨ÙTÏrÌ¨ß¨ÙMËrÍß_ËÓ!ÓoÏrÔÙ
Î2ËÔÒËÔ

Ê!ÌÏÐmÐmÒmÓ!ÒmÔÏ]ÉËÔaÉTÊÃÎ2ÒÛ   



'Û



0ã,Ïäúü×ÒÔPÍÞËÔÎÏÌ¨ÐçÒæîÊ!Ì_ðÊÃÎ2×'ÔTË]ÿÒÎ%ÒÌDÉåË]ÿÒmÔÖô:ÒÓÃÓÃð

Ë×_ÉÊ!Î2Ê ÒßÌ'Òmì'ÔÏrÓQÌ'ÒmÉTÙÏrÌßß_ÒÐçÊoÙPÊ!ËÌMÉTÔÒÒÙÛ5Ý_àõ
Ó!Ê!òÒÏÐçÔTËÙTÙÚÒmÉô:ÒÒmÌ






ÏÐmÐçì¨ÔTÏÐçØÏÌ¨ßÌ¨ËrÉ0ÙPì'ÔT×'ÔTÊ!ÙÊ!Ì'Õ*ÉÑÏ]ÉÚ¨ÏrÕÕÒßÚ(ÒmÑ¨Ï4ÿÒÙÐÏrÓ!ÊÃÚ'ÔÏ]ÉTÊÃËÌæÙË7ÊÃÉJÊoÙ

ÉÑ'Ò%ÙPÊ mÒËrÍÉÑ'ÒÑ'Ê!ÓÃÓoÐçÓ!ÊÃÎÚ'Ê!Ì'ÕÙÒçÉUÉTË7Ê!Ì¨ÐçÓ!ì¨ß_Ò%ÏrÓ!ÓgËrÍ

?A@rqA@UsPALU_IK`t]F_IC_XfM _LuZ+vuMOw_yxzQU` _Y[ZIC_` {}|}~ N 6 a " aH,N | 
6 }a|} ~ " a A
NN 8
|  
,
&



6 a " a $
[H}

U
}
py a  }
U +
5ÏrÚ¨ÓÃÒ

0ÙÑ'Ë]ôUÙJÉÑ'Ò2ÉTËrÉTÏÓôÒmÊ!ÕÑDÉMÕÊ!ÿÒmÌäÉËÒÏÐÑäÉØD×(ÒËrÍóÎ%ÒmÉÔTÊ!ÐaËÌ Üà *ÏrÌ¨ßâÛÎ2Ë_ß_ÒÓÍÞËÔgÒÏÐÑ8Ñ¨Ò

Ï4ÿÒmÔÏrÕÒëÏÐçÔTËÙTÙ0ÉTÑ'ÒÙÑ'Ë]ôUÙ#ÉÑÏ]ÉÏrÌß

óÙPÉÔTÊ!òDÊ!Ì'ÕÓÃØYß_Ê (ÒÔÒÌDÉëÍÞËÔÜæ:ÏrÌßÔÒÐçÒÊÃÿÒÉTÑ'Ò×¨ÔÒmÍÞÒmÔTÔÒßÍÞËÔ

ß_Ê (ÒÔÒÌÉ

ó

â

æ

Ýâ

äÏrÔTÒÚ(ÒÙPÉËÌñÝ_àõÛÔTÒmÿÒmÔÙPÒßáÍÞËÔ ×ÒÔPÍÞËÔÎÏÌ¨ÐçÒ8Ê!Ì¨ÐçÔTÒÏÙPÒÙË]ÿÒÔ5ÉÑ'ÒUÚ(ÒÙPÉÊÃÌÐçÔTÒÏÙÒÊ!ÌÒ uÒÐ|ÉTÊÃÿÒUÙÏÎ2×'ÓÃÒUÙÊ mÒàÛ 8Ñ'Ò2ôÒmÊ!ÕÑDÉTÙMÏÔÒó æÝÝôÑ'ÒmÔTÒ

Î2ËÙPÉ7ôÒmÊ!ÕÑDÉÛ8Ñ¨ÒmÔTÒ

ÏrÓoÙPË

ÏÔÒAÙPì¨Ú¨ÙÉÏrÌDÉÊoÏrÓMß_Ê uÒmÔTÒmÌ¨ÐmÒÙêÚÒmÉô:ÒÒmÌèÉÑ'ÒNÎ2Ë_ß_ÒmÓÖÉØQ×ÒÙÉTÑ'Ò7Ú(ÒÙPÉ%ÐÏrÓ!ÊÃÚ'ÔÏ]ÉTÊÃËÌN×ÒÔPÍÞËÔÎÏrÌÐçÒÙÒmÓ!ÒÐçÉÊ!ËÌæDÝ_à5õÎ2ÒçÉTÔÊoÐmÙãÞÔTÊ!ÕÑDÉNÐmËÓ!ì'Î2ÌåñÔTÒÐçÒÊÃÿÒaÉTÑ'Ò8Î2ËÙPÉôÒmÊ!ÕÑDÉ

Ê!ÌAÉÑ'ÒÒÌ¨ÙÒmÎÚ¨ÓÃÒ%ÍÞËÔâÉTÔÒÒÙÏrÌ¨ß*Ó!Ë]ôbô:ÒÊÃÕÑDÉÚÏrÕÕÒßÉÔTÒmÒÙmÛ 8Ñ'ÊoÙ

ÉÒÌ

Ý

ëÉTÑ'ÒAÎ%Ë_ß_ÒÓ!ÙêôÊÃÉÑÑ¨ÒmÌ7ÏrÌ2ÒÌ¨ÙPÒÎÚ'Ó!ÒÊoÙË×_ÉTÊÃÎ2Ê mÒß%ÉËÝ_àõáôÊÃÉÑÒÌ¨ÙPÒÎÚ'Ó!Ò

ÒXÐçì'ÔTÔTÒmÌDÉÓ!ØÏÔÒÉÔÏrÊ!Ì'Ê!Ì'Õß'Ï]ÉÏ'æ_Ì'ËÉíì¨ÙPÉÉÑ'ÒMÑ'ÒÓ!ß_ð¼Ëì_ÉMúmòîÙTÏrÎ2×'Ó!ÒÙÛÉTÔÒÒÙ

4ð¼ÍÞËÓoß2ÐçÔTËÙTÙ5ÿ4ÏÓÃÊoß'ÏrÉÊ!ËÌÉËMÊ!Ì_ðÜÏÕÕÒßFÉÔTÒmÒÙ<Ñ¨Ï4ÿÒÉTÑ'ÒèÚ(ÒÙPÉÝ'à5õÐmÔÒÏÙÊÃÌ¨ÕÝ_à5õÐmÔÒÏÙÒ
ÉTÑ'Ò

Ý_àõöøáù'ãúû*ü]ýJåPþ

pÐÏrÌÚ(ÒUì¨ÙÒß2ôÊÃÉÑ7ÒÌ¨ÙÒmÎÚ¨ÓÃÒUÙÒmÓ!ÒÐçÉÊ!ËÌÛéÝ

Î2ÒçÉTÔÊoÐmÙÛÕÊ!ÿÒÙ

ÉË

ÏrÌß

ÒÙÉ7ôÒmÊ!ÕÑDÉ2ÍÞËÔÚ(ËDËDÙÉTÒßà

æ5Ú'ì_É%ÉÑ'ÒÙPÒîô:ÒÊÃÕÑÉÙÏÔÒÛ Ü ÓoÙPËæ Ü
XÙ0ÕÒçÉ Î2Ë_ßQðäÝöã,ôÑ¨Ï]É2ÉTÑ'Ò Ü
XÙ7Ë×_ðÏrÌ¨ßÉTÊÃÎ2Ê mÒ4å|æfÚ'ì_ÉMÓ!Ë]ôô:ÒÊÃÕÑDÉXÍÞËÔ'ËÔöÒmï_ÏÎ2×'ÓÃÒæ*ËÌ

Ñ'Ê!ÕÑFô:ÒÊÃÕÑDÉéÉËÜÛ 8Ñ'ÊoÙÙPì¨Õrð

ÕÒÙPÉTÙÉÑ¨ÏrÉ:ÒmÌÙPÒÎÚ'Ó!ÒJÙÒmÓ!ÒÐçÉÊ!ËÌ7ÊoÙaÏÚ'ÓÃÒXÉËÒçï_×'Ó!ËÊÃÉÉÑ'ÒââÏÌ¨ßéà

ß_ÊÃÍ ðÔTÒm×¨ÔÒÙPÒÌÉÙgÏÓoÏrÔTÕÒa×ËÉÒmÌDÉTÊ!ÏÓDÙTÏ4ÿQÊÃÌ'ÕDÙfÊ!ÌÏÐÐçÒÓÃÒÔTÏrÉËÔÉÊ!Î%ÒÛÍÞÒÔÒÌDÉ:ÙÉTÔÒÌ'ÕrÉTÑ¨ÙaÏÌ¨ßÚ'ÊoÏÙÒÙËrÍfÉÑ¨ÒJß_Ê (ÒÔÒÌDÉaÓÃÒÏrÔTÌ'Ê!Ì'ÕÏrÓÃð

"!$#&%(')+*,''%.-/102/,3546*678*698:<;=-98>?A@CBD@FEHG8IKJCLGNMOJKPRQSGTQULSVWJKICICX ICJKY[ZJCQ\^]F_ M `a^bdcfehgCicfjKgCkfl &m Tn
2oÕËÔTÊÉTÑ'ÎÙaôÑ¨ÒmÌîË×_ÉÊ!Î2Ê ÊÃÌ¨Õ%ÏÌîÒmÌÙPÒÎÚ'Ó!ÒUÉTËÒÏÐÑ7Î2ÒçÉTÔÊoÐÌ¨ÙÒmÎÚ¨ÓÃÒîÙÒmÓ!ÒÐçÉÊ!ËÌ*ì¨ÙÒÙÉÑ'Ò*ÙPÒmÉÉËÕÊÃÿÒÒmÌ¨ÙÒmÎÚ'Ó!ÒUÙPÒÓÃÒÐ|ÉÊ!ËÌÏÌ%ì¨Ì_Í,ÏrÊ!Ô:Ïß_ÿ]ÏrÌDÉTÏÕÒ8Ë]ÿÒÔËrÉÑ¨ÒmÔ8Ñ'Òäÿ4ÏÓÃÊoß'ÏrÉÊ!ËÌèÙÒçÉîôËì¨Ó!ßáÚ(ÒéÌ¨ÒmÒß'ÒßÉTËëÙÒçð

Ó!ÒÐçÉÉÑ'ÒM×ÏrÔÏrÎ2ÒçÉTÒmÔÙaÍÞËÔÒÏÐÑêÏÓÃÕËÔTÊÉTÑ'ÎaãÞ×ÏrÔÏrÎ2ÒçÉTÒmÔUÙÒçð

Ó!ÒÐçÉÊ!ËÌåçæ5ÏÌ¨ßAÉÑ¨ÒmÌNÉË0×¨Ê!ÐòéÉÑ'Ò7Ú(ÒÙPÉÏÓÃÕËÔTÊÉTÑ'ÎÙÒmÓ!ÒÐçÉÊ!ËÌåçÛã,Î%Ë_ß_ÒÓÌ¨ÙPÒÎÚ'Ó!Ò2ÙPÒÓÃÒÐ|ÉÊ!ËÌäì¨ÙÒÙJÉÑ'Ò2ÿ]ÏrÓ!Êoß'Ï]ÉTÊÃËÌäÙÒçÉ

?A@F@PDYFM GNMOJKPRQG8I"PD`M

ÉÔÏrÊ!Ì

ÒÌ¨ÙÒmÎÚ¨ÓÃÒÙmÛ UÊ!Ó!Ó!ÐmÓÃÊ!ÎÚ'Ê!Ì'ÕËÌÉTÑ'ÒJÿ]ÏÓÃÊoß'Ï]ÉTÊÃËÌîÙÒçÉß'ËDÒÙaÌ¨ËrÉÎ2Ë_ß_ÒÓ!ÙÛÏÌ¨ßîÉËÒÏÐÑ×¨ÔËÚ'ÓÃÒÎÛ

ì'Ê!Óoß_ÊÃÌ¨Õ<Ó!ÊÃÚ'ÔÏrÔTÊ!ÒÙÊoÙîÒmï_×ÒÌ¨ÙPÊ!ÿÒÛÒÐÏrì¨ÙÒ0Î2ËQß'ÒmÓoÙÏÔÒAÊ!Ì¨ß_Ò×ÒÌ¨ß_ÒÌÉæÊÉ7ÊoÙÒÏÙØ*ÉË*×¨ÏÔTÏÓÃÓ!ÒmÓ!Ê mÒÎ2ËQß'ÒmÓXÐmÔÒÏ]ÉÊ!ËÌ



f

ÏÌ¨ßMß_ÊoÙÉTÔÊ!Ú'ì_ÉTÒÉTÔTÏÊÃÌ'Ê!Ì'ÕUÏÐçÔTËÙTÙfÎÏÐÑ'Ê!Ì'ÒÙÛ É#ÉTÏròÒÙÏÚËì_É~Ñ'Ëì'ÔÙÉTËXÉTÔTÏÊÃÌÉÑ'ÒüÎ2Ë_ß_ÒmÓoÙ5ì¨ÙÊÃÌ'ÕÏXÐmÓÃìÙÉTÒmÔËrÍ'ÉTÒmÌ

àgÊÃÌQì_ïÎÏÐÑ'Ê!Ì'ÒÙmÛ 0Ë_ß_ÒmÓgÉÔÏrÊ!Ì'Ê!Ì'ÕÊ!ÙXÏrì_ÉTËÎÏ]ÉTÒßfÛ 8Ñ'ÒÔÒÊoÙÖÌ'Ë×¨ÏrÔÏrÎ2ÒmÉÒmÔJÉì¨Ì'ÊÃÌ¨ÕËÔÖÒmï'ÏrÎ2ÊÃÌ¨ÊÃÌ'Õê×(ÒmÔÍÞËÔTÎ2ÏÌ¨ÐçÒËÌÿ]ÏÓÃÊoß'Ï]ÉTÊÃËÌ0ÙPÒmÉTÙÛÙPìÏrÓ!ÓÃØêÌ'ËËÌ¨ÒÎ2ËQß'ÒmÓgÊoÙXÐmÔÊÃÉÊoÐmÏÓ.æÙPË7ÊÃÉÍÞËÔ×¨ÏÔTÏÎ2ÒçÉÒÔÙÒmÓ!ÒÐçÉÊ!ËÌæÎ2Ë_ß_ÒmÓ8ÙPÒÓÃÒÐ|ÉTÊÃËÌæÏrÌ¨ßNÒÌ¨ÙÒmÎ%ð

ÊoÙ%Ì'ËrÉÌ'ÒÐçÒÙÙTÏrÔTØéÉTËNôÏÊÉ2ì'ÌDÉÊ!ÓUÏrÓ!Ó8Î%Ë_ß_ÒÓ!Ù2ÏÔÒîÉÔÏrÊ!Ì'ÒßÚ¨ÓÃÒÐçÔTÒÏrÉÊ!ËÌÛÉTËäìÙPÒÉÑ'ÒÓ!ÊÃÚ¨ÔTÏÔØÛÐÏrÌÚ(ÒÊÃÉÑÎ2ÏÌQØNÏÓÃÕËÔTÊÉTÑ'ÎÙÿ]ÏÓÃÊoß'Ï]ÉTÊÃËÌëß'ÏrÉTÏ

×'ì_ÉîÚÏÐòñÊ!ÌÉÑ'ÒÉTÔTÏÊÃÌáÙÒçÉÏÌ¨ßëÉÑ'Ò0Î2Ë_ß_ÒÓXÔTÒçðp

8Ñ'Ê!Ù%×'ÔTË]ÿQÊ!ß'ÒÙ%ÏrÌëÏÌQØð¼ÉÊ!Î2Ò

Ï4ÿËÔ Of2 .¡¢d£¤.¡6¥¦¡¢§h ¨©¤ª"§h«d¡¢¬­¤.®$¯§±°8¡².¡¢¬©¤³F´µN¡¢¶®·F¸®f¹ ¡¢ºh¶§h¬¤.¨¡»A¬¶¼¡²½z¾ºh¡¢¶¢¿
ÀTÁTÂTÃ(Ä ÀÅÅ Æ©Ç¢Å Ã ÆdÄ È ÉRÅ ÀRÊÈ Ë Ì Ê ÈNÍÇ Í2ÎTÌ Ç²ÀTÈÀTÝ Ò8Ò8ÒÒ ÓNÓNÔ±Ô±ÓÓÕ©Ú©ÓfÓÖÜ ÓNÓNÔrÔ±ÓN×Ø8×Û×ÙÞ ÓNÓNÔrÔhÜ©×ÓNÞfÙ×OÚÜ ÓNÓNÔ±Ô±ØfÓfÙfØÛÙfÙÞ ÓNÓNÔÔ±ÛÓØfÚ©Ó©ÞfÛÙ ÓNÓNÔhÔ±Ó©ÚdÛÛ ØÞfÙÚ ÓNÓNÔrÔ±Óf×ÙfÓfÙÓfÓÕ ÓNÓNÔ±Ô±ÓfÓfÖfÓÛ8ÓfÓ× ÓNÓNÔhÔ±Ó©ÜfÛÕ©ØÖÚÚÇßÁTÏRÄ Í ÓNÓNÔ±Ô±ÓfÓÓfÚ©ÓfÓNÙ× ÓNÓNÔ±Ô±ÓfÓfÓØfÛÓfÓÙ ÓNÓNÔrÔ±Óf×ÓfÓÖfÕ©ÙÓ ÓNÓNÔhÔ±ÓfÚ©ÞfÞ Û8Õ©Ö× ÓNÓNÔhÔ±Ó©ÚfÛÕdÛÖNØ× ÓNÓNÔ±Ô±ÓfÓNÖ×OÞfÚdÛÞ ÓNÓNÔ±ÔhÓÕdÛÜfÕdÜ©ÛÙ ÓNÓNÔ±Ô±ÓfÞfÓÙ ÓfÕdÛÓ ÓNÓNÔ±ÔhÓÚ©Ø©ÚfÛÚdÛÜ
ËËÀNÇ²ÄÐ ÁTÁ8Ä Ä ÓNÓNÔ±ÔrÓf×fÓ×ÓfÚÓ× ÓNÓNÔ±ÔrÓf×OÓÜÓfÚ©ÞØ ÓNÓNÔ±Ô±ÓfÓÓfÚfÓfÜ©ÓÓ ÓNÓNÔ±Ô±ÓfÓÓÜØfÕfÞÚ ÓNÓNÔ±Ô±ÓfÓfÓØNÜ×Ö× ÓNÓNÔ±Ô±ÓfÓ©ÛÓ ÚfÕ©ÓÚ ÓNÓNÔ±Ô±ÓfÓÓÚdÛÜfÕÜ ÓNÓNÔ±Ô±ÓNÓÚf×OÓÞfØÚ ÓNÓNÔ±Ô±ÓNÓÕd×ÛØfÙÞË Ç²Ä Ç²ÄNÍÊ ÓNÔhÕfÕfÜ©Ö ÓNÔ±ÙfÙ Ü©Þ ÓNÔr×OÕ©ÓÕ ÓNÔr×Ù ÜfÕ ÓNÔr×ÞfÓfÙ ÓNÔhÜdÛÕ©Þ ÓNÔ±ÓfÓfÓfÓ ÓNÔ±ÓfÓÓfÓ ÓNÔ±ÓfØÚ×ÅÉÏ Ä àDÊÌ ÀÅÅ Æ©Ç¢Å Ã ÆdÄ È ÉRÅ ÀRÊÈ Ë Ì Ê ÈNÍÇ Í2ÎTÌ Ç²ÀTÈÀTÝ Ò8Ò8ÒÒ ÓNÓNÔ±ÔrÓN×OÕ©×fÖf×ÛÓ ÓNÓNÔ±ÔrÓN×Ù×OÓÙÚÜ ÓNÓNÔ±ÔhÓfÜfÓfÕfÓÜ©ÖÕ ÓNÓNÔ±ÔhÓfÚfØÜNÞ×ÞÚ ÓNÓNÔ±ÔhÓÚ©Ö©ÚfÛÚ©ÞÞ ÓNÓNÔ±ÔhÓfÚ©ÓÓ8Þf×ÙÙ ÓNÓNÔ±Ô ÛÓfÞØfÙ©ÕÛ× ÓNÓNÔ±Ô ÛÓfÖÚfÙfÙfÙÙ ÓNÓNÔ±Ô±ÓØfÙ©ÜfÛÚØ×ÇßÁTÏRÄ Í ÓNÓNÔ±Ô±ÓÓfÙfÚ©ÓfÓfÙÙ ÓNÓNÔ±Ô±ÓNÓÜf×OÙNØfÖ× ÓNÓNÔ±Ô±ÓfÓNÞ×OÕ©ÚdÛÞ ÓNÓNÔrÔhÚ©×ÓØØfÞNÖ× ÓNÓNÔrÔhÚd×ÛÓÜ©Ú©ÞØ ÓNÓNÔ±Ô±ÓÓÜfÚfÓÖÕÜ ÓNÓNÔ±Ô ÛÓNÓf×ÓfÞfØØ ÓNÓNÔ±Ô±ÓNØfÙ×OØ©ÞÛÜ ÓNÓNÔ±ÔhÓfÚ©ØÓfÓfÕ©ÖÓË ËÀNÇ²ÄÐ ÁTÁ8Ä Ä ÓNÓNÔ±ÔhÓfÜfÓ©Ü©ÛØfÖØ ÓNÓNÔ±Ô±ÓfÙNÓ×OØ©Ü©ÛÙ ÓNÓNÔ±ÔrÓf×OÓNÚ©Öf×ÖÞ ÓNÓNÔ±ÔhÓfÚfÓÕfÖÞ©ÛÚ ÓNÓNÔ±ÔhÓNÚ©Øf×OÖÜ©ÓÕ ÓNÓNÔ±Ô±ÓfÙ©ÛÓÙ©ØfÛÞ ÓNÓNÔ±Ô±ÓNÓ©Û×OÜ©Ú©ÙØ ÓNÓNÔ±Ô±ÓÓÚÕÚfÚfÚÕ ÓNÓNÔ±ÔhÓ©Ú©ÛÖØfÚ©ÞØË Ç²Ä Ç²ÄNÍÊ ÓNÔr×OÕ©ÓÚ ÓNÔr×ØfÛØ ÓNÔr×ÖN×Û ÓNÔ±ÓfÞÓ©Û ÓNÔ±ÓÕ©ÖÕ ÓNÔ±ÓÜfÖfØ ÓNÔ±ÓfÓfÓfÓ ÓNÔ±ÓfÓÓfÓ ÓNÔ±ÓfÓfÖÜ
á²âtãäTåßãæç8èãHåßãèãOéáßêâäAë2ãOäTå(ãOæ­ç8èãOåéOìfä¦çRãzá²í²ìêä8ã îïRå(êä8ðñäTãò ãOì©î8áßãOãOî+óõ2ãOíö÷áætêå"âNãîNìãOåßèå{øìá²âíßãôì î8ìdó©îpìfætêèìâ ç8íßèã}ã æñâNò îNãOãOäèå"á èrò ì©ãôá²ããí äTõåßùFãêæçTç8í²ìèãôíßêãOêråå
éOìfä çRãç8ï8êèúáûçDãüýâ íßãtá ã{þRãOí(üýâ íßæ{ìäTéãætãáßí²êrétêåHÿä8â äçDãOéOìfïTåßã6á ò ãzèêç8í¢ìfí²êã å2áò ò ãæ{å(ãOèó ãOå"îNâ­äTâfá"î8ãþDãäTî âä{á ñ ò ãætãáßí²êéá ò ì©á ñ êèèçDãïTåßãOî}áßââ þNáßêætê  ã2á ò ããäTåßãæç8èã õ  ò êrå
æ{ìÿãOåætâNîNãOè+èêç8í¢ìfí²êã åó ãí²øpí²ãïTå²ìfç8èãõ âí ñ ìí²î å.á²ãþ ñ êåßã ãäTåßãæç8èã åßãèãOéáßêâä êrå ã péêãOä á õ î8îNêä8ð,ìætâNîNãOèá²â,ìfäHãOäTå(ãOæ­ç8èãâ ä8èø}íßã ï8êí²ãOåFìdóãí¢ìfð êäTðìætâNîNãè å"þ8í²ãOî8êéáßêâäTå ñ êá ò á ò ã$ãäTåßãæç8èã åþTíßã îNêéáßêâäTåñö÷üHò á êé ò ãí²ã[êrå ìíßã $üýâætí âN îNãèrá å{ò ãá²âå(ê é  ã{âââ ü"åßãôá ò üýãíßâ ò æêèáèéèêæ­êråpç8êêrä8åð[îNâåßãä8á ã õ
ò
ò
ò ä 
 á²êætãOå6üýâí}ãOìé ò åßãèãOéáßêâäyå(áßãþFõö÷üå(ãOèã é áßêâäôêrå,í²ï8
å(áßãOþTåOá ò ãéâ å(áFâfü8ãOäTåßãæ­çTèãåßãèãOéáßêâäHêråFâ ä8èø "!$#%!'&(ì åßåßï8ætêä8ðá ãHætãá²íßêréHéìä çDãûéâ ætþ8ïNáßã î¦êä " õRö÷üUá ãætãáßí²êéWêråpætò â íßãy*ã )NþRãOäTå(êóãyá ò ì+ä "ýã õ ðR,õ .-./ í²ò ã 0ï8êí²ãOå{åßâíßáßêä8ðìfäTî á ïTåê1å !dèâ2ð "3í²ãOéâ ætþ8ïNáßêä8ð
á ò ãzæãáßí²êézîNâ æêäTìfáßãOåOò 5õ 4,åßêä8ðtì 6 879 êætþ8èãætãOä á¢ì©á²êâ :äåßãèãOéáßêä8ð ìfä<ãäTåßãæç8èã üýí²âæ ìuèêç8í¢ìfí²ø âf;ü  <>= ? ? ?
ætâNîNãOèå +ì ò êèèréèêæ­ç8êä8ð åßãá ñ êúá ò @<BA ? ? ?tþDâêäá²å +ìfäTî
C<D=$?2?Wá¢ìfÿã åtìfçDâïNápì^æêäïNáßãôâ äuì^ætãOîNêï8
æ 0 þDâ ñ ãíñìçRââ í²ïNÿNFáå.á¢=$ì©?á²êâ ætäAêõtäïNö÷á²üãOåßåãá²èãOâ{éáßç8êï8âäêèî êråHá çT"ã ìfð GTðäTã ìEî è+ã= äT?åßãá²æêætç8ã è*åã êõ úáHá¢ìfÿ ãOåòÀ Ï8Ð ÑzÄÓNÓNÔhÔ±ÓfÚfÖfÕfÞNÚ©Ó×ÓNÓNÔrÔhÚf×ÓÜ©ÞNÜ××
ÓNÓNÔ±Ô±ÓfÓfÓÙfÞfÜdÛÖ
ÓNÔ±ØN×OÕ©ØÀ Ï8Ð ÑzÄÓNÓNÔ±Ô±Ó©ØfÛ8ÙÚ×Ó×ÓNÓNÔ±ÔhÓÚ©ÓNÜfÜ©×OÞÕÓNÓNÔ±Ô±ÓNØÜ©×OÞÜ©ÙÚ
ÓNÔ±ÓfÖfÓfÙ

þDãíßüýâí²æ{ìfäTéãûâfüã ìé ò âüá ò ã å(ããäTåßãæç8èãOå,éOìfäyçDããó©ìfèïp0ìfáßã 1î ï8êré¢ÿèø âä¦á ãûætãáßí²êr$é 8á ãHãäRå(ãOæ­ç8èã­éìäôçRãHâ þNáß,ê 0æt ê  ã îHáßâ}á ò ì©áæãáßò í²êéçø$á ò êråðò í²ãã îNø dç8í²ïNáßãüýâí¢éãåßãOìfí¢é ò õð ââî ãOäTå(ãOæ­ç8èãtïTåßïTìfèèø êèèçDãtüýâï8äRî^êü6å(â ætãçRìåßã0èãó ãèUæâNîNãOèå}âízéâ æ­ç8êäTì©á²êâ ñ äT å6âü2á ò ãOæ øêãèrîWð ââNîôþDãlí 0
üýâ íßæ{ìäTéã{â ä á ò ìfá­ætãáßí²êréfõ èá ò âï8ð òñ ãî8âyä8âfáÿäTâ ñòætâ ãñ áßí²áßêéOâ,*å â áþNáßêãHætãê äT ãåßãá æò ãç8èçRãûìéåßãìf0÷ä¦èãOçDóãHãè âætþNâá²êî8ætãêèråFãOá²î â,á²æ{âìfá äø$ãæôâfüNõ á ò ãOåßã
òò
H:Iwv:I8x]iYaUsR8yzxNR9cp^bY|{} ce~ ~N9f2cpMN UaRäTåßãæ­çTèãå(ãOèã é á²êâ äêrå2å.á²í²ìêð á(üýâí ìfí¢îzüýâíç8êäTìfí²øûéèrìå²å(,ê 0
Déì©á²êâ äWìfäTî¦í²ãðí²ãOå²åßêâ äAõö÷üá ò ò ã­çTìñ åßã0÷èãOóãOèDætâNîNãOèå,ætìÿãþTíßã îNêéáßêâäTå2üýâíæï8èáßêþ8èãzéèrìå²å(ã å*ä8âûætâNîNêGDéì©á²êâ ätáßâHá ò ããOäTåßãæ­çTèã å(ãOèã é áßêâä&þ8íßâNéãOîNï8í²ãyêrå¦ä8ãOéãOå²åßìíßøüýâí¦æ­ï8èáßê,0éèì åßå­þ8í²âç8èãæ{åOõWö÷ü,å(â æãçTìåßã 0÷èãOóãOèætâî8ãèråû7 æ{ìfÿ ãpþ8í²ã 0
î8êéáßêâäTåâä8ã[î8êé ò âáßâ æ­ø ìfá¦ì á²êætdã Cãõ ðTõ n #yå 6ãOpä 0åßãæç8èã­å(ãOèã é á²êâ ä¦êrå6ã ìåßêã å.á}êúüá ò ã­çTì å(*ã 0 èãó ãè+ætâNîNãèrå,ìíßãéâæç8êäTãOî[åßâpá ì©á$á ãøWí²ãá²ï8íßä ìþTíßã îNêéáßã îyþ8í²âçTìç8êèêá.ø
üýâ ípãOì é ò éèrìå²åõò ^ã ò ò ìdóãôä8âáøãáã)NþDãí²êætãOä á²ãOî ñ êá òæï8èáß,ê 0÷éèì åßåãOäTå(ãOæ­ç8èãHå(ãOèã é á²êâ äAõGqEklkF äTåßãæ­çTèã åßãèãOé á²êâ ä ïTåßãOåuüýâ í ñ ìfí¢î å(áßãOþ ñ êåßã å(ãOèã é áßêâäýüåtïTå(ïRìfèèøìfí²ãpá²í²ìêäTãOîá²â[ætêä8êætê ãWéí²â å²åûãOäáßí²âþø çTí²èâã æ1åá òèêç8ì©áí¢ìfìfí²í²êã6ã åââfþNüá²êá ætò êâ ïTã å²î{ìfäRáßâûî8åá ò âfãzü2ðætêóâNãOîNä{ãèrþRåãOá²í(âpüýâ ç8íßæ{ï8êìfèîôäRéãOã6äTåßætãæãlá 00â íûå ïTìíßã î[ãOíßí²âí õ UíßãOãOåûìäTî yå­ïTåßïTìfèèø[ætì Nêætê ã í²êréfhõ 4,åßêä8ð ìó©ìíßêãá.ø âfüûèãOìfí²ä8êä8ðìfèðâ íßêá ò ætåìfäTîSþR'ì 0ì ééï8í²ì éø õ "ââ å(áßêä8ð­ìfèråßâûêå"îNã å(êðä8ã îtáßâæ{ì Nêætê ãzìééï í¢ìfætãáßãíåßãá(á²êäTð å"ìfþ8þDãOìí²åáßâçRã$ã ãOéáßêóã,üýâíðãOä8ãí¢ì©á²êä8ðí¢ìéøõ âætãûætãáßí²êéOå,åßïTé ò ìåþTíßã éêrå(êâä ©íßã éìèèAìäTîìíßã ò ìí²î[áßâWâ þNáßêætê ãtáßâRõ "ãOéìïTåßãætâNîNãOèìdóãOí²ìðêä8ðêrå èèêãOç8éí²áßìêâíßä:êãOååUæâfüâîNå.êáó êãæí¢å(þD2ã âíßò á²êìð äò áFüýãOïTìfìfáßèï8êá.í²øãêrætåUâNá òîNìfãOèáåOêõ á2éìfäTä­åßãâæþNç8á²êèætã$êåß ã ã 0
üCì å.á ãäTåßãæç8èãyåßãèãOéáßêâäSéìäuáßí²ø ìîTîNêäTðãóãOíßøæâNîNãOè ãOäTåßãæ­çTèãHþDãíßüýâí²æ{ìfäTéã$áßâpìfäø ãOìåßêèø¦éâætþ8ï8áßãOîôþDãíßüýâlí 0êäSá ò ã[èêçTí²ìíßø áßâá ò ã^ãäTåßãæç8èã ì©áã ìé ò (å áßãOþAõö÷üûá ò ã æ{ìäTéãûætãáßí²êéõ  )NþDãí²êætãäá¢å ñ êá ò åßãó ãäôáßã å.ázþ8í²âç8èãæ{å:H I,J8I5K;L8M NOPN Q N R8S%TVUXWRZY\[F]2^'_`Ua^bOdceR8f2]gh]iM*^'N f
Fjkj

l
m
Xn 7 #
$)

8o
')
p0

qn
sr
t-./uo "*(s*l¡ ¢;$£*¤¢l¥w£*¦¦l§e'¨El§a'8¦l*¢©ª,«¦3ª, £¬¥,2P£2¦3¥w¦3l*iª,­%®e¦*¦l*¢¯©eª°¦§' ±pa3¡2l¢²$ª°ª
l§ek¢pp*ªw¦³a¥°£ª°±p¥°e´m¢ppª°¦kl$¥°e¨q¥,l§©$´2´ ¥°e´³©s2¦3l¥°e´³e µ¶b­ ¦l¥w$m¢pp*ª8b· *$´2¥,´¸¹qº"»½¼F¼¾¿kÀFÁÃÂÄÅXÆFÁ3Ç3ÀFÁ3¿kÈÉËÊqÀ.¾ÇÌ¯ÁÍkÎÏ9ÎaÁ¾ÐÑtÒtÒÔÓ ¨«(±¦l¯Õ Ö;·'$ª°±e¦k$¤×Ë$e´2¥,e´¡ ¢Ø×ÚÙÜÛ(l×ÝÙÞ ßÃà'áiâ`ãä åß*Þ¸5æ%F±¦lFçèkèé¨q¥§uê¤±a£ª°¥°p $mp¥w¦3$a£ ½ê¤±£*ª,¥wp $½p¥°¦3 £t¨«*¥°´ §ië©i­ë´2 ¥,½'l¥°¸zæE ª°¦l±a¦3"p¥w¦$£*k¨«*¥°´ §il uçèkè³p$uª,p£* ª,ª°­¨V¥,´2§ilb·2*$´2¥,e´a¸ì¶§eí2*e*ª$¨q¥w§¦:¡2ª°p£*$ª°ª°­F¨«*¥°´ §il b· 3¬ ´ ¥°e´·'$­(¡ ¢îÕ'ïq¯Õ2ðïq¥,¢¦¤l§eF¢¥,e¥°¢±¢ñp¥w¦3$a£©s¨«*t$­¨Vs ¥°i¦q¥°1l§e" ¥,¦3*¸ò ÒtÒÔÓ ¨V¯l$¥°%¦(¨q¥§E´22p¥°*i"e¦£*i©2£íil2 E·'$­tl§eui±¢©s*( «§e¥wepX±ee¥,¦óiÛ ³5Õp³ôa³9õ³8öiÕp³
ÛbÕ$õe÷$§e¢2¢*il±e¢²óbøe³Zøe¸ùÕp³Zøe¸ùúp³Zøe¸ ûp÷2¸qæEp :ü ±a¦3m·' ª,¥we'¥,2P¦3*¦(l%e¨V¥,´2§ip£b­%2( lª°­E¦3l2p¬¥,e´a¸ÚýÃ¦2³(¨«P¦ þl§de*¦$¢;$­+e¥ÿZ**iap£§¦q¦ll§$k¦32¢ ¦.±ep3®e.2¶'· *l®e ¸;Ó ¨«¯·'$­1l§e;¦leª°¥ll¥°e´1£*l¥,ll¥° ³el±e¥,´2pl¥° ¦³
 ¦l¢$l§¥,e´ 8$eªw £*¥° h Ôµ¶b­ ¦l¥w$h¦l¢ l§e¥°e´ ¸æE±a¦3«$ª°ª2 pl§e FìÔ¢ppª°¦8¥,µ«±eil¥°e ü ¦ýè \ £í'$´2µ¶b­2¦³:ý Fö
³  5ì"
³ 5ìqø³ Vôa³ ¤³  ¸æE$ªw¦3´ e*'kl¦«$8­a Vô ô  Vô¨q¥,l§eel±p¬¥°e!´ "³ Vô ô2#µ $ Vô2ôq¨q¥§(µ¶b­ ¦l¥w$"¦3¢$§e¥,´ ³b$a% ¬&&'(P¨q¥,l)§ $ª°2£¥w$¦l¢ l§e¥°e´ ¸ ** +¤'· i¦
, k2¢¥,e´22¦³Õ$ø2ø -ö ¤¡ .e¦£¥,p¥,2¦¶$.Vô ôa¸/ 1ò 032 ;Ó ¨«u©a$´Õ úl*¦$.  £§Ô­a2¸ê2£§El* ¥,m21¯©si ¦3l$1¦l ¢ª, ¥°¦q2ep m§e"ª,¥°©e$­ ³
2¦.¨«*ª°ª82¦ql§(® ª8©a$´ ´21*a¦3¢©eª°l§a' b·2*$´2¦¶$ª°ª
§e¦l(l¦¸qæz¥§ /546 2  ¨«(©ai¦  £§1l(­a2¸µ«2¦3l¥°e´E£*$d'· *l®e ³5¦lE¨V1 ed©a2¦3l  ìq¦§eª°¥°©e l­t$ l*ó'Õp³Zô³:õ³ ÛÖe³öiÕp³:Ö$ôa³5Û Õ$õ³:Õ ú$Ö³:úpÛ Õe³5Ûø2Õ$ô³
Õ ø$ôiõp÷u¦3l*a¦ .©a2¦3l¥°e´a¸æz¥,l§ /547 2 4798;: ¨«u±¦l
¦3l±¢a¦ ¦l¥,´ ª°Xª,· *ª(e£¥w¦l¥,2+l<¦ ¨q¥§úëe¥ÿZ**i¦leª°¥ll¥°e´ë£*l¥,ll¥we³¶  £§z©sii¦ó'Õp³qô³.õ³ÛÖ³¶öiÕp³.Ö ô³
ÛbÕ$õ³pÕ2ú$Ö³úpÛ Õe³ZÛø2Õ$ô³eÕ$ø ô2õ³2ôiø û Ö³põeÛûiÕ÷"¦*¦¸
46=*8?> Ó ¨V"±¦l"¢2¦3¶í2*e*ªw¦«¥, !@%:¥°´ §i9 A222£§e¥°¢¦³
Û û û2-û uóª°¥,$ ³s ª°­e ¢¥w$ªke*´ *Õ , öe³«2p¥w$ª.¨q¥,l§¨q¥w§+óbøe¸ ø øÛ ³ø¸ ø2ø2úp³Vøe¸ øeÛ ³Vøe¸ ø2úp³«øe¸°Û ³«øe¸ùúp³kÛ2³«Õp÷ ÷t$·' l­ul§e(*´2±eªw$C¥ B 'l¥° 1  ¢*l D©­ £l ¦q 9*¡ ¢ Û ø EGFPÜÛ ø H$¸ ì¶§e½ ±pe±pX e´ ë$@ ¦Ô¥w¦
ICJ KML<NOKQP ¥,¦3l  ë$ I ø L Û P ¸zì8E¢; í 1l§e @  e¬e¥°£l¥° ¦.£* ¢$l¥°©eª,"¨q¥,l§$§e*q¢ppª°¦³e¨« ±a¦39 +ª°$3ü ¦¢*l§epëlP£* · 31 !@  2±pl±p¦Ôel2©$©¥,ª°¥¥, ¦©­®3l¥°e´§e*¢ lm¦3¥°´ ¢2¥°R +ª°$3³Û û û2-û ¸
ò*SUT6V&W!X5YZ![\^]_Z VG` >V§a$ª, ¦ V§e¥°±$¯ç" §¦l±eí «çb¨« ´ ±£§¥ §eª,s¯¨q¥§§e¥°e¥,l¥w$ªqp ¦3¥°´ ë$k¦3¢©eª°¦3ª, £l¥° ¸EæE1l§ eíPì8 ­

a ±a$ª,l¥°*¥e¡ V§e*ª°m¨q¥,l§;§e bdc%+Vêe +Vê«ì½e'e³i$2±e£ ª°ªw$©s 'l2¦;$& ¡ 1§e*ª°é¨q¥,l§|l§eQ7
' fg a3¡2l¢;$a£ ¢*l¥°£ ¸
h ¾6i ¾:Î¾¿Zj ¾Ðµ«ªw$í2 ³6F
 ¸,³ , t
 k2B ³lF ¸ Û û û2õ- ¸ m%V ý.*s2¦l¥ ­u ¢;'¬£§e¥°e"ª, $e¥°e´;e'$©2¦3 ¦*¸µ«*¥°¢;$:³9¸nÛ û û2Ö-¸Dµ« ´ ´2¥,e´½ee¥°£l ¦¸poqr<s!tvuxw
y wzq"{|u}tvu!~$6³ Ue³8Û Õ$ö sÛô2øe¸F¥°3*¥°£§³ì"¸ a ¸# Õ$ø2ø ø ¸Xê¤¦l*¢¯©eª°u¢§epe¦¥,\¢;'¬£§e¥°ePª° le¥°e´a¸etv{<|5|u}w{|uxq tuGq)"{kks<uof t xCw*Cq<|t w{Ox<w³|Û ZÛ úe¸F¥°3*¥°£§³ì"¸ a ¸°³ , µ« í¥,¥ ³ a ¸&Ûû2û2ú-¸e2ª,·¥°e´¢±eª,l¥,¬£ªw ¦¦"ª°$e¥°e´e ©eª°*¢;¦"·¥°1l l¬`£* l £¥,e´u ±pe±p£pp ¦*.
¸ "{|uxql  {|(t rtq"|u}wvt~-wuxr<wd w|w<q{zrzspx³ 2¸

F ¢¥°e´ i¦*³+5¸6 Õ$ø ø2ø-¸9µ¶b­  ¦3¥w$b· *$´2¥,´$5£ªw ¦¦3¥,®¦$azl§E'·2*l®e3¥,´de ©eª°*¢¸¡{kUrU¢$£¤¥s¦|u^§w{|uxq"¨ tuxq"5.u©¢ª"u«oq-rzstvuxw y w<q{|u^tvu ~¬e:¸kÕ2Õ$ö eÕ$ö2-ø ¸t2l´i$ç ±p¡¢;$e}³ p$f ­e$a£¥w¦l£*³}(¸a ±a$ª,l¥°*¥ ³(¸°³V§e*3l¥ ³Z¸7(¸,³Vl2¢³7¸°³ , A2 §¦32:³
 ¸63 Ûû2û û- ¸p ±ees l¶·2£l «¢; £§e¥°e£ªw ¦¦l¥®¦« ¦q p¬¤eª°¥, Fb·¥,¥w¦e$e¸{kUrU¢!® t~Usvsª" y  tv{¯<"{|uxwO°dw<¨rtwuxr<w"{kks}zs¸
A222£§e¥°¢¦³pì"¸7 Û û û2û- ¸  í¥,e´ªw$´ *¬`¦£* ª, ¦3·¢ ª° le¥°e´e £l¥w£*$ª ¸e±"²q"uxr<w³tvu´ªw{|uGwGowvs}©±¸
ç"2§b·¥ ³x ¸°³ , A 2§e:³ a ¸Gb ¸ Ûû2û µ ¸.æE es*¦.¡2k¡ '¬
l±l(¦l±e©¦l.¦l*ª°£¥,2:¸e {|(t rtq"xu}wvt~ wuxr<w ³&¶*¤ ¸**íZ³3F ¸°³mì¶§e¥, ¦l¦l ³µ ¸°³ , .b  £í *¢;$:³ª¯¸1 Õ$ø2ø2Õ- ¸^q~ w<± nt·- {kwªU±wtvu ~nquG±%¯<U| tvu ~  ì8£§ee¥w£* ª-q *¬s l#)¶ ¬Ãì#¶ ¬ÃÕ ø øiÕb¬`ôiú ¸±ee³35+ ¸°³ , + $¢ il³;µ ¸33 Ûû û2Ö- ¸¸V ¢a*l¥,l¥° $¢2e´te¨« íp¦"¥,¢e'·  ¦£2¢¢¥,3*ms*l¡ ¢;$£* ¸
 ±²quxr<w¹tvu»ºw{kq"5u©"{|ªq"(t"u{kUrzw<|tvu ~x<¨wn¸¼ ¥zB ³ª¸ª Û û û û  ¸½­ $l±eë¦l*ª°£l¥°  ¡2%¦3¢©eª°¦¸ d¾!k d%¡ :¸ö $µ ûp ö õ$ô! ¸+ª°$3 ³A ¸3 Ûû2û û- ¸¤+  © ©e¥°ª,¥w¦¥°£V ±ele±p¦5¡ ¤¦l±ees l ·  £¬l2¯¢;2£§e¥,¦$d£ ¢ l¥w¦l X%*´2±eª° l¥¿*B  Ôª,¥°í ª,¥,¬§epP¢§epe¦¸À±"²q"uxr<w*tvu y q{(~ woq{(~tvu;Cqz|tv¨w{<%e:¸ÖÛ|}bµ ô  ¸+l'·22¦3³V­ ¸°³ , F ¢¥°e´ i¦*³5+ ¸  Õ$ø2ø ö  ¸ì8lu¥°p±£l¥° ¡ ¯e © ©e¥,ª°¥,­2¬`©2¦3 $eí¥°e´2¦¸Roqr<s!tvuxw y w<q{|u^tvu ~ ³
Á 2 ¸e £§$e¥° ³e ¸d Õ$ø2øeÛU ¸dì¶§e©sii¦¥,´E ee2 £§PlÔ¢;'¬£§e¥°eª, $e¥°e´}.\'· *·¥,¨¸_|uÀo!dÀf{kks<uº"u}tvuGwzq"{ ®|(tv5q tuQquG±fCqz|t rzq"(t"u ¸ÂÃÄCÄ¿Å¿ÆÇÈ6ÉeÊ^ËCÉÌ7Ç"È}Í"Î¥ÏÐ<ÑlÉ.Ê^ËCÉÒÇÐkÃ^Ç"È}Ç}É.ÓOËCÉÔÖÕÄ¿Ã×ÉØËÙ(ÚÛÛ-Û-Ü ËÞÝßUÇàkÃ}Ðkß©á^Ï!Ï-âãà©ä)ØÖ×ªßàzÇ"å Ä¿ß©ÇÐkÈ}ÅCÈÍQÇÄCÍ-ÏÐzÅàzæ×àkæ^ÇàdÅ¿×$ç}ÐkÏÆ-ß©â×ªÏ!ÑßÄ7ÐzÏá}Ã}âàzÈß©âzâ©Ë.è ékê©ë<ì<ì<íîvï!ðñnêãòOóvô}ì
õGìöìï^ó§ì<ìï}ó¥ôø÷ï}óìé|ïGùó îêïGùúªû.êï©òìékìïxë<ì_ê"ï«üùë<ô!îvïxìý ìzù"é|ï}îvï!ð"Ëþ ÏÄ¿çxßÐkà©ÉlÿË Ë Ù  ÚÜ Ë Â àzÇ  ßUÑfÍß©ÈßÐ<Ç"Ä¿Å UÇàzÅCÏ^Ë ìékùúìó eêé ñ<É É Ú Ú  Ë

Getting the Most Out of Ensemble Selection∗
Rich Caruana, Art Munson, Alexandru Niculescu-Mizil
Department of Computer Science
Cornell University
Technical Report 2006-2045{caruana, mmunson, alexn} @cs.cornell.edu
Abstract

An ensemble is a collection of classifiers whose predictions are combined with the goal of achieving better performance than the constituent classifiers. A large body of research now exists showing that ensemble learning often increases performance (e.g. bagging [3], boosting [21], stacking [25]).
Recently, ensemble selection [7] was proposed as atechnique for building ensembles from large collections ofdiverse classifiers. Ensemble selection employs greedy forward selection to select models to add to the ensemble, amethod categorized in the literature as overproduce andchoose [20]. Compared to previous work, ensemble selection uses many more classifiers, allows optimizing to arbitrary performance metrics, and includes refinements toprevent overfitting to the ensemble’s training data—a larger
problem when selecting from more classifiers.
In this paper we analyze four previously unexplored aspects of ensemble selection. First, we evaluate ensemble

selection’s performance when all the models are calibrated
to place their predictions on a canonical scale. Making
calibrated models available to ensemble selection provides
significant improvement on probability measures such as
squared error and cross-entropy. It appears, however, thatcalibration does not make ensemble selection itself more
effective; most of the benefit results from improvements inthe base-level models and not from better ensemble building.
Second, we explore how ensemble selection behaves
with varying amounts of training data available for the critical forward selection step. Despite previous refinements
to avoid overfitting the data used for ensemble hillclimbing [7], our experiments show that ensemble selection isstill prone to overfitting when the hillclimb set is small.
This is especially true if their model bagging procedure is
not used. Surprisingly, although ensemble selection overfits with small data, reliably picking a single good model
is even harder—making ensemble selection more valuable.
With enough hillclimbing data (around 5k points), overfitting becomes negligible. Motivated by these results, we
present a method for embedding cross-validation inside ensemble selection to maximize the amount of hillclimbing
data.1 Cross-validation boosts the performance of ensemble
selection, doubling its previously reported benefit. Whileadding cross-validation to ensemble selection is computationally expensive, it is valuable for domains that require
the best possible performance, and for domains in whichlabeled data is scarce.
Ensemble selection’s ability to optimize to any performance metric is an attractive capability of the method thatis particularly useful in domains which use non-traditional
performance measures such as natural language processing [14]. Because of this, the third aspect we investigate
is what benefit, if any, comes from being able to optimize to any metric. Our experiments reinforce the intuition
that it is best to optimize to the target performance metric;∗ This technical report is an expanded version of a paper accepted at the2006 International Conference on Data Mining.

1 This is different from wrapping cross-validation around ensemble selection, which would not increase the data available for hillclimbing.

We investigate four previously unexplored aspects of ensemble selection, a procedure for building ensembles ofclassifiers. First we test whether adjusting model predictions to put them on a canonical scale makes the ensembles
more effective. Second, we explore the performance of ensemble selection when different amounts of data are available for ensemble hillclimbing. Third, we quantify the benefit of ensemble selection’s ability to optimize to arbitrary
metrics. Fourth, we study the performance impact of pruning the number of models available for ensemble selection.
Based on our results we present improved ensemble selection methods that double the benefit of the original method.

1 Introduction

however, they also show that minimizing squared error or
cross-entropy frequently yields ensembles with competitive
performance—seemingly regardless of the metric.
Fourth, we test ensemble selection’s performance when
only the best X% models are available for selection. Theseexperiments confirm our intuition that the potential for overfitting increases with more models. Using only the top 1020% of the models yields performance better than or equivalent to ensemble selection without this model pruning.

high variance. Ensemble selection uses bagging over models to reduce this variance. Multiple ensembles are built
from random subsets of the models, and then averaged together. This is analogous to the feature bagging methods
proposed by Bay [1] and Bryll et al. [5] and used in random
forests [4].
Another technique used in the original paper is allowing
models to be added to the ensemble more than once. This
provides two benefits. First, models added multiple times
get more weight in the ensemble average. Second, whenmodels are added without replacement, ensemble performance deteriorates quickly after the best models have been
exhausted because poorer models must then be added. This
makes deciding when to stop adding models to the ensemble
critical because overshooting the optimal stopping point canyield much worse performance. Selection with replacement
allows selection to continue adding copies of good models instead of being forced to add inferior models. This, inturn, makes deciding when to stop adding models far lesscritical. All of the experiments in this paper use these threetechniques.

2 BackgroundIn this section we briefly review the ensemble selection
procedure first proposed by Caruana et al. [7]. Ensemble selection is an overproduce and select ensemble method carried to an extreme where thousands of models are trainedusing many different learning methods, and overfitting is
moderated by applying several techniques.
In ensemble selection, models are trained using as many
learning methods and control parameters as can be applied
to the problem. Little or no attempt is made to optimize
the performance of the individual models; all models, nomatter what their performance, are added to the model library for the problem. The expectation is that some of themodels will yield good performance on the problem, either
in isolation or in combination with other models, for any
reasonable performance metric.
Once the model library is collected, an ensemble is built
by selecting from the library the subset of models thatyield the best performance on the target optimization metric. Models are selected for inclusion in the ensemble using
greedy forward stepwise model selection. The performanceof adding a potential model to the ensemble is estimated using a hillclimbing set containing data not used to train themodels. At each step ensemble selection adds to the ensemble the model in the library that maximizes the performanceof the ensemble to this held-aside hillclimbing data.
When there are thousands of models to select from, thechances of overfitting increase dramatically. Caruana et al.
describe two methods to combat overfitting. The first controls how ensembles are initialized. The second performs
model bagging—analogous to feature bagging [1, 5]—to reduce the variance of the selection process.
Ensemble Initialization: Instead of starting with anempty ensemble, Caruana et al. suggest initializing ensembles with the N models that have the best uni-model performance on the hillclimb set and performance metric.
Bagged Ensemble Selection: It is well known that feature subset selection (e.g. forward stepwise feature selection) is unstable when there are many relevant features [2].
Ensemble selection is like feature selection, where models
are features and model subsets are found by forward stepwise selection. Because of this, ensemble selection also has

3 Methodology
We use all of the learning methods and data sets used by
Caruana et al. [7], and all of the performance metrics except
CAL (a probability calibration metric) and SAR (a metricthat combines accuracy, squared error, and ROC area). In
addition, we also train models with logistic regression (LOGREG), naı̈ve bayes (NB), and random forests (RF) [4], andexperiment with four additional data sets: MG, CALHOUS,
COD, and BACT. All of the data sets are binary classification problems. The learning methods and data sets aredescribed in Appendix A and B, respectively. The performance metrics we study are described in the following
subsection.

3.1 Performance Metrics
The eight performance metrics we use can be divided
into three groups: threshold metrics, ordering/rank metrics
and probability metrics.
The threshold metrics are accuracy (ACC), F-score(FSC) and lift (LFT). For thresholded metrics, it is not important how close a prediction is to a threshold, only if it isabove or below threshold. See Giudici [10] for a descriptionof Lift Curves. Usually ACC and FSC have a fixed threshold (we use 0.5). For lift, often a fixed percent, p, of cases
are predicted as positive and the rest as negative (we use
p = 25%).
The ordering/rank metrics depend only on the orderingof the cases, not the actual predicted values. As long asordering is preserved, it makes no difference if predicted
2

Table 1. Performance with and without model calibration. The best score in each column is bolded.
ACC FSCLFT ROC APR BEP RMS MXE MEAN
ES-BOTH
0.920 0.888 0.967 0.982 0.972 0.964 0.932 0.944
0.946
ES-PREV
0.922 0.893 0.967 0.981 0.966 0.965 0.919 0.932
0.943
ES-NOCAL
0.919 0.897 0.967 0.982 0.970 0.965 0.912 0.925
0.942
ES-CAL
0.912 0.847 0.969 0.981 0.969 0.966 0.935 0.940
0.940
BAYESAVG-BOTH
0.893 0.814 0.964 0.978 0.963 0.956 0.918 0.934
0.928
BAYESAVG-CAL
0.889 0.820 0.962 0.977 0.960 0.955 0.912 0.925
0.925
MODSEL-BOTH
0.871 0.861 0.939 0.973 0.948 0.938 0.901 0.916
0.918
BAYESAVG-PREV
0.881 0.789 0.956 0.970 0.956 0.947 0.893 0.911
0.913
MODSEL-PREV
0.872 0.860 0.939 0.973 0.948 0.938 0.879 0.892
0.913
MODSEL-CAL
0.870 0.819 0.943 0.973 0.948 0.940 0.892 0.910
0.912
MODSEL-NOCAL
0.871 0.858 0.939 0.973 0.948 0.938 0.861 0.871
0.907
BAYESAVG-NOCAL 0.875 0.784 0.955 0.968 0.953 0.941 0.874 0.892
0.905
values fall between 0 and 1 or 0.89 and 0.90. These metrics
measure how well the positive cases are ordered before negative cases and can be viewed as a summary of model performance across all possible thresholds. The rank metrics
we use are area under the ROC curve (ROC), average precision (APR), and precision/recall break even point (BEP).
See Provost and Fawcett [19] for a discussion of ROC froma machine learning perspective.
The probability metrics are minimized (in expectation)
when the predicted value for each case coincides with thetrue conditional probability of that case being positive class.
The probability metrics are squared error (RMS) and crossentropy (MXE).

0.14 from a neural net does not necessarily mean the samething as a prediction of 0.14 from a boosted tree or SVM.
Predictions from neural nets often are well-calibrated posterior probabilities, but predictions from SVMs are just normalized distances to the decision surface. Averaging predictions from models that are not on commensurate scales
may hurt ensemble performance.
In this section we evaluate the performance of ensemble selection after “translating” all model predictions to thecommon “language” of well-calibrated posterior probabilities. Learning algorithms such as boosted trees and stumps,
SVMs, or naı̈ve bayes have poorly calibrated predictions[15]. A number of methods have been proposed for mapping predictions to posterior probabilities. In this paper weadopt the method Platt developed for SVMs [18], but which
also works well for other learning algorithms [15]. Platt’s
method transforms predictions by passing them through asigmoid whose parameters are learned on an independentcalibration set. In this paper, the ensemble selection hillclimb set is used for calibration as well.
Table 1 shows the performance of ensemble selection(ES), model selection (MODSEL),2 and Bayesian model
averaging (BAYESAVG) [8], with and without calibrated
models. Results are shown for four different model libraries: 1) only uncalibrated models (NOCAL), 2) only calibrated models (CAL), 3) both calibrated and uncalibrated
models (BOTH), and 4) only SVMs are calibrated, to mimicprior experiments [7] (PREV). Each entry is the average offive folds on each of the eleven problems. The last column
shows the mean performance over all eight metrics. Rows
are sorted by mean performance.
Comparing results for ensemble selection with and without calibration (ES-CAL and ES-NOCAL), we see that calibrating models improves RMS and MXE (significant at .05)
but hurts FSC. There is little difference for LFT, ROC, APR

3.2 Comparing Across Performance Metrics
To permit averaging across metrics and problems, performances must be placed on comparable scales. Following
Caruana et al. [7] we scale performance for each problem
and metric from 0 to 1, where 0 is baseline performanceand 1 is the best performance achieved by any model or ensemble. We use the following baseline model: predict p forevery case, where p is the percent of positives in the data.
One disadvantage of normalized scores is that recoveringa raw performance requires knowing what performances define the top and bottom of the scale, and as new best models
are found the top of the scale may change. Note that thenormalized scores presented here differ from those reported
in Caruana et al. [7] because we are finding better models
that shift the top of the scales. The numbers defining thenormalized scales can be found in Appendix C.

4 Ensembles of Calibrated Models
Models trained by different learning algorithms do notnecessarily “speak the same language”. A prediction of

2 Model selection chooses the best single model using the hillclimb set.

3

and BEP. For model selection we see the same trends: calibrated models yield better RMS and MXE and worse FSC.
The magnitudes of the differences suggest that most if not
all of the improvement in RMS and MXE for ensemble selection with calibrated models is due to having better models in the library rather than from ensemble selection taking
advantage of the common scale of the calibrated models.
We are not sure why calibration makes FSC performance
worse for both MODSEL and ES, but again suspect that thedifferences between ES-CAL and ES-NOCAL are due todifferences in the performance of the base-level models.
Having both calibrated and uncalibrated models in thelibrary (ES-BOTH and MODSEL-BOTH) gives the best ofboth worlds: it alleviates the problem with FSC while retaining the RMS and MXE improvements. For the rest of
the experiments in this paper we use libraries containing
both calibrated and uncalibrated models.
Unlike with ensemble selection, using calibrated models
for Bayesian model averaging improves performance on all
metrics, not just RMS and MXE (significant at .05). With
calibrated models, Bayesian averaging outperforms model
selection but is still not as good as ensemble selection.

selection is hurt less by a small hillclimbing set than model
selection, suggesting that it is less prone to overfitting thanmodel selection. Because of this, the benefit of ensemble
selection over the best models appears to be strongest whentraining data is scarce, a regime [7] did not examine. (Theyused 5k training data with 1k points held aside for ensemble
stepwise selection.) As the size of the hillclimbing sets goesfrom 1k to 10k, ensemble selection maintains its edge overmodel selection.
With small hillclimb sets, using bagging with ensemble
selection is crucial to getting good performance; without
it, mean performance using a 100 point hillclimb set drops
from 0.888 to 0.817. In contrast, bagging provides very
little if any benefit when a very large hillclimb set is used
(more than 5000 points with our data sets).

6 Cross-Validated Ensemble Selection
It is clear from the results in Section 5 that the larger the
hillclimb set, the better ensemble selection’s performance
will be. To maximize the amount of available data, we apply
cross-validation to ensemble selection. Simply wrapping
cross-validation around ensemble selection, however, will
not help because the algorithm will still have just a fractionof the training data available for hillclimbing. Instead, we
embed cross-validation within ensemble selection so that allof the training data can be used for the critical ensemble
hillclimbing step. Conceptually, the procedure makes crossvalidated models, then runs ensemble selection the usual
way on a library of cross-validated base-level models.
A cross-validated model is created by training a modelfor each fold with the same model parameters. If there are
5 folds, there will be 5 individual models (each trained on
4000 points) that are ‘siblings’; these siblings should only
differ based on variance due to their different training samples. To make a prediction for a test point, a cross-validated
model simply averages the predictions made by each of thesibling models. The prediction for a training point (that subsequently will be used for ensemble hillclimbing), however,
only comes from the individual model that did not see thepoint during training. In essence, the cross-validated model
delegates the prediction responsibility for a point that will
be used for hillclimbing to the one sibling model that is notbiased for that point.
Selecting a cross-validated model, whether during model
selection or ensemble selection, means choosing all of thesibling models as a unit. If 5-fold cross-validation is used,
selection chooses groups containing 5 sibling models at atime. In this case, when selection adds a cross-validated
model to a growing ensemble, it really adds 5 different models of the same model type to the ensemble, each of which
receives the same weight in the ensemble average.
We ran ensemble selection with 5-fold cross-validation;

5 Analysis of Training Size
The original ensemble selection paper demonstrated themethod’s effectiveness using relatively small hillclimbing
sets containing 1000 data points. Since the data used forhillclimbing is data taken away from training the individual models, keeping the hillclimb set small is important.
Smaller hillclimb sets, however, are easier to overfit to, particularly when there are many models from which to select.
To explore ensemble selection’s sensitivity to the size ofthe hillclimb set, we ran ensemble selection with hillclimb
sets containing 100, 250, 500, 1000, 2500, 5000, and 10000
data points. In each run we randomly selected the points forthe hillclimb set and used the remainder for the test set. The
hyperspectral and medis data sets contained too few points
to leave sufficient test sets when using a 10K hillclimbing
set and were omitted. Due to time constraints and the costof generating the learning curves, we only used one random
sample at each size and did not repeat the experiment.
Figure 1 shows learning curves for our eight performance measures and their mean. Each graph is an averageover 9 problems. The x-axis uses a logscale to better showwhat happens with small hillclimbing sets. Normalized performance scores are plotted on the y-axis. For comparison,
the graphs include the performance achieved by picking thesingle best model (MODSEL).
Unsurprisingly, the performance achieved with both ensemble selection and model selection using only 100 points
for hillclimbing is quite bad. As data increases, both methods do better as they overfit less. Interestingly, ensemble4

baggingno bagging
modsel10000

hillclimb data size

baggingno bagging
modsel1000

hillclimb data size

baggingno bagging
modsel1000

normalized performance

normalized performance

hillclimb data size

10000

0.9
0.8
0.7
baggingno bagging
modsel

0.5
0.4
100

1000
hillclimb data size

baggingno bagging
modsel
1000

10000

MEAN

1

0.6

10000

hillclimb data size

MXE

baggingno bagging
modsel
1000

10000

0.98
0.96
0.94
0.92
0.9
0.88
0.86
0.84
0.82
100

hillclimb data size

RMS
1
0.95
0.9
0.85
0.8
0.75
0.7
0.65
0.6
0.55
100

1000
BEP

normalized performance

normalized performance

normalized performance
10000

1
0.98
0.96
0.94
0.92
0.9
0.88
0.86
0.84
0.82
100

baggingno bagging
modselhillclimb data size

APRbaggingno bagging
modsel
1000

10000

1
0.98
0.96
0.94
0.92
0.9
0.88
0.86
0.84
100

hillclimb data size

ROC
1
0.98
0.96
0.94
0.92
0.9
0.88
0.86
0.84
100

LFT

normalized performance

1000

1
0.95
0.9
0.85
0.8
0.75
0.7
0.65
0.6
0.55
100

normalized performance

FSC
normalized performance

normalized performance

ACC
1
0.95
0.9
0.85
0.8
0.75
0.7
0.65
0.6
0.55
0.5
100

10000

1
0.95
0.9
0.85
0.8
0.75
0.7
100

baggingno bagging
modsel1000

10000

hillclimb data size

Figure 1. Learning curves for ensemble selection with and without bagging, and for picking the best
single model (modsel).
Table 2. Performance with and without cross-validation for ensemble selection and model selection.
ACC FSCLFT ROC APR BEP RMS MXE MEAN
ES-BOTH-CV
0.935 0.926 0.982 0.996 0.992 0.977 0.984 0.989
0.973
MODSEL-BOTH-CV 0.907 0.923 0.971 0.985 0.968 0.963 0.945 0.961
0.953
ES-BOTH
0.920 0.888 0.967 0.982 0.972 0.964 0.932 0.944
0.946
MODSEL-BOTH
0.871 0.861 0.939 0.973 0.948 0.938 0.901 0.916
0.918
this is analogous to normal ensemble selection with a 5000
point hillclimb set. Table 2 shows the results averaged over
all the problems. Not only does cross-validation greatly improve ensemble selection performance, it also provides thesame benefit to model selection. Five-fold cross-validated
model selection actually outperforms non-cross-validated
ensemble selection by a small but noticeable amount. However, ensemble selection with embedded cross-validation
continues to outperform model selection.

selection achieves a raw accuracy score of 90%, and crossvalidated ensemble selection achieves 95% accuracy, thenthe percent reduction in loss is 50%—the loss has been reduced by half. The MEAN row is the average improvement
for each metric, across datasets. For comparison, the PREV
row is the performance of the original non-cross-validated
ensemble selection method (i.e. no cross-validation andonly SVMs are calibrated).
Embedding cross-validation within ensemble selection
doubles its benefit over simple model selection (from 6.90%
to 12.77%). This is somewhat of an unfair comparison; if across-validated model library is available, it is just as easy
to do cross-validated model selection as it is to do crossvalidated ensemble selection. The last row in Table 3 shows

Table 3 provides a different way to look at the results.
The numbers in the table (except for the last row) are thepercent reduction in loss of cross-validated ensemble selection, relative to non-cross-validated model selection, thebaseline used in Caruana et al. [7]. For example, if model
5

Table 3. Percent loss reduction by dataset.
ADULT
BACT
CALHOUSCOD
COVTYPEHS
LETTER.p2
LETTER.p1
MEDIS
MG
SLAC
MEAN
PREV
MEANcv

ACC
2.77
2.08
7.95
5.73
6.68
13.66
15.21
21.55
2.77
4.45
2.49
7.76
4.96
2.89

FSC
5.89
3.83
9.49
7.46
7.26
16.36
14.50
25.66
-0.05
1.98
3.27
8.70
4.56
3.07

LFT
8.72
16.42
48.00
14.33
12.35
12.32
100.00
0.29
2.08
4.25
13.65
21.13
16.22
10.82

ROC
7.45
4.13
8.69
9.14
11.34
37.53
32.84
69.10
6.33
11.84
6.92
18.67
8.43
9.97

the percent loss reduction of cross-validated ensemble selection compared to cross-validated model selection. Comparing PREV and MEANcv , we see that after embedding
cross-validation, ensemble selection provides slightly lessbenefit over model selection than un-cross-validated ensemble selection did over un-cross validated model selection.
While training five times as many models is computationally expensive, it may be useful for domains where thebest possible performance is needed. Potentially more interesting, in domains where labeled data is scarce, crossvalidated ensemble selection is attractive because a) it does
not require sacrificing part of the training data for hillclimbing, b) it maximizes the size of the hillclimbing set (which
Figure 1 shows is critical when hillclimb data is small), andc) training the cross-validated models is much more feasible
with smaller training data.

APR
6.70
5.49
8.81
10.52
14.99
37.78
33.05
45.29
7.28
12.65
9.62
17.47
6.24
9.37

BEP
7.58
1.76
6.15
7.11
7.64
16.77
15.85
19.25
4.62
6.04
2.73
8.68
5.15
2.84

RMS
2.26
1.42
7.17
2.39
7.80
12.65
17.13
19.59
1.40
2.57
1.66
6.91
3.27
2.54

MXE
4.08
4.15
12.74
3.79
12.92
27.43
29.47
34.58
2.70
6.10
3.33
12.84
6.39
4.22

MEAN
5.68
4.91
13.63
7.56
10.12
21.81
32.26
29.41
3.39
6.23
5.46
12.77
6.90
5.71

Table 4. Performance of ensemble selection
when forced to optimize to one set metric.
ES-BOTH-CV
ES-BOTH

RMS
0.969
0.935

MXE
0.968
0.936

OPTMETRIC
0.973
0.946

significantly smaller.
The scatter plots in Figure 2 plot the performance of optimizing to RMS against the performance of optimizing to
OPTMETRIC, with one graph per target metric. Again, we
can see that ensemble selection performs somewhat betterwith OPTMETRIC. Always optimizing RMS is frequently
very competitive, especially when performance gets close
to a normalized score of 1. This is why the benefit of direct
metric optimization is so small for cross-validated ensemble selection. These results suggest that optimizing RMS(or MXE) may be a good alternative if the target metric is
too expensive to use for hillclimbing.

7 Direct Metric Optimization
One interesting feature of ensemble selection is its ability to build an ensemble optimized to an arbitrary metric. To
test how much benefit this capability actually provides, we
compare ensemble selection that optimizes the target metric with ensemble selection that optimizes a predetermined
metric regardless of the target metric. For each of the 8
metrics, we train an ensemble that optimizes it and evaluate
the performance on all metrics. Optimizing RMS or MXE
yields the best results.
Table 4 lists the performance of ensemble selection for a)always optimizing to RMS, b) always optimizing to MXE,
and c) optimizing the true target metric (OPTMETRIC).
When cross-validation is not used, there is modest benefit
to optimizing to the target metric. With cross-validation,
however, the benefit from optimizing to the target metric is

8 Model Library Pruning
Including a large number of base level models, with awide variety of parameter settings, in the model libraryhelps ensure that at least some of the models will havegood performance regardless of the metric optimized. At
the same time, increasing the number of available models
also increases the risk of overfitting the hillclimb set. Moreover, some of the models have such poor performance thatthey are unlikely to be useful for any metric one would want
to optimize. Eliminating these models should not hurt performance, and might help.
In this section we investigate ensemble selection’s per6

ACC

FSC

1

LFT

1

ROC

1

1

MEAN
1
0.57

1

0.24

APR

1

0.89

BEP

1

1

0.92

RMS

1

1
MXE

1

1
0.75

0.86

1

0.89

1

0.82

1

0.84

1

1

Figure 2. Scatter plots of ensemble selection performance when RMS is optimized (x-axis) vs when
the target metric is optimized (y-axis). Points above the line indicate better performance by optimizing
to the target metric (e.g. accuracy) than when optimizing RMS. Each point represents a different data
set; circles are averages for a problem over 5 folds, and X’s are performances using cross-validation.
Each metric (and the mean across metrics) is plotted separately for clarity.
The graphs in Figure 3 show the average behavior acrossour 11 data sets. Ensemble selection’s behavior under pruning may in fact vary when each data set is considered individually. Averaging across problems could hide different
peak points. Figure 4 shows RMS performance for each of
the problems.

formance when employing varying levels of library pruning. The pruning works as follows: the models are sortedby their performance on the target metric (with respect tothe hillclimb set), and only the top X% of the models areused for ensemble selection. Note that this pruning is different from work on ensemble pruning [12, 22, 23, 26, 13].
This is a pre-processing method, while ensemble pruning
post-processes an existing ensemble.
Figure 3 shows the effect of pruning for each performance metric, averaged across the 11 data sets and 5
folds using non-cross-validated ensemble selection withand without bagging. For comparison, flat lines illustrate
the performance achieved by model selection (modsel) andnon-pruned ensemble selection (es-both). The legend isshown in the ACC graph.
The figure clearly shows that pruning usually does not
hurt ensemble selection performance, and often improves
it. For ACC, LFT, and BEP pruned ensemble selection(the line with boxes) seems to yield the same performance
as non-pruned ensemble selection . For the other metrics,
pruning yields superior performance. Indeed, when using
more than 50% of the models performance decreases. Interestingly, library pruning reduces the need for bagging,presumably by reducing the potential for overfitting.3

Although performance starts to decline at different pruning levels for the different problems, it is clear that larger
model libraries increase the risk of overfitting the hillclimb
set. Using 100% of the models is never worthwhile. At
best, using the full library can match the performance of using only a small subset. In the worst case, ensemble selection overfits. This is particularly evident for the COD data
set where model selection outperforms ensemble selection
unless pruning is employed.
While further work is needed to develop good heuristics
for automatically choosing an appropriate pruning level fora data set, simply using the top 10–20% models seems to bea good rule of thumb. An open problem is finding a better
pruning method. For example, taking into account model
diversity (see for example [11, 17]) might find better pruned
sets.

3 The bagging line at 100% does not always match the es-both line,

even though these should be equivalent configurations. This is particularly
evident for FSC, the highest variance metric. The sorting performed before
pruning alters ensemble selection’s model sampling, resulting in additionalvariance.

7

baggingno bagging
es-bothmodsel-both
0

20

40

60

80

100

0.905
0.9
0.895
0.89
0.885
0.88
0.875
0.87
0.865
0.86

0

20

% models

40

20

40

60

80

100

0

20

0.97
0.965
0.96
0.955
0.95
0.945

0

20

40

60

80

100

0.93
0.92
0.91
60

80

100

0

20

% models

40

60

80

100

80

100

80

100

0.97
0.965
0.96
0.955
0.95
0.945
0.94
0.935

0

20

40

60

% models
MEAN
normalized performance

normalized performance

0.94

0.965
0.96
0.955
0.95
0.945
0.94
0.935
0.93
0.925
0.92
0.915

40
BEP

normalized performance

normalized performance
normalized performance

0.94
0.935

MXE

0.95

40

0.95
0.945

% models

0.975

RMS

20

0.96
0.955

% models

0.96

0

100

0.98

% models

0.9

80

0.965

APR
normalized performance

0

60

0.97

% models

ROC
0.99
0.988
0.986
0.984
0.982
0.98
0.978
0.976
0.974
0.972

LFT
normalized performance

FSC
normalized performance

normalized performance

ACC
0.925
0.92
0.915
0.91
0.905
0.9
0.895
0.89
0.885
0.88
0.875
0.87

60

80

100

0.955
0.95
0.945
0.94
0.935
0.93
0.925
0.92
0.915

% models

0

20

40

60

% modelsFigure 3. Pruned ensemble selection performance.

9 Discussion

ing the predictions of the sibling models, we use only thepredictions of one of the siblings. Using this procedure weconstruct five ensemble models, one for each fold, and report their mean performance. This provides a measure of
the benefit due to the increase in the size of the hillclimb set(from cross-validation) while eliminating the bagging-like
effect due to sibling model averaging.
In the other experiment, we use the smaller hillclimb sets
used by un-cross-validated ensemble selection, but we doaverage the predictions of the sibling models. We again
construct five ensemble models, one for each fold, and report their mean performance. This allows us to identify theperformance increase due to the bagging-like effect of averaging the predictions of the sibling models.
Table 5 shows the results of these experiments. Entries inthe table show the improvement provided by using a larger
hillclimb set (ES-HILL) and by averaging the sibling models (ES-AVG) as a percentage of the total benefit of crossvalidated ensemble selection. For example, looking at the
ACC column, increasing the size of the hillclimb set from1k to 5k yields a benefit equal to 32.9% of the total benefit
provided by cross-validated ensemble selection, and aver-In this section we further analyze the benefit of embedding cross validation within ensemble selection and also
briefly describe other work we are doing to make ensemble selection models smaller and faster.

9.1 Benefits of Cross-Validation
The results in Section 6 show that embedding crossvalidation within ensemble selection significantly increases
the performance of ensemble selection. There are two factors that could explain this increase in performance. First,
the bigger hillclimbing set could make selecting models to
add to the ensemble more reliable and thus make overfitting harder. Second, averaging the predictions of the siblingmodels could provide a bagging-like effect that improves
the performance of the base-level models. To tease apartthe benefit due to each of these factors we perform two additional experiments.
In one experiment, we use the same hillclimbing set as
cross-validated ensemble selection, but instead of averag8

0

20

40

60

80

100

0.97
0.96
0.95
0.94
0.93
0.92
0.91
0.9
0.89
0.88
0.87

0

20

40

% models

0.92
0.91
0.9
0.89
0.88
40

60

80

100

0.945
0.94
0.935
0.93
0.925
0.92
0.915
0.91
0.905

0

20

% models

20

40

60

40

60

80

100

80

100

0.96

0.95
20

40

60

80

100

0

20

40

normalized performance

0.91
0.9
0.89
0.88
0.87
40

60

80

100

0.94
0.92
0.9
0.88
0.86
0.84
0.82
0.8
0.78
0.76

0

20

40

60

80

100

SLAC: RMS

0.92

20

60

% models

MG: RMSnormalized performance

0.98
0.97
0.96
0.95
0.94
0.93
0.92
0.91
0.9
0.89
0.88

% models

0100

MEDIS: RMS

0.955

0

80

% models

0.965

0.945

60

HS: RMSnormalized performance

normalized performance40% models

0.97% models

0.86

20

LETTER.p2: RMS
normalized performance00% models

LETTER.p1: RMS
0.96
0.95
0.94
0.93
0.92
0.91
0.9
0.89
0.88

100

normalized performance

0.93

20

80

baggingno bagging
es-bothmodsel-both

COVTYPE: RMS
normalized performance

normalized performance

COD: RMS

0

60

0.985
0.98
0.975
0.97
0.965
0.96
0.955
0.95
0.945
0.94
0.935
0.93

% models

0.94

0.87

CALHOUS: RMS
normalized performance

BACT: RMS
normalized performance

normalized performance

ADULT: RMS
0.98
0.975
0.97
0.965
0.96
0.955
0.95
0.945
0.94

80

100

% models

0.97
0.96
0.95
0.94
0.93
0.92
0.91
0.9

0

20

40

60

80

100

% modelsFigure 4. RMS performance for pruned ensemble selection.

aging the sibling models yields a benefit equal to 80.5%.
The third row in the table is the sum of the first two rows.
If the sum is lower than 100% the effects from ES-HILL
and ES-AVG are super-additive, i.e. combining the two effects provides more benefit than the sum of the individual
improvements. If the sum is higher than 100% then the two
effects are sub-additive. For ACC, the sum is 113.4%, indicating that the effects of these two factors are sub-additive:
the total performance is slightly less than would be expected
if the factors were independent. Except for the high variance metrics, FSC and ACC, the sums are close to 100%,
indicating that the two effects are nearly independent.
The learning curves in Figure 1 suggest that increas-

ing the size of the hillclimb set from 1k to 5k would explain almost all of the benefit of cross-validation. These
results, however, show that on average across the eight metrics the benefit from ES-HILL and ES-AVG are roughly
equal. About half of the benefit from embedding crossvalidation within ensemble selection appears to result fromthe increase in the size of the hillclimb set, and the otherhalf appears to result from averaging the sibling models.
Increasing the size of the hillclimb set via cross-validation(as opposed to having more data available for hillclimbing)provides less benefit in practice because there is a mismatch
between the base-level models used to make predictions on
the hillclimbing set and the sibling-averaged models that9

Table 5. Breakdown of improvement from cross-validation.
ES-HILL
ES-AVG
SUM

ACC
32.9%
80.5%
113.4%

FSC
37.2%
13.6%
50.8%

LFT
48.0%
54.0%
102.0%

ROC
38.8%
59.0%
97.8%

will be used in the ensemble. In other words ensemble selection is hillclimbing using slightly different models thanthe ones it actually adds to the ensemble.

APR
40.8%
55.7%
96.5%

BEP
19.4%
77.4%
96.8%

RMS
55.1%
46.8%
101.9%

MXE
56.7%
51.8%
108.5%

MEAN
41.1%
54.9%
96.0%

In comparison to model selection, however, ensemble selection seems much more resistant to overfitting when data is
scarce. Further experiments varying the amount of training
data provided to the base-level models are needed to see if
ensemble selection is truly able to outperform model selection by such a significant amount on small data sets.
Counter to our and others’ intuition [9], calibrating models to put all predictions on the same scale before averaging
them did not improve ensemble selection’s effectiveness.
Most of calibration’s improvement comes from the superiorbase-level models.
Our experiments show that directly optimizing to a target metric is better than always optimizing to some predetermined metric. That said, always optimizing to RMS or
MXE was surprisingly competitive. These metrics may begood optimization proxies if the target metric is too expensive to compute repeatedly during hillclimbing.
Finally, pruning the number of available models reduces
the risk of overfitting during hillclimbing while also yielding faster ensemble building. In our experiments pruningrarely hurt performance and frequently improved it.

9.2 Model Compression
While very accurate, the ensembles built by ensemble selection are exceptionally complex. On average, storing thelearned ensemble requires 550 MB, and classifying a single
test case takes about 0.5 seconds. This prohibits their use
in applications where storage space is at a premium (e.g.
PDAs), where test sets are large (e.g. Google), or where
computational power is limited (e.g. hearing aids). In a separate paper we address these issues by using a model compression [6] method to obtain models that perform as wellas the ensembles built by ensemble selection, but which are
faster and more compact.
The main idea behind model compression is to train afast and compact model to approximate the function learned
by a slow, large, but high performing model. Unlike the true
function that is unknown, the function learned by the highperforming model is available and can be used to label large
amounts of synthetic data. A fast, compact and expressivemodel trained on enough synthetic data will not overfit andwill closely approximate the function learned by the original model. This allows a slow, complex model such as amassive ensemble to be compressed into a fast, compact
model with little loss in performance.
In the model compression paper, we use neural networks
to compress ensembles produced by ensemble selection. On
average the compressed models retain more than 90% of the
improvement provided by ensemble selection (over model
selection), while being more than 1000 times smaller and1000 times faster.

Acknowledgments
We thank Lars Backstrom for help with exploring alternative model calibration methods and the anonymous reviewers for helpful comments on paper drafts. This work
was supported by NSF Award 0412930.

References[1] S. D. Bay. Combining nearest neighbor classifiers throughmultiple feature subsets. In ICML, pages 37–45, 1998.
[2] L. Breiman. Heuristics of instability in model selection.
Technical report, Statistics Department, University of California at Berkeley, 1994.
[3] L. Breiman. Bagging predictors. Machine Learning,24(2):123–140, 1996.
[4] L. Breiman. Random forests. Machine Learning, 45(1):5–
32, 2001.
[5] R. K. Bryll, R. Gutierrez-Osuna, and F. K. H. Quek. Attribute bagging: Improving accuracy of classifier ensembles by using random feature subsets. Pattern Recognition,
36(6):1291–1302, 2003.

10 Conclusions
Embedding cross-validation inside ensemble selection
greatly increases its performance. Half of this benefit is due
to having more data for hillclimbing; the other half is due toa bagging effect that results from the way cross-validation
is embedded within ensemble selection. Unsurprisingly, reducing the amount of hillclimbing data hurts performance
because ensemble selection can overfit this data more easily.
10

A Learning Methods[6] C. Bucila, R. Caruana, and A. Niculescu-Mizil. Model compression: Making big, slow models practical. In Proc. of the
12th International Conf. on Knowledge Discovery and Data
Mining (KDD’06), 2006.
[7] R. Caruana, A. Niculescu-Mizil, G. Crew, and A. Ksikes.
Ensemble selection from libraries of models. In ICML,
2004.
[8] P. Domingos. Bayesian averaging of classifiers and the overfitting problem. In ICML, pages 223–230. Morgan Kaufmann, San Francisco, CA, 2000.
[9] R. P. W. Duin. The combining classifier: To train or not totrain? In ICPR (2), pages 765–770, 2002.
[10] P. Giudici. Applied Data Mining. John Wiley and Sons, New
York, 2003.
[11] L. I. Kuncheva and C. J. Whitaker. Measures of diversity inclassifier ensembles and their relationship with the ensemble
accuracy. Machine Learning, 51(2):181–207, 2003.
[12] D. D. Margineantu and T. G. Dietterich. Pruning adaptive
boosting. In ICML, pages 211–218. Morgan Kaufmann,
1997.
[13] G. Martı́nez-Munoz and A. Suárez. Pruning in ordered bagging ensembles. In ICML, pages 609–616, New York, NY,
USA, 2006. ACM Press.
[14] A. Munson, C. Cardie, and R. Caruana. Optimizing toarbitrary NLP metrics using ensemble selection. In HLTEMNLP, pages 539–546, 2005.
[15] A. Niculescu-Mizil and R. Caruana. Predicting good probabilities with supervised learning. In ICML’05, 2005.
[16] C. Perlich, F. Provost, and J. S. Simonoff. Tree induction
vs. logistic regression: A learning-curve analysis. J. Mach.
Learn. Res., 4:211–255, 2003.
[17] A. H. Peterson and T. R. Martinez. Estimating the potential for combining learning models. In Proc. of the ICML
Workshop on Meta-Learning, pages 68–75, 2005.
[18] J. Platt. Probabilistic outputs for support vector machinesand comparison to regularized likelihood methods. In Adv.
in Large Margin Classifiers, 1999.
[19] F. J. Provost and T. Fawcett. Analysis and visualization ofclassifier performance: Comparison under imprecise classand cost distributions. In Knowledge Discovery and Data
Mining, pages 43–48, 1997.
[20] F. Roli, G. Giacinto, and G. Vernazza. Methods for designing multiple classifier systems. In Multiple Classifier Systems, pages 78–87, 2001.
[21] R. Schapire. The boosting approach to machine learning: An
overview. In In MSRI Workshop on Nonlinear Estimation
and Classification, 2001.
[22] W. N. Street and Y.-H. Kim. A streaming ensemble algorithm (SEA) for large-scale classification. In KDD, pages
377–382, 2001.
[23] G. Tsoumakas, L. Angelis, and I. Vlahavas. Selective fusion of heterogeneous classifiers. Intelligent Data Analysis,9(6):511–525, 2005.
[24] I. H. Witten and E. Frank. Data Mining: Practical Machine
Learning Tools and Techniques. Morgan Kaufmann, San
Francisco, second edition, 2005.
[25] D. H. Wolpert. Stacked generalization. Neural Networks,
5:241–259, 1992.
[26] Y. Zhang, S. Burer, and W. N. Street. Ensemble pruning via
semi-definite programming. Journal of Machine Learning
Research, 7:1315–1338, 2006.

In addition to the learning methods used by Caruana
et al. [7] (decision trees, bagged trees, boosted trees andstumps, KNN, neural nets, and SVMs), we use three moremodel types: logistic regression, naı̈ve bayes, and random
forests. These are trained as follows:
Logistic Regression (LOGREG): we train both unregularized and regularized models, varying the ridge parameterby factors of 10 from 10−8 to 104 . Attributes are scaled tomean 0 and standard deviation 1.
Random Forests (RF): we use the Weka implementation [24]. The forests have 1024 trees, and the size of thefeature set to consider at each split is 1, 2, 4, 6, 8, 12, 16 or20.
Naı̈ve Bayes (NB): we use the Weka implementation andtry all three of the Weka options for handling continuous attributes: modeling them as a single normal, modeling themwith kernel estimation, or discretizing them using supervised discretization.
In total, around 2,500 models are trained for each data
set. When calibrated models are included for ensemble selection the number doubles to 5,000.

B

Data Sets

We experiment with 11 binary classification problems. ADULT, COV TYPE, HS, LETTER.P1, LETTER.P2, MEDIS, and SLAC were used by Caruana etal. [7]. The four new data sets we use are BACT, COD,CALHOUS, and MG. COD, BACT, and CALHOUS are
three of the datasets used in Perlich et al. [16]. MG is a
medical data set. See Table 6 for characteristics of the 11
problems.
Table 6. Description of problems
PROBLEM
ADULT
BACT
COD
CALHOUSCOV TYPEHS
LETTER . P 1
LETTER . P 2
MEDIS
MG
SLAC

11

# ATTR
14/104
11/170
15/60
9
54
200
16
16
63
124
59

TRAIN

TEST

4000
4000
4000
4000
4000
4000
4000
4000
4000
4000
4000

35222
34262
14000
14640
25000
4366
14000
14000
8199
12807
25000

% POZ
25%
69%
50%
52%
36%
24%
3%
53%
11%
17%
50%

Table 7. Scales used to compute normalized scores. Each entry shows bottom / top for the scale.
ADULT
BACT
CALHOUSCOD
COVTYPEHS
LETTER.p1
LETTER.p2
MEDIS
MG
SLAC

ACC
0.752 / 0.859
0.692 / 0.780
0.517 / 0.889
0.501 / 0.784
0.639 / 0.859
0.759 / 0.949
0.965 / 0.994
0.533 / 0.968
0.893 / 0.905
0.831 / 0.900
0.501 / 0.726

FSC
0.398 / 0.705
0.818 / 0.855
0.681 / 0.893
0.666 / 0.796
0.531 / 0.804
0.389 / 0.894
0.067 / 0.917
0.696 / 0.970
0.193 / 0.447
0.290 / 0.663
0.667 / 0.751

LFT
1.000 / 2.842
1.000 / 1.345
1.000 / 1.941
1.000 / 1.808
1.000 / 2.487
1.000 / 3.656
1.000 / 4.001
1.000 / 1.887
1.000 / 2.917
1.000 / 3.210
1.000 / 1.727

ROC
0.500 / 0.915
0.500 / 0.794
0.500 / 0.959
0.500 / 0.866
0.500 / 0.926
0.500 / 0.985
0.500 / 0.999
0.500 / 0.996
0.500 / 0.853
0.500 / 0.911
0.500 / 0.813

C Performance Scales
Table 7 lists the performance numbers that determine thenormalized scores. Each entry contains the baseline performance (bottom of the scale) and the best performanceachieved by any model or ensemble (top of the scale).

12APR
0.248 / 0.808
0.692 / 0.891
0.517 / 0.964
0.499 / 0.864
0.362 / 0.879
0.243 / 0.962
0.036 / 0.975
0.534 / 0.997
0.108 / 0.462
0.170 / 0.740
0.501 / 0.816

BEP
0.248 / 0.708
0.692 / 0.824
0.517 / 0.895
0.499 / 0.782
0.361 / 0.805
0.241 / 0.898
0.035 / 0.917
0.533 / 0.970
0.107 / 0.469
0.169 / 0.686
0.501 / 0.727

RMS
0.432 / 0.312
0.462 / 0.398
0.500 / 0.283
0.500 / 0.387
0.480 / 0.320
0.428 / 0.198
0.184 / 0.067
0.499 / 0.157
0.309 / 0.272
0.375 / 0.278
0.500 / 0.420

MXE
0.808 / 0.442
0.891 / 0.697
0.999 / 0.380
1.000 / 0.663
0.944 / 0.478
0.797 / 0.195
0.219 / 0.025
0.997 / 0.125
0.491 / 0.365
0.656 / 0.373
1.000 / 0.755