Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length

Authors: xuezhe  ma
xiaomeng  yang
wenhan  xiong
beidi  chen
lili  yu
hao  zhang
jonathan  may
luke  zettlemoyer
omer  levy
chunting  zhou
a i at meta
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 04:40:20

Abstract: The quadratic complexity and weak length extrapolation of Transformers limits their ability to scale to long sequences, and while sub-quadratic solutions like linear attention and state space models exist, they empirically underperform Transformers in pretraining efficiency and downstream task accuracy. We introduce MEGALODON, an neural architecture for efficient sequence modeling with unlimited context length. MEGALODON inherits the architecture of MEGA (exponential moving average with gated attention), and further introduces multiple technical components to improve its capability and stability, including complex exponential moving average (CEMA), timestep normalization layer, normalized attention mechanism and pre-norm with two-hop residual configuration. In a controlled head-to-head comparison with LLAMA2, MEGALODON achieves better efficiency than Transformer in the scale of 7 billion parameters and 2 trillion training tokens. MEGALODON reaches a training loss of 1.70, landing mid-way between LLAMA2-7B (1.75) and LLAMA2-13B (1.67). This result is robust throughout a wide range of benchmarks, where MEGALODON consistently outperforms Transformers across different tasks, domains, and modalities.

Main paper: 
 introduction: 
 In many real-world applications, such as multi-turn conversation, long-document comprehension, and video generation, large language models (LLMs) must efficiently process long sequential data, understand internal long-range dynamics, and generate coherent output. The Transformer architecture (Vaswani et al., 2017) , despite its remarkable capabilities, faces challenges with quadratic computational complexity and limited inductive bias for length generalization, making it inefficient for long sequence modeling (Wang et al., 2024; Zhou et al., 2024) . Even with recently proposed distributed attention solutions (Li et al., 2023b; Liu et al., 2024) , computing a single training step of a 7B parameter model over a 1M-token sequence is more than 100 times slower than performing the equivalent computation using 256 separate sequences of 4K tokens each.Techniques like efficient attention mechanisms (Tay et al., 2020; Ma et al., 2021) and structured state space models (Gu et al., 2022a; Poli et al., 2023; Gu and Dao, 2023) have been introduced to overcome these limitations, aiming to enhance scalability and performance. However, the practical application of these methods still falls short of Transformers (Tay et al., 2022; Gu and Dao, 2023) . This work introduces an unlimited context model that outperforms the canonical Transformer architecture on real-world language modeling. We introduce MEGALODON, an improved MEGA architecture (Ma et al., 2023) , which harnesses the gated attention mechanism with the classical exponential moving average (EMA) (Hunter, 1986) approach ( §2). To further improve the capability and efficiency of MEGALODON on large-scale longcontext pretraining, we propose multiple novel technical components. First, MEGALODON introduces the complex exponential moving average (CEMA) component, which extends the multi-dimensional damped EMA in MEGA to the complex domain ( §3.1). Then, MEGALODON proposes the timestep normalization layer, which generalizes the group normalization layer (Wu and He, 2018) to autoregressive sequence modeling tasks to allow normalization along the sequential dimension ( §3.2). To improve large-scale pretraining stability, MEGALODON further proposes normalized attention ( §3.3), together with pre-norm with two-hop residual configuration by modifying the widely-adopted preand post-normalization methods ( §3.4). By simply chunking input sequences into fixed blocks, as is done in MEGA-chunk (Ma et al., 2023) , MEGALODON achieves linear computational and memory complexity in both model training and inference.Empirically, we demonstrate the potential of MEGALODON as a general architecture for modeling long sequences, by evaluating its performance across multiple scales of language modeling, as well as downstream domain-specific tasks. Through a direct comparison with LLAMA2, while controlling for data and compute, MEGALODON-7B significantly outperforms the state-of-the-art variant of Transformer used to train LLAMA2-7B (Touvron et al., 2023) on both training perplexity (Figure 1 ) and across downstream benchmarks (Table 1 ). Evaluation on long-context modeling, including perplexity in various context lengths up to 2M and long-context QA tasks in Scrolls (Parisotto et al., 2020) prove MEGALODON's ability to model sequences of unlimited length. Additional experimental results on small/medium-scale benchmarks, including LRA (Tay et al., 2021) , ImageNet (Deng et al., 2009) , Speech Commands (Warden, 2018) , WikiText-103 (Merity et al., 2017) and PG19 (Rae et al., 2019) , demonstrate the robust improvements of MEGALODON across scales and modalities.2 Background: Moving Average Equipped Gated Attention (MEGA)In this section, we setup notations, briefly review the key components in the MEGA architecture (Ma et al., 2023) , and discuss the existing problems in MEGA.Following the notations in MEGA, we use X = {x 1 , x 2 , . . . , x n } ∈ R n×d and Y = {y 1 , y 2 , . . . , y n } ∈ R n×d to denote the input and output sequences with length n, and assume the representations of the input and output sequences have the same dimension d. 
multi-dimensional damped ema: 
 MEGA embeds an EMA component into the calculation of the attention matrix to incorporate inductive biases across the timestep dimension. Concretely, the multi-dimensional damped EMA first expands each dimension of the input sequence X individually into h dimensions via an expansion matrix β ∈ R d×h , then applies damped EMA to the h-dimensional hidden space. Formally, for each dimension j ∈ {1, 2, . . . , d}:u (j) t = β j x t,j h (j) t = α j ⊙ u (j) t + (1 -α j ⊙ δ j ) ⊙ h (j) t-1(1)y t,j = η T j h (j) twhere u (j) t ∈ R h is the expanded h-dimensional vector for the j-th dimension at timestep t. α ∈ (0, 1) d×h , δ ∈ (0, 1) d×h are the decaying and damping factors, respectively. h (j) t ∈ R h is the EMA hidden state for the j-th dimension at timestep t. η ∈ R d×h is the projection matrix to map the h-dimensional hidden state back to 1-dimensional output y t,j ∈ R.In the gated attention mechanism in MEGA, the output from EMA (1) is used to compute the shared representation (Hua et al., 2022) , because it encodes contextual information through EMA. Subsequently, MEGA introduces the reset gate, the update gate , and computes the candidate activation with the update gate and the residual connection. The technical details are provided in Appendix A.To reduce the quadratic complexity in the full attention mechanism, MEGA simply split the sequences of queries, keys and values in (14-16) into chunks of length c. The attention in ( 17) is individually applied to each chunk, yielding linear complexity O(kc 2 ) = O(nc). Technically, the EMA sub-layer in MEGA helps capture local contextual information near each token, mitigating the problem of losing contextual information beyond chunk boundaries in the chunk-wise attention.Despite the impressive successes of MEGA, it still suffers its own problems: i) the performance of MEGA with chunk-wise attention still fails behind the one with full attention, due to the limited expressiveness of the EMA sub-layer in MEGA. ii) for different tasks and/or data types, there are architectural divergences in the final MEGA architectures. For example, different normalization layers, normalization patterns (pre-norm vs. post-norm) and attention functions (f (•) in ( 17)) are applied to different data types (see Ma et al. (2023) for details). iii) There are no empirical evidences showing that MEGA is scalable for large-scale pretraining. 
megalodon: 
 To address the aforementioned problems of MEGA, in this section we describe the novel technical advancements of MEGALODON.As discussed in Ma et al. (2023) , the EMA component can be regarded as a simplified state space model with diagonal state matrix. Directly inspired from Gu et al. (2022b) , as almost all matrices diagonalize over the complex plane, a straight-forward idea to improve EMA capability is to extend to work over the complex number system C. We propose the complex exponential moving average (CEMA), which re-writes Eq. ( 1):EQUATIONwhere α, δ ∈ R d×h are the real number parameters same as in EMA. Different from EMA, η ∈ C d×h in CEMA are complex numbers. θ j ∈ R h , j ∈ {1, 2, . . . , d} are the h arguments. To uniformly space the h arguments over the period 2π, we parameterize θ j as:EQUATIONwhere the learnable parameter ω ∈ R d depicts the d base angles. By decaying the absolute value of each h t , CEMA preserves the decaying structure in kernel weights, which is a key principle to the success of convolutional models on long sequence modeling (Li et al., 2023c) .Despite the impressive performance of Layer Normalization combined with Transformer, it is obvious that layer normalization cannot directly reduce the internal covariate shift along the spatial dimension (a.k.a timestep or sequential dimension) (Ioffe and Szegedy, 2015) . Group Normalization (Wu and He, 2018) normalizes hidden representations both along the timestep dimension and a subset of the feature dimension, which has obtained improvements over Layer Normalization on a range of computer vision tasks. However, it cannot be directly applied to Transformer on auto-regressive sequence modeling, due to the leakage of future information via the mean and variance across the timestep dimension.In MEGALODON, we extend Group Normalization to the auto-regressive case by computing the cumulative mean and variance. Formally, suppose an input sequence X = {x 1 , x 2 , . . . , x n } ∈ R n×d , and k groups along the feature dimension with d g = d/k elements per group. Then, the mean and variance of the first group at timestep t ∈ {1, 2, . . . , n} are:EQUATIONFigure 2 illustrates Layer Normalization and Timestep Normalization. To efficiently and precisely calculate the cumulative mean and variance in each timestep, we provide hardware-friendly implementation on modern hardware (GPU) (see Appendix B.1).Previous studies have investigated the saturation and instability issues in the original scaled dotproduct attention (17) . A number of novel techniques have emerged to modify the scaled dot-product attention, among which normalized attention mechanisms, such as (scaled-) cosine attention (Luo et al., 2018; Liu et al., 2022) and QK-normalization (Henry et al., 2020) , have stood out for the simplicity and effectiveness. L Y O Q t d Z U T 9 L K t t 8 X s G 4 o E l r 0 4 m 0 L x M s A I W 0 G 5 w 0 i v K 4 y W u j O z h X + 7 w 9 b M O G j r j + O N p B + v e N u o B y E x O N T 2 C 7 V n y B u X 2 9 b l d t O l p u U 0 s z d d e l X d V n j W y M M p V Y 1 X t s f 7 i T + k L k t f L 7 1 + 0 m 2 5 7 m O v v u a u X q O r F 9 6 Z E f a i 9 T A 3 Q / W b 0 V d h w I b f 6 s G u i O l 9 Z n u f t f Q + t b 1 0 k E 6 z O t J W 6 p U x d x d 6 a e o 4 X L A 2 T Y f j A o C 5 5 N M R Z u G e u q R 1 Y 8 6 / f u r c T z 3 A b V C d W 0 Q E 7 8 2 k r c n i W T t + J n k O h a f 8 G D d b l Z u t N j d r X u F P a e E b z l 6 + f O m X W T T 0 J k K d T 9 H I y z M h I k w b 1 T r k s Y / h U 9 1 g 8 f D U G Z l j N L V M U j G m e 2 X z f 8 + y c r R R O 9 r 4 Q U c 4 6 T Q E f R I b W 1 H N R + P 1 k F A t l r e + X r 5 c q B Q c n h + H G S a F c d I y U e T M 8 G q j j 8 6 U u X o y 1 T X r a q 3 F l d W 8 v R 9 O o v a 1 U P S m 1 7 n T a + 0 H e z 1 Z 1 h x P r m r q T I A K N e N V V x / b F t O / K e B + 3 b / v 9 r d T r W + A o 1 b X i 6 d Z a Q 6 i W O k 1 V h d 4 r q O F u q o c j u I s K z S t r w y v L y s D p A b J r N t M O r L A W J j P d L E U + P F s s 2 l Q + n E 0 5 A Z v z X W R z A w 1 f + I S h G z v o J k 5 T Q l y o X J d L q I 4 S z W s V h e d Z I l X + k X k Y 8 y K S u M g / Z n y / S D T r E j Q 7 / M 7 h J 7 P 7 Y o W D d o n Z u A y A 2 I C l w m I G b r M c F 7 r r Q C X A u o 0 c p k R M a H L h M S M X W Z M T O Q y E T H v X O Y d G 9 y 9 S 9 1 T p 9 h l 4 r n W c p F 4 k c C 4 x c J 6 + K j O P L O L y 9 6 7 i Z D e M E u / l J 4 q b 1 G T j + o A c r b G S y r f q e s 7 p b t m L p M R k 7 t M z i b x 3 q X e U 6 f C Z Q p i h M s I 5 k 6 6 l K R O E 5 e Z E F O 6 T E n M 1 G W m x D y 4 z A M x j y 7 z y A b 3 w a U + z E 2 5 Z a M A k 3 R W 2 B g o q 1 C Z z e z D D Q V P P X A 5 N q F i L X S b 3 b B k J A u S c k A w i 5 A y I J i F R z k k e M h g I J j F R T k i e M S H E h L O Q q I c E 8 z i o Z w Q z I K h f E f w O w b f E 8 y i o I w J j h m c E J z w E b K 1 5 o u c E c w k X e Y E 5 w x + T z D T c l k Q z I R c C o I F 3 1 e C J R 8 h W x W u 3 5 J g J t 5 y S j B T b v l A M J N t + U j w I 4 M / E G z 1 u h W D e l r W T 4 C F q 9 3 q r D f C a z 2 h w a i v 9 Y w G I 8 H W U x q M D t v P a T B q b D 2 p w U i y 9 a w G I 8 v W 0 x q M N l v P a + Q W n t h g V N o 8 s y 2 3 8 N A G o 9 f m s W 2 5 x O U S Z / J t Z 3 J N L j y W w Y i 4 e T B b r u 1 k r s m F h z M Y U T e P Z 8 s t P J / B q L v 1 h A Y j 8 d Y z G o z O W 0 9 p M G J v P a f B K L 7 9 p A a j + 8 V n N U Z E E Q V 1 w Z K s U Z S s s Y B N 1 g l f p 6 B K N g j e Y P A m w Z s M 3 i J 4 i 8 H b B G 8 z e I f g H Q b v E r z L 4 D 2 C 9 / j A 9 w n f Z + Y H B B 8 w u E d w j 8 G H B B 8 y + I j g I w Y f E 3 z M h 9 I n v M / M T w g + Y f A p w a c M P i P 4 j M H n B J 8 z + I L g C w Z f E n z J R 3 h F + B U z vv g 3 K X L X X L u y u W u O H f t c t e c u 3 G 5 G 8 7 d u l w t / U t e D Z c f Q D 9 d 4 F P t a t 2 5 z F K Y 2 S d d i y U T A 9 0 l l D 7 q M l n h d Y l s q A p m y M A g A 1 p N X Z Q g R M W I L k U Q o X K 4 r I Z C d Y e u O h C h a k P X G o h Q j a E r D E S o s t B 1 B S I R u / 0 7 A 1 E Z o Y s I R K h 4 0 K U D I j F b C Y M k h K Q G S d k K G o R K A l 0 Q I E K F g C 4 D E K F i V u d + R C j n 6 4 y P C H s Q 0 4 k e I U r w Z b U z b F 9 K g 1 A y 1 6 k c E U r h O o E j Q o l b p 2 1 E q E j V u R q R D 2 3 H p v N U B a U f 5 2 O 1 5 / r f W o b l o B K I 1 o Y F 6 Q m M X l t U V O w n g 6 H q Y S 6 I y B I I F a 7 / J V j L U k n S B m i J H h H C v x k m o j B R n f W / 1 N n K t 5 J u P Z X Z j M 9 g p i R r W 6 j Y g F q o 1 i G b 1 k y p 1 L Z Q p S N q o U L D O b 2 z U f o c E 4 s j j q i F s n z H B o + a v G c L N F N a r K c / U z q 0 L V x R t p S o w Y x a q L + c W q i 9 9 9 R C 3 R V 8 t W Z K c P U i z Z T W b A u X e 0 I t 1 F l J L d T Y l F q o r w d q o b Y e q Y W 6 + j C v f i D D r P t Q 3 V q n X N Q b p V q d a B F Z p w D Q + R U h y q s 6 q y J C 2 V T n U k Q o h + o M i g h l T p 0 3 E a F a T i d L R C h J 6 h S J C K V G n R g R o Y S o 0 y E i l A Z 1 E k S E k p 9 O f Y h Q y t M J D x G q 0 3 S W Q 4 S V a D q 5 I U R J T a c 0 R C i V 6 U S G C C U w n b 4 Q o X J M 5 y x E z p n r C w N R i t I J C h F K T D o t I U L p S C c j R C g J 6 R S E y A 1 z f W u g W 7 a L l C Y G P E s k / X F 1 L N / h F V t A e w o o p l e d B P X 0 q m B W 3 J k J a H M 6 n U M q 1 K / E m x D E f g E o r f G a O o 3 w l q Y C F K N I v V S F N M i G U R q i N 3 8 S K 0 S M 6 u t k P h P q f f A Z y E U O B l k 8 / C E 3 g 4 c 5 x u K z 5 k v d V O h f F U 0 i r R z q N 9 r V 5 K Q p O l P B Y k C u W 4 z K T L l h s Q 1 a e L l p Q Y o E u W U x i g W 5 b T G K B r l j M Y o H u W s x i g i 5 Z z G K C b l v M Y o K e W A x i g v Z s 1 i P D f r Q g h Q c 8 s h i F B 7 y 2 G J U / s m + x f r M 4 Y k F K U j k q c U o T O S Z x S h Q 5 L n F q K y T F x a j W J G X F q N o k V c W o 3 i R 1 x a j i J E 3 F m M x I 2 8 t W N V p K O e d w s / H h g 7 t Y 3 D g P I i E 6 w w m c Y Q b D K Z j M t x k M C k k 3 G I w i S T c Z j D p J N x h M E k l 3 G U w q S X c Y z A J J t x n M G k m P G D w A S 1 O 2 G M 4 H a r h I Y N J P O E R g 0 k / 4 T G D S U J h n 8 F 0 z o Y n D D 5 h Q z l l O G k p P G M w y S k 8 Z z A p K r x g M I k q v G Q w e w w O r x h O 2 g q v G U z y C m 8 Y T A 8 E 4 S 2 D j c b U r + + y q t 9 E / Z 5 l w N + z i H W C S V 9 i g 1 C S l 9 g k V K v r h b e p f / m Y C P B 8 T 4 D 0 8 N 4 x D L 2 t Z W 8 A g a 9 w O Y 6 E N 8 0 m 8 R A h b I E n 9 O 8 k W G F O C k 9 9 4 J P F 6 E h 9 L A M P O V a c + j f g 6 s d Q s U 1 3 J I W K H U J J o G K X U N K n 2 C O U v a U R + w S T P M U B o X S o i R 6 h p E 1 x S O g h c 3 x E M E l T H B N K y h R 9 Q k m Y 4 o R Q O t / E K a G k S n F G K H u G F e c E k y j F B a G k S X F J K B 1 1 4 o r Q K + b 4 m m A S p L g h l P Q o b g m 1 c t x K s R Y E / X D h m 9 c x D 1 X B S A V B z 3 0 s U B X j G r V Q r u v U Q p l u U F m r q s R N I v H U 2 6 I W K m m b W q i g H W q h c n a p h Y r Z o x Y K Z Z 9 a K J A D a q E w e t R C Q R y y w a A Q j o h E A R x T C z e + z 0 x x x 0 + I x J 0 + p R b u 8 B m 1 c G P P q Y U b e k E t 3 M h L a u E G X l E L 9 + 2 a W r h f N + z 2 u F G 3 7 I Z V 8 V U V X m r b g G + b N D W Y O l x U I O u P 9 z C a D b z s T S M 5 z i b S w w L I m 2 J + y 6 F w S y S g G s m p j 6 r 7 y 1 o I 2 v B J c Q i 6 g I J G B Q W 6 h I J G D Q W 6 i A K q o i p 0 s 0 b Z 6 z l d R 0 G j k A J d S U G j l A J d S 0 G j m A J d T Q G V U x b e q 2 H 2 A k 6 X V N C o q U A X V d C o q k C X V U B 1 V Y U e 1 i h 7 0 a Y r K 2 i U V q B r K 6 D i y s L 9 G u 5 z + K S G 2 R s 1 X W J B o 8 Y C X W R B o 8 o C X W Z B o 8 4 C X W h B o 9 I C X W p B o 9 Y C X W x B o 9 o C X W 4 B 1 V s W v q l h 9 o Z M l 1 z A a y 5 8 g s A E J I s J e J N 0 C E X 8 q D 5 9 G v r S 9 0 J I o c D c o 9 q R Q L 0 P J i o R u d r N l e l 8 l r + d 3 R X J T D d 0 G l R e I c m j I s I E 6 P S v P 0 I c P O r k p z 8 k U T f B b N n w b b 8 x G f s S H + T d W z i W f W 7 Z n 7 c N J s m G E H 9 s I t q g n o l p z Z t P M / 3 K q t + w a o w 7 i o d Q W d 7 p R j 3 8 u g c e F j I L x r 5 Q H 9 X 6 E 5 n p R y s o n C E 2 P o b N j U 0 9 y K r L 0 w E M w b E z z R a 7 A g k 8 e q y d a a I Y A v N D s W s d + 3 n s B z C v v 8 r p V c D c e + F V 1 + 4 C u / 2 3 5 n X 2 2 2 q O p C f Y h z + 9 J n s 6 5 5 m + 4 R Z V x l a 5 0 T M u 5 v b 1 n E s U E M 7 r F 2 5 N K p A 0 S d W K R p F a P d d M Z C O Z + A 9 k a Y G m H W a N T H 8 J Z d 7 E P f W S x x M 1 / Q / q L U F j c g e 9 O f 8 O 6 q D 3 Z A 8 v / Y K G o B r N G 0 j 8 x y 9 w + 4 u M W Z 4 9 2 Y K N r C R a N X S i u 8 r i U e E n 6 p 3 V e J o V W L M K / 1 F 4 z 3 v f v X q u P g D S X 6 t P U v N F q 8 h R A k J / h f b 8 D u K Y 2 d g 3 p y + 8 d U y F G P e p + u s R g x 4 S 9 T W c K o y N U 2 a t P k f N J q H O n r p O j i Q s a / c i 8 4 Y Z K H f T 6 D 7 K Y R j 5 K 4 0 P m r M i i d W b / v m s 9 9 3 q v I X M U l B c t 4 2 T U 9 3 v V X M / N J k r K m / p p u X Q + + 4 u S k f y s d k 1 9 w v 1 G h n P D l 8 F z B n g i S v 8 E L w o 9 d K s K v I l P K x 4 G + N M q P X J V E E Y j L 1 N f C Z O 4 U v h D b L s f k W F J X v b c 5 y r Q z o r / o Q 6 L 0 I 9 A v z 3 b l l d f c x Q H Z f G M N L v b F o s t W L R T P + 9 w O I c N X W u P h W M Q d 7 5 A 4 y 1 O J s O C v D v n 7 3 9 f K n b / H 8 l n l 5 c v l r p / n n l 9 c n r p W / W q / + P 4 r P O 7 z t / 6 H z V 6 X b + 0 v m m s 9 v p d y 4 6 Q S f t / L 3 z j 8 4 / u / / q / r v 7 f f c / x v T T T 6 o + v + s 4 f 7 r / / R + P Q l N k < / l a t e x i t > x Layer input CEMA output < l a t e x i t s h a 1 _ b a s e 6 4 = " e l 2 k j u N X N N 2 b Q y u J n h 8 m m S f 2 t m s = " > A A A y K 3 i c l V v J k t v I E e W M t 7 G 8 z d h H X x B u K W b s a H W I G o X t 4 / S + s b v Z + 6 L W K E A w C U K N T a g i 2 C 0 G / R e + 2 l / g r / H J D l / 9 H 8 6 q Q i G z 0 K A U V s R I q P e y E r W 8 r E y A m E E e R 0 K + e P G v z z 7 / w Q 9 / 9 O O f f P H T J z / 7 + S 9 + + a s v v / r 1 h c g m R Q D n Q R Z n x d X A F x B H K Z z L S M Z w l R f g J 4 M Y L g d 3 6 4 q / L K E Q U Z a e y Y c c 3 i R + m E a j K P A l Q q 9 v E 1 + O B 6 P Z / d f z t 1 8 u v V h 5 o f 9 4 j y + 6 1 c V S p / r T f / t V 9 8 n t M A s m C a Q y i H 0 h X n d f 5 P L N z C 9 k F M Q w f 3 I 7 E Z D 7 w Z 0 f w m u 8 T P 0 E x J u Z H v P c e 4 b I 0 B t l B f 6 X S k + j v M f M T 4 R 4 S A Z o q c Y o m p w C F 3 H K o 1 g e J I 0 h y N G f 3 8 y i N J 9 I S A M z g t E k 9 m T m q Z X x h l E B g Y w f 8 M I P i g g n 4 Q V j v / A D i e v n 3 O L e z O H J k 2 f e g V / c e Q L 7 4 W o K L x t 5 g Z + b a z W 1 A k Z Q F F E a q p s M o z I S 1 m w U h Z M C 0 G 0 K 0 y B L E j 8 d z m 4 R j G E k 5 7 P Z L S T e N z 2 8 / v 0 c b 9 M 0 C n D F o b B m 6 7 q l D J t 2 R R S O a 2 8 n q t F m J b P c 2 p x l e Z v F I J M y S 6 z R m m 4 9 s q s m 7 l s z / / H Q K 5 O B N R k s c h J Y i 2 C R x d B a D P V t n n k 7 O L 1 Y T d H z P b R X 2 w g j D I q h h 4 u T u D 7 w W o H z 1 9 0 3 6 G U w 8 p a 6 y g l 6 2 d L b Y v Y N R Q L L X p x N o X g e Y I i t o N x h p N c V R k v d m d n C v 9 x i a 2 Y c t P X H 8 U b S j 1 e 8 L d S D k B g c a v u F 2 j P k j c s t 6 3 K r 6 V L T c p r Z m y 6 9 r G 4 r P G v k 4 Z S q x k v b 4 / 3 E H 1 K X p W + X X j 3 q t l z 3 s V f f c l e v 0 N U z 7 9 Q I e 9 F 6 m J u h + s 3 o q z B g w 2 / 1 Y F f E 9 D 6 1 v U 9 b e p / Y X j p I p 1 k d a S v 1 y p i 7 C 7 0 0 d R w u W J u m w 3 E B w F z y 6 Q i z c I 9 d 0 r o x 5 9 8 + d u 6 n H u A 2 q M 4 t I o L 3 Z t L W Z P G s H T + T P I f C U 3 6 M m 8 3 K z W a b m 1 W v 8 K e 0 8 A 1 n z 5 8 / 9 8 s s G n o T o c 6 n a O T l m R A R 5 o 1 q H f L Y x / C p b r B 4 e O q M z D G a W i a p G N O 9 s v m / Z 1 k 5 W q 8 d r X / S E U 4 6 D U G f x M Z W V P P R e D 0 k V I v l r a / n z x c q B Y f n x 2 G G S W G c t E w U O T O 8 2 u i j M 2 W u H k 1 1 1 b p a b X F l N W / v h 5 O o f S 0 U v e l 1 5 v R a / W S v R 8 u a 4 8 l V T Z 0 J U K F m v O r q Y 9 t i + j c F 3 K / 7 9 9 3 + d q r 1 D X D U 6 n r x N C v N Q R Q r v c b q A s 9 1 t F B X l c N R n G W F p v W V 4 f V l Z Y D U I J l 1 m 0 l H F h g L 8 5 m u l g I / n m 0 0 D U o / j o b c 4 K 2 5 L p K Z o e a P X I K Q 7 R 0 0 M 6 c p Q S 5 U r s t F F G e p h t X q o p M s 8 U q / i H y M W V F p H K Q / U 7 7 v Z Z o V C f p 9 e o v Q 0 7 l d 0 a J B + 8 Q M X G Z A T O A y A T F D l x n O a 7 0 V 4 F J A n U Y u M y I m d J m Q m L H L j I m J X C Y i 5 p 3 L v G O D u 3 O p O + o U u 0 w 8 1 1 o u E i 8 S G L d Y W Q 8 f 1 J l n d n H Z e z c R 0 h t m 6 d f S U + U t a v J B H U D O 1 n h J 5 T t 1 f a d 0 1 8 x l M m J y l 8 n Z J N 6 7 1 H v q V L h M Q Y x w G c H c S Z e S 1 G n i M h N i S p c p i Z m 6 z J S Y e 5 e 5 J + b B Z R 7 Y 4 D 6 4 1 I e 5 K b d s F G C S z g o b A 2 U V K r O Z f b q h 4 K k H L s c m V K y F b r M b l o x k Q V I O C G Y R U g Y E s / A o h w Q P G Q w E s 7 g o R w S P + F B C w l l I l G O C W T y U E 4 J Z M J T v C H 7 H 4 D u C W R S U M c E x g x O C E z 5 C t t Z 8 k T O C m a T L n O C c w e 8 J Z l o u C 4 K Z k E t B s O D 7 S r D k I 2 S r w v V b E s z E W 0 4 J Z s o t 7 w l m s i 0 f C H 5 g 8 A e C r V 4 3 Y 1 B P y / o J s H C 1 W 5 3 1 R n i t J z Q Y 9 b W e 0 W A k 2 H p K g 9 F h + z k N R o 2 t J z U Y S b a e 1 W B k 2 X p a g 9 F m 6 3 m N 3 M I T G 4 x K m 2 e 2 5 R Y e 2 m D 0 2 j y 2 L Z e 4 X O J M v u 1 M r s m F x z I Y E T c P Z s u 1 n c w 1 u f B w B i P q 5 v F s u Y X n M x h 1 t 5 7 Q Y C T e e k a D 0 X n r K Q 1 G 7 K 3 n N B j F t 5 / U Y H S / + K z G i C i i o C 5 Y k l W K k l U W s M k a 4 W s U V M k 6 w e s M 3 i B 4 g 8 G b B G 8 y e I v g L Q Z v E 7 z N 4 B 2 C d x i 8 S / A u H / g e 4 X v M f J / g f Q b 3 C O 4 x + I D g A w Y f E n z I 4 C O C j / h Q + o T 3 m f k x w c c M P i H 4 h M G n B J 8 y + I z g M w a f E 3 z O 4 A u C L / g I L w m / Z O Z X B F 8 x + J r g a w b f E H x j 6 u b W k 9 Z V H h j p M a G u M o V r + T F u j X P r L r f O u Q 2 X 2 + D c p s t t c m 7 L 5 b Y 4 t + 1 y 2 z z a t D o Z u c M 7 7 r r c L u f 2 X G 6 P c / s u t 8 + 5 n s v 1 O H f g c g e c O 3 S 5 Q 2 c S R y 5 5 x D v 2 X a 7 P u W O X O + b c i c u d c O 7 U 5 U 6 d w Z y 5 5 B n v e O 5 y 5 5 y 7 c L k L z l 2 6 3 C X n r l z u i n P X L n f N u R u X q 6 V / w a v h 8 g P o p w t 8 q n 1 R d y 6 z F G b 2 S d d i y c R A t w m l j 7 p M V n h d I h u q g h k y M M i A V l M X J Q h R M a J L E U S o H C 6 r o V D d o a s O R K j a 0 L U G I l R j 6 A o D E a o s d F 2 B S M R u / 8 5 A V E b o I g I R K h 5 0 6 Y B I z F b C I A k h q U F S t o I G o Z J A F w S I U C G g y w B E q J j V u R 8 R y v k 6 4 y P C H s R 0 o k e I E n x Z 7 Q z b l 9 I g l M x 1 K k e E U r h O 4 I h Q 4 t Z p G x E q U n W u R u R D 2 7 H p P F V B 6 c f 5 W O 2 5 / r e W Y T m o B K K 1 Y U F 6 A q P X F h U V + 8 l g q H q Y C y K y B E K F 6 3 8 J 1 r J U k r Q B W q J H h P B v h o k o T F R n / S 9 1 t v K t p F t P Z T b j M 5 g p y d o W K j a g F q p 1 y K Y 1 U y q 1 L V T p i F q o 0 H B O 7 2 y U P s f E 4 o g j a q E s 3 7 H B o y b v 2 A L N l B b r 6 c + U D m 0 L V 5 Q t J W o w o x b q L 6 c W a u 8 9 t V B 3 B V + t m R J c v U g z p T X b w u W e U A t 1 V l I L N T a l F u r r n l q o r Q d q o a 4 + z K s f y D D r 3 l e 3 1 i k X 9 U a p V i d a R N Y o A H R + R Y j y q s 6 q i F A 2 1 b k U E c q h O o M i Q p l T 5 0 1 E q J b T y R I R S p I 6 R S J C q V E n R k Q o I e p 0 i A i l Q Z 0 E E a H k p 1 M f I p T y d M J D h O o 0 n e U Q Y S W a T m 4 I U V L T K Q 0 R S m U 6 k S F C C U y n L 0 S o H N M 5 C 5 E z 5 v r c Q J S i d I J C h B K T T k u I U D r S y Q g R S k I 6 B S F y z V z f G O i G 7 S K l i Q H P E k l / X B 3 L t 3 j F F t C e A o r p V S d B P b 0 q m B V 3 a g L a n E 5 n k A r 1 K / E G B L F f A E p r v K p O I 7 y l q Q D F K F I v V S E N s m G U h u j N n 8 Q K E a P 6 O p n P h H o f f A p y k Y N B F g 8 / 5 W Z w P 8 d Y f N J 8 q Z s K / a u i S a S V Q / 1 G u 5 q c N E V n K l g M y D W L U Z k p 1 y 2 2 T g s v N y x I k S A 3 L U a x I L c s R t E g t y 1 G 8 S B 3 L E Y R I X c t R j E h 9 y x G U S H 3 L U Z x I X s W 6 7 F B H 1 i Q g k M e W o z C Q x 5 Z j M o / 2 b d Y n z k 8 t i A F i T y x G I W J P L U Y B Y o 8 s x i V d f L c Y h Q r 8 s J i F C 3 y 0 m I U L / L K Y h Q x 8 t p i L G b k j Q W r O g 3 l v F 3 4 + d j Q o X 0 M D p w H k X C N w S S O c J 3 B d E y G G w w m h Y S b D C a R h F s M J p 2 E 2 w w m q Y Q 7 D C a 1 h L s M J s G E e w w m z Y T 7 D N 6 n x Q l 7 D K d D N T x g M I k n P G Q w 6 S c 8 Y j B J K O w z m M 7 Z 8 J j B x 2 w o J w w n L Y W n D C Y 5 h W c M J k W F 5 w w m U Y U X D G a P w e E l w 0 l b 4 R W D S V 7 h N Y P p g S C 8 Y b D R m P r 1 X V b 1 m 6 j f s w z 4 e x a x R j D p S 6 w T S v I S G 4 R q d T 3 z N v Q v H x M B n u 8 J k B 7 e O 4 a h t 7 n s D S D w F S 7 H k f C m 2 S Q e I o Q t 8 I T + n Q Q r z E n h q Q 9 8 s h g d q Y 9 l 4 D 7 H i l P / B l z 9 G C q 2 6 I 6 k U L F N K A l U 7 B B K + h S 7 h L K 3 N G K P Y J K n 2 C e U D j X R I 5 S 0 K Q 4 I P W C O D w k m a Y o j Q k m Z o k 8 o C V M c E 0 r n m z g h l F Q p T g l l z 7 D i j G A S p T g n l D Q p L g i l o 0 5 c E n r J H F 8 R T I I U 1 4 S S H s U N o V a O m y n W g q A f L n z z O u a + K h i p I O i 5 j w W q Y l y l F s p 1 j V o o 0 3 U q a 1 W V u E E k n n q b 1 E I l b V E L F b R N L V T O D r V Q M b v U Q q H s U Q s F s k 8 t F E a P W i i I A z Y Y F M I h k S i A I 2 r h x v e Z K e 7 4 M Z G 4 0 y f U w h 0 + p R Z u 7 B m 1 c E P P q Y U b e U E t 3 M B L a u G + X V E L 9 + u a 3 R 4 3 6 o b d s C q + q s J L b R v w b Z O m B l O H i w p k / f E e R r O B l 7 1 p J M f Z R H p Y A H l T z G 8 5 F G 6 J B F Q j O f V R d X 9 Z C 0 E b P i o O Q R d Q 0 K i g Q J d Q 0 K i h Q B d R Q F V U h W 7 U K H s 9 p + s o a B R S o C s p a J R S o G s p a B R T o K s p o H L K w r s 1 z F 7 A 6 Z I K G j U V 6 K I K G l U V 6 L I K q K 6 q 0 I M a Z S / a d G U F j d I K d G 0 F V F x Z u F / D f Q 4 f 1 z B 7 o 6 Z L L G j U W K C L L G h U W a D L L G j U W a A L L W h U W q B L L W j U W q C L L W h U W 6 D L L a B 6 y 8 L X N c z e k O m S C 3 j N h U 8 Q m I B k M Q F v k g 6 h i B / U p 0 9 D X / p e C C k U m H t U O x K o 9 8 F E J S J X u 7 k y n c / y t 7 P b I p n p h k 6 D y i s k e V R E m A C d / v V H i I M H n f z 0 h y T q J p g t G 7 7 t N y Z j X + K D v H s L x 7 L P L f v z t s E k 2 R D i j 0 1 E G 9 Q z M a 1 5 8 2 m m X 1 n 1 G 1 a N c U f x E C r L W 9 2 o h 1 / 3 w M N C Z s H Y F + q j W n 8 i M / 1 o B Y U z x M b H s L m x q Q d Z d X k 8 g C E 4 d q b Z Y l c g g U e P t T N N F E N g f i h 2 r W M / j / 0 A 5 v V X O b 0 K m H v P v O r a X W C 3 / + a 8 z n 6 b z Z H 0 B P v w p 9 d k T + Y 8 0 z f c o s r Y K j d 6 x s X c v p 5 z i Q L C e f 3 C r U k F k i a p W t E o U q v n m o l s J B P / n i w t 0 L T D r J H p L 6 H M m 7 j H X v J 4 o q b / Q b 0 l a E x u v z f n 3 0 H t 9 x 7 t 4 Y V f 0 B B U o 3 k D i f / 4 B W 5 / k T H L 0 0 d b s J 6 V R K u G T n S X W T w q / E S 9 s x p P s w J r V u E / C O 9 p 7 / u X T 9 U H Q P p r 9 U l q v m g V O U p A 6 K / Q n t 5 C H D M b + + b 0 m b e G q R D j P l V / P W D Q Q 6 K + h l O F s X H K r N X n q N k k 1 N l T 1 8 m R h G X t X m T e M A P l b h r d R T k M I 3 + l 8 U F z V i S x e t M / n / W + f z F v I b M U F N d t 4 + R U 9 3 v Z 3 A 9 N 5 o r K W 7 p p O f S + v 4 3 S k X x o d s 3 9 Q r 1 G x r P D V w F z C n j i C j 8 E L 0 q 9 N K u K f A n 3 K 9 7 6 O B N q f T J V E A Z j b w O f i V P 4 W n i D L L t b U W H J 3 v Y c 5 e q Q z o o / o M 6 L U I 8 A / 7 1 d V l c f M 1 T H p T G M 9 D u b F k u t W D T T f y + w O E N N n a l P B W O Q t / 4 A Y y 3 O p o M C / L s n b 7 9 c 6 j b / X 4 n H F x c v V 7 p / X H l 1 / Gr p u 7 X q / 6 P 4 o v P b z u 8 6 3 3 S 6 n T 9 1 v u v s d P q d 8 0 7 Q y T p / 7 f y t 8 / f u P 7 r / 7 P 6 7 + x 9 j + v l n V Z / f d J w / 3 f / + D 3 y c U 5 U = < / l a t e x i t >x 0< l a t e x i t s h a 1 _ b a s e 6 4 = " h I x 9 X T h z v l 7 x 2 x j m g V y u O d 5 J 6 w 8 = " > A A A y J n i c l V t Z k 9 v G E V 4 7 l 6 N c d v K Y F 1 R W K j u p 1 Z Y o q 5 I 8 e u + L u 8 u 9 D 1 N W g W A T x C 4 u Y Y b g r l j M b 8 h r 8 g v y a / K W S u U t P y U 9 M x h 0 D x a U K 6 q y h P m + n s Y c X 0 8 3 Q H i Q x 5 G Q r 1 7 9 5 5 N P f / D D H / 3 4 J 5 / 9 9 N n P f v 6 L X / 7 q 8 y 9+ f S m y S R H A R Z D F W X E 9 8 A X E U Q o X M p I x X O c F + M k g h q v B / Y b i r 0 o o R J S l 5 / I x h 7 e J H 6 b R K A p 8 i d B l P / S T x H / 3 + f K r 1 V f 6 j / f 0 o l N d L C 9 V f 3 r v v u g 8 6 w + z Y J J A K o P Y F + L b z q t c v p 3 5 h Y y C G O b P + h M B u R / c + y F 8 i 5 e p n 4 B 4 O 9 P D n X s v E B l 6 o 6 z A / 1 L p a Z T 3 m P m J E I / J A C 0 T X 4 5 F k 1 P g I k 5 5 F C u D p D E E O f r z 2 1 m U 5 h M J a W B G M J r E n s w 8 t S j e M C o g k P E j X v h B E e E k v G D s F 3 4 g c e m c W z y Y O T x 7 9 s I 7 9 I t 7 T 2 A / X E j h Z S M v 8 H N z r a Z W w A i K I k p D d Z N h V E b C m o 2 i c F I A u k 1 h G m S 4 / O l w 1 k c w h p G c z 2 Z 9 S L y v u n j 9 + z n e p m k U 4 I p D Y c 0 2 d E s Z N u 2 K K B z X 3 k 5 V o 8 1 K Z r m 1 O c / y N o t B J m W W W K N 1 3 X p i V 0 3 c t 2 b + 0 6 F X J g N r M l j k J L A W w S K L o b U Y 6 t u 8 8 H Z x e r G a o u d 7 a K + 2 E U Y Y D 0 M P F y d x f e C 1 A u f f d t 6 i l 8 H I W + 4 o J + h l W 2 + L 2 T c U C a x 4 c T a F 4 m W A 0 b W K c o e R X l c Y L X d m Z g v / 0 s f W z D h o 6 4 / j j a Q f r 3 r b q A c h M T j U 9 g u 1 Z 8 g b l 9 v W 5 X b T p a b l N L M 3 X X 5 d 3 V Z 4 1 s j D K V W N 1 7 b H + 4 k / p C 7 L X y + / e d J t p e 5 j r 7 7 m r t 6 g q x f e m R H 2 o v U w N 0 P 1 m 9 F X Y c C G 3 + r B r o j p f W Z 7 n 7 X 0 P r W 9 d J B O s z r S V u u V M X c X e m n q O F y w N k 2 H 4 w K A u e T T E W b h n r q k d W P O v 3 7 q 3 E 8 9 w G 1 Q n V t E B O / N p K 3 J 4 l k 7 f i Z 5 D o W n / B g 3 W 5 W b r T Y 3 a 1 7 h T 2 n h G 8 5 e v n z p l 1 k 0 9 C Z C n U / R y M s z I S J M G d U 6 5 L G P 4 V P d Y P H w 1 B m Z Y z S 1 T F I x p n t l 8 3 / P s n K 0 U T v a + F 5 H O O k 0 B H 0 S G 1 t R z U f j 9 Z B Q L Z a 3 v l 6 + X K g U H J 4 f h x k m h X H S M l H k z P B q o 4 / O l L l 6 M t U 1 6 2 q t x Z X V v L 0 f T q L 2t V D 0 p t e 5 0 2 v t e 3 s 9 W d Y c T 6 5 q 6 k y A C j X j V V c f 2 x b T v y n g X t 2 / 5 / a 3 U 6 1 v g K N W 1 4 u n W W k O o l j p N V Y X e K 6 j h b q q H I 7 i L C s 0 r a 8 M r y 8 r A 6 Q G y a z T T D q y w F i Y z / q q 2 g j 8 e L b Z NC j 9 O B p y g 3 f m u k h m h p o / c Q l C t n f Q z J y m B L l Q u S 4 X U Z y l G l a r i 0 6 y x C v 9 I v I x Z k W l c Z D + T P l + k G l W J O j 3 e R + h 5 3 O 7 o k W D 9 o k Z u M y A m M B l A m K G L j O c 1 3 o r w K W A O o 1 c Z k R M 6 D I h M W O X G R M T u U x E z J 3 L 3 L H B 3 b v U P X W K XS a e a y 0 X i R c J j F s s q o e P 6 s w z u 7 j i 3 U 2 E 9 I Z Z + q X 0 V H m L m n x U B 5 C z N V 5 S + U 5 d 3 y n d N X O Z j J j c Z X I 2 i f c u 9 Z 4 6 F S 5 T E C N c R j B 3 0 q U k d Z q 4 z I S Y 0 m V K Y q Y u M y X m w W U e i H l 0 m U c 2 u A 8 u 9 W F u y i 0 b B Z i k s 8 L G Q F m F y s x E 1 G D E g q c e u B y b U L E W u s 1 u W D K S B U k 5 I J h F S B k Q z M K j H B I 8 Z D A Q z O K i H B E 8 4 k M J C W c h U Y 4 J Z v F Q T g h m w V D e E X z H 4 H u C W R S U M c E x g x O C E z 5 C t t Z 8 k T O C m a T L n O C c w e 8 J Z l o u C 4 K Z k E t B s O D 7 S r D k I 2 S r w v V b E s z E W 0 4 J Z s o t H w h m s i 0 f C X 5 k 8 A e C r V 6 3 Y l B P y / o J s H C 1 W 5 3 1 R n i t J z Q Y 9 b W e 0 W A k 2 H p K g 9 F h + z k N R o 2 t J z U Y S b a e 1 W B k 2 X p a g 9 F m 6 3 m N 3 M I T G 4 x K m 2 e 2 5 R Y e 2 m D 0 2 j y 2 L Z e 4 X O J M v u 1 M r s m F x z I Y E T c P Z s u 1 n c w 1 u f B w B i P q 5 v F s u Y X n M x h 1 t 5 7 Q Y C T e e k a D 0 X n r K Q 1 G 7 K 3 n N B j F t 5 / U Y H S / + K z G i C i i o C 5 Y k j W K k j U W s M k 6 4 e s U V M k G w R s M 3 i R 4 k 8 F b B G 8 x e J v g b Q b v E L z D 4 F 2 C d x m 8 R / A e H / g + 4 f v M / I D g A w Z 3 C e 4 y + J D g Q w Y f E X z E 4 G O C j / l Q e o T 3 m P k J w S c M P i X 4 l M F n B J 8 x + J z g c w Z f E H z B 4 E u C L / k I r w i / Y u b X B F 8 z + I b g G w b f E n x r 6 u b W k 9 Z V H h j p M a G u M Y V r + T F u n X M b L r f B u U 2 X 2 + T c l s t t c Wn M 3 L n f D u V u X q 6 V / y a v h 8 g P o p w t 8 q n 1 V d y 6 z F G b 2 S d d i y c R A / Y T S R 1 0 m K 7 w u k Q 1 V w Q w Z G G R A q 6 m L E o S o G N G l C C J U D p f V U K j u 0 F U H I l R t 6 F o D E a o x d I W B C F U W u q 5 A J G K 3 v z M Q l R G 6 i E C E i g d d O i A S s 5 U w S E J I a p C U r a B B q C T Q B Q E i V A j o M g A R K m Z 1 7 k e E c r 7 O + I i w B z G d 6 B G i B F 9 W O 8 P 2 p T Q I J X O d y h G h F K 4 T O C K U u H X a R o S K V J 2 r E f n Q d m w 6 T 1 V Q + n E + V n u u / 6 1 l W A 4 q g W h t W J C e w O i 1 R U X F f j I Y q h 7 m g o g s g V D h + l + C t S y V J G 2 A l u g R I f y b Y S I K E 9 V Z / 0 u d r X w r 6 d Z T m c 3 4 D G Z K s r a F i g 2 o h W o d s m n N l E p t C 1 U 6 o h Y q N J z T O x u l z z G x O O K I W i j L O z Z 4 1 O Q 9 W 6 C Z 0 m I 9 / Z n S o W 3 h i r K l R A 1 m 1 E L 9 5 d R C 7 b 2 n F u q u 4 K s 1 U 4 K r F 2 m m t G Z b u N w T a q H O S m q h x q b U Q n 0 9 U A u 1 9 U g t 1 N W H e f U D G W b d h + r W O u W i 3 i j V 6 k S L y D o F g M 6 v C F F e 1 V k V E c q m O p c i Q j l U Z 1 B E K H P q v I k I 1 X I 6 W S J C S V K n S E Q o N e r E i A g l R J 0 O E a E 0 q J M g I p T 8 d O p D h F K e T n i I U J 2 m s x w i r E T T y Q 0 h S m o 6 p S F C q U w n M k Q o g e n 0 h Q i V Y z p n I X L O X F 8 Y i F K U T l C I U G L S a Q k R S k c 6 G S F C S U i n I E R u m O t b A 9 2 y X a Q 0 M e B Z I u m N q 2 O 5 j 1 d s A e 0 p o J h u d R L U 0 6 u C W X F n J q D N 6 X Q O q V C / E m 9 C E P s F o L T G a + o 0 w l u a C l C M I v V S F d I g G 0 Z p i N 7 8 S a w Q M a q v k / l M q P f B Z y A X O R h k 8 f D 7 3 A w e 5 h i L z 5 o v d V O h f 1 U 0 i b R y q N 9 o V 5 O T p u h M B Y s B u W 4 x K j P l h s U 2 a O H l p g U p E u S W x S g W 5 L b F K B r k j s U o H u S u x S g i 5 J 7 F K C b k v s U o K u S B x S g u Z N d i X T b o Q w t S c M g j i 1 F 4 y G O L U f k n e x b r M Y c n F q Q g k a c W o z C R Z x a j Q J H n F q O y T l 5 Y j G J F X l q M o k V e W Y z i R V 5 b j C J G 3 l i M x Y y 8 t W B V p 6 G c d w o / H x s 6 t I / B g f M g E q 4 z m M Q R b j C Y j s l w k 8 G k k H C L w S S S c J v B p J N w h 8 E k l X C X w a S W c I / B J J h w n 8 G k m f C A w Q e 0 O G G X 4 X S o h o c M J v G E R w w m / Y T H D C Y J h T 0 G 0 z k b n j D 4 h A 3 l l O G k p f C M w S S n 8 J z B p K j w g s E k q v C S w e w x O L x i O G k r v G Y w y S u 8 Y T A 9 E I S 3 D D Y a U 7 + + y 6 p + E / V 7 l g F / z y L W C S Z 9 i Q 1 C S V 5 i k 1 C t r h f e p v 7 l Y y L A 8 z 0 B 0 s N 7 x z D 0 t l a 8 A Q S + w u U 4 E t 4 0 m 8 R D h L A F n t C / k 2 C F O S k 8 9 Y F P F q M j 9 b E M P O R Y c e r f g K s f Q 8 U 2 3 Z E U K n Y I J Y G K X U J J n 2 K P U P a W R u w T T P I U B 4 T S o S a 6 h J I 2 x S G h h 8 z x E c E k T X F M K C l T 9 A g l Y Y o T Q u l 8 E 6 e E k i r F G a H s G V a c E 0 y i F B e E k i b F J a F 0 1 I k r Q q + Y 4 2 u C S Z D i h l D S o 7 g l 1 M p x K 8 V a E P T D h W 9 e x z x U B S M V B F 3 3 s U B V j G v U Q r m u U w t l u k F l r a o S N 4 n E U 2 + L W q i k b W q h g n a o h c r Z p R Y q Z o 9 a K J R 9 a q F A D q i F w u h S C w V x y A a D Q j g i E g V w T C 3 c + B 4 z x R 0 / I R J 3 + p R a u M N n 1 M K N P a c W b u g F t X A j L 6 m F G 3 h F L d y 3 a 2 r h f t 2 w 2 + N G 3 b I b V s V X V X i p b Q O + b d L U Y O p w U Y G s P 9 7 D a D b w i j e N 5 D i b S A 8 L I G + K + S 2 H w i 2 R g G o k p z 6 q 7 i 9 r I W j D J 8 U h 6 A I K G h U U 6 B I K G j U U 6 C I K q I q q 0 M 0 a Z a / n d B 0 F j U I K d C U F j V I K d C 0 F j W I K d D U F V E 5 Z e K + G 2 Q s 4 X V J B o 6 Y C X V R B o 6 o C X V Y B 1 V U V e l i j 7 E W b r q y g U V q B r q 2 A i i s L 9 2 q 4 x + G T G m Z v 1 H S J B Y 0 a C 3 S R B Y 0 q C 3 S Z B Y 0 6 C 3 S h B Y 1 K C 3 S p B Y 1 a C 3 S x B Y 1 q C 3 S 5 B V R v W f i m h t k b M l 1 y A a + 5 8 A k C E 5 A s J u B N 0 i E U 8 a P 6 9 G n o S 9 8 L I Y U C c 4 9 q R w L 1 P p i o R O R q N 1 e m 8 1 n + b t Y v k p l u 6 D S o v E K S R 0 W E C d D p X 3 + E O H j U y U 9 / S K J u g t m y 4 d t + Y z L 2 J T 7 I u 7 d w L H v c s j d v G 0 y S D S H + 2 E S 0 Q T 0 T 0 5 o 3 n 2 Z 6 l V W v Y d U Y d x Q P o b L s 6 0 Y 9 / L o H H h Y y C 8 a + U B / V + h O Z 6 U c r K J w h N j 6 G z Y 1 N P c i q y 9 M B D M G x M 8 0 W u w I J P H q s n W m i G A L z Q 7 F r H f t 5 7 A c w r 7 / K 6 V b A 3 H v h V d f u A r v 9 t + Z 1 9 t t q j q Q r 2 I c / 3 S Z 7 O u e Z v u E W V c Z W u d E z L u b 2 9 Z x L F B D O 6 x d u T S q Q N E n V i k a R W j 3 X T G Q j m f g P Z G m B p h 1 m j U x / C W X e x D 3 1 k s c T N f 0 P 6 i 1 B Y 3 I H 3 T n / D u q g + 2 Q P L / 2 C h q A a z R t I / M c v c P u L j F m e P d m C j a w k W j V 0 o r v K 4 l H h J + q d 1 X i a F V i z C v 9 R e M + 7 3 7 1 + r j 4 A 0 l + r T 1 L z R a v I U Q J C f 4 X 2 v A 9 x z G z s m 9 M X 3 j q m Q o z 7 V P 3 1 i E E P i f o a T h X G x i m z V p + j Z p N Q Z 0 9 d J 0 c S V r R 7 k X n D D J S 7 a X Q f 5 T C M / N X G B 8 1 Z k c T q T f 9 8 1 v 3 u 1 b y F z F J Q X K e N k 1 P d 7 3 V z P z S Z K y p v 6 a b l 0 P 2 u H 6 U j + d j s m v u F e o 2 M Z 4 e v A u Y M 8 M Q V f g h e l H p p V h X 5 E h 5 W v Y 1 x J t T 6 Z K o g D M b e J j 4 T p / C l 8 A Z Z d r + q w p K 9 7 T n O 1 S G d F X 9 A n R e h H g H + 2 1 9 R V x 8 z V M e l M Y z 0 O 5 s W S 6 1 Y N N N / L 7 A 4 R 0 2 d q 0 8 F Y 5 B 9 f 4 C x F m f T Q Q H + / b N 3 n y 9 3 m v + vx N O L y 9 e r n T + u v j l 5 s / z N e v X / U X y 2 9 N u l 3 y 1 9 t d R Z + t P S N 0 u 7 S 7 2 l i 6 V g 6 W 7 p r 0 t / W / p 7 5 x + d f 3 b + 1 f m 3 M f 3 0 k 6 r P b 5 a c P 5 3 / / g 8 Y F F F d < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M r l 2 v q 5 b VF J E L B E Y L i e O I / 8 A 3 H w = " > A A A y J H i c l V v J c t z I E e W M t z G 9 a e w I X 3 x B m J J n 7 K A Y p G b C 9 n G 4 b 0 2 y u S / T G g U a n Y 2 G i E 2 o a p B U u / 0 x P v h i f 4 p v D h 9 8 8 W f 4 7 K w q F D I L j d a E F S E R 9 V 5 W o p a X l Q k Q 6 u d x J O T q 6 r 8 / + v g 7 3 / 3 e 9 3 / w y Q 8 X f / T j n / z 0 Z 8 8 + / f m V y M Z F A J d B F m f F T d 8 X E E c p X M p I x n C T F + A n / R i u + / e b i r 8 u o R B R l l 7 I p x x e J 3 6 Y R s M o 8 C V C b 5 7 9 s p f 4 c t Q f T k 6 n v d / Y 6 8 P p m 2 d L q y u r + o 8 3 e 7 F W X S w t V H + 6 b z 5 d / W 9 v k A X j B F I Z x L 4 Q X 6 + t 5 v L 1 x C 9 k F M Q w X e y N B e R + c O + H 8 D V e p n 4 C 4 v V E T 2 D q v U B k 4 A 2 z A v + m 0 t M o 7 z H x E y G e k j 5 a q i G K J q f A e Z z y K J b 7 S W M I c v j H 1 5 M o z c c S 0 s C M Y D i O P Z l 5 a p m 8 Q V R A I O M n v P C D I s J J e M H I L / x A 4 m I u L r 7 w j v z i 3 h N o g 8 s o v G z o B X 5 u r t U 0 C h h C U U R p q B w O o j I S 1 m w Y h e M C c J Q p P A R Z k v j p Y N J D M I a h n E 4 m P U i 8 z z t 4 / d v p d H H G K M D V h c K a b e q W M m z a F V E 4 q r 2 d q U a b l c x y a 3 O R 5 W 0 W / U z K L L F G G 7 o 1 Y 1 d N 3 L d m / u z Q K 5 O + N e n P c x J Y i 2 C e x c B a D P R t X n h 7 O L 1 Y T d H z P b R X W w Z D j I a B h 4 u T u D 7 w W o H T r 9 d e o 5 f + 0 F t a U 0 7 Q y 4 7 e F r N v K A h Y 9 u L s A Y q X A c b W y m I P X e p 1 h e H S 2 s R s 4 Z 9 7 2 J o Y B 2 3 9 c b y R 9 O M V b w f 1 I C Q G g t p + o f Y M e e N y x 7 r c a b r U t H z I 7 E 2 X X l W 3 F Z 4 1 8 n B K V e O V 7 f F u 7 A + o y 9 I X S 1 / O d F u u + 9 i r L 7 i r L 9 H V C + / c C H v e e p i b o f r N 6 K s w Y M N v 9 W B X x P Q + t 7 3 P W 3 q f 2 V 4 6 I B + y O t J W 6 p U x d x d 6 a e o 4 n L M 2 T Y e j A o C 5 5 N M R Z u F m X d K 6 M e d f z D r 3 U w 9 w G 1 T n F h H B O z N p a z J / 1 o 6 f c Z 5 D 4 S k / x s 1 2 5 W a 7 z c 2 6 V / g P t P A N Z y 9 f v v T L L B p 4 Y 6 H O p 2 j o 5 Z k Q E S a M a h 3 y 2 M f w q W 4 w f 3 j q P M w x m l o m q R j T v b L 5 v 2 d Z O d q s H W 1 + q y O c d B q C P o m N r a j m o / F 6 S K g W y 1 t f L 1 / O V Q o O z 4 / D D B P A K G m Z K H J m e L X R B 2 f K X M 1 M d d 2 6 W m 9 x Z T V v 7 4 e T q H 3 N F b 3 p d e H 0 W v / W X j P L m u P J V U 2 d C V C h Z r z q 6 k P b Y v o 3 B d y t + 3 f d / n a q 9 Q 1 w 1 O p 6 / j Q r z U E U K 7 3 G 6 g L P d b R Q V 5 X D Y Z x l h a b 1 l e H 1 Z W W A V D + Z r D W T j i w w F q Y T X R g F f j z Z a h q U f h w N u M E b c 1 0 k E 0 N N Z 1 y C k O 0 d N D O l K U E u V K 7 L R R R n q Y b V 6 q K T L P F K v 4 h 8 j F l R a R y k P 1 G + H 2 W a F Q n 6 f d 5 D 6 P n U r m j R o H 1 i + i 7 T J y Z w m Y C Y g c s M p r X e C n A p o E 5 D l x k S E 7 p M S M z I Z U b E R C 4 T E f P W Z d 6 y w d 2 7 1 D 1 1 i l 0 m n m o t F 4 k X C Y x b L K k H T + r M M 7 u 4 7 L 0 d C + k N s v Q z 6 a l S F j X 5 p A 4 g Z 2 u 8 p P K d u r 5 T u m v m M h k x u c v k b B L v X O o d d S p c p i B G u I x g 7 q R L S e o 0 d p k x M a X L l M Q 8 u M w D M Y 8 u 8 0 j M k 8 s 8 s c G 9 d 6 n 3 U 1 N u 2 S j A J J 0 V N g b K K l Q m E / s g Q 8 F T D 1 y O T K h Y C 9 1 m N y w Z y Y K k 7 B P M I q Q M C G b h U Q 4 I H j A Y C G Z x U Q 4 J H v K h h I S z k C h H B L N 4 K M c E s 2 A o 3 x L 8 l s H 3 B L M o K G O C Y w Y n B C d 8 h G y t + S J n B D N J l z n B O Y P f E c y 0 X B Y E M y G X g m D B 9 5 V g y U f I V o X r t y S Y i b d 8 I J g p t 3 w k m M m 2 f C L 4 i c H v C b Z 6 3 Y 5 B P R n r J 8 D C 1 W 5 1 1 h v h t Z 7 Q Y N T X e k a D k W D r K Q 1 G h + 3 n N B g 1 t p 7 U Y C T Z e l a D k W X r a Q 1 G m 6 3 n N X J z T 2 w w K m 2 e 2 Z a b e 2 i D 0 W v z 2 L Z c 4 n K J M / m 2 M 7 k m 5 x 7 L Y E T c P J g t 1 3 Y y 1 + T c w x m M q J v H s + X m n s 9 g 1 N 1 6 Q o O R e O s Z D U b n r a c 0 G L G 3 n t N g F N 9 + U o P R / f y z G i O i i I K 6 Y E n W K U r W W c A m G 4 R v U F A l m w R v M n i L 4 C 0 G b x O 8 z e A d g n c Y v E v w L o P 3 C N 5 j 8 D 7 B + 3 z g B 4 Q f M P N D g g 8 Z 3 C G 4 w + A j g o 8 Y f E z w M Y N P C D 7 h Q + k S 3 m X m p w S f M v i M 4 D M G n x N 8 z u A L g i 8 Y f E n w J Y O v C L 7 i I 7 w m / J q Z 3 x B 8 w + B b g m 8 Z f E f w n a m b W 0 9 a V 3 l g p M e E u s 4 U r u X H u A 3 O b b r c J u e 2 X G 6 L c 9 s u t 8 2 5 H Z f b 4 d y u y + 3 y a N P q Z O Q e 7 7 j v c v u c O 3 C 5 A 8 4 d u t w h 5 z o u 1 + H c k c s d c e 7 Y 5 Y 6 d S Z y 4 5 A n v 2 H W 5 L u d O X e 6 U c 2 c u d 8 a 5 c 5 c 7 d w Z z 4 Z I X v O O l y 1 1 y 7 s r l r j h 3 7 X L X n L t x u R v O 3 b r c L e f u X K 6 W / h W v h s v 3 o J 8 u 8 K l 2 t e 5 c Z i l M 7 J O u x Z K x g X o J p Y + 6 T F Z 4 X S I b q o I Z 0 j d I n 1 Z T F y U I U T G i S x F E q B w u q 6 F Q 3 a G r D k S o 2 t C 1 B i J U Y + g K A x G q L H R d g U j E b v / W Q F R G 6 C I C E S o e d O m A S M x W w i A J I a l B U r a C B q G S Q B c E i F A h o M s A R K i Y 1 b k f E c r 5 O u M j w h 7 E d K J H i B J 8 W e 0 M 2 5 f S I J T M d S p H h F K 4 T u C I U O L W a R s R K l J 1 r k b k f d u x 6 T x V Q e n H + U j t u f 5 Z y 7 D s V w L R 2 r A g P Y H R a 4 u K i v 2 k P 1 A 9 z A U R W Q K h w v V P g r U s l S R t g J b o E S H 8 l 2 E i C h P V W f + k z l a + l X T r q U w m f A Y T J V n b Q s U G 1 E K 1 D t i 0 J k q l t o U q H V I L F R p O 6 Z 2 N 0 u e I W B x x R C 2 U 5 V s 2 e N T k P V u g i d J i P f 2 J 0 q F t 4 Y q y p U Q N Z t R C / e X U Q u 2 9 o x b q r u C r N V G C q x d p o r R m W 7 j c Y 2 q h z k p q o c Y e q I X 6 e q Q W a u u J W q i r 9 6 b s O 1 J Z 9 7 G 6 t U 6 5 q D d K t T r R I r J B A a D z K 0 K U V 3 V W R Y S y q c 6 l i F A O 1 R k U E c q c O m 8 i Q r W c T p a I U J L U K R I R S o 0 6 M S J C C V G n Q 0 Q o D e o k i A g l P 5 3 6 E K G U p x M e I l S n 6 S y H C C v R d H J D i J K a T m m I U C r T i Q w R S m A 6 f S F C 5 Z j O W Y h c M N e X B q I U p R M U I p S Y d F p C h N K R T k a I U B L S K Q i R W + b 6 z k B 3 b B c p T f R 5 l k i 6 o + p Y 7 u E V W 0 B 7 C i i m U 5 0 E 9 f S q Y F b c u Q l o c z p d Q C r U b 4 S 3 I I j 9 A l B a o 3 V 1 G u E t T Q U o h p F 6 q Q p p k A 2 i N E R v / j h W i B j W 1 8 l 0 I t T 7 4 H O Q 8 x z 0 s 3 j w b W 7 6 j 1 O M x c X m S 9 1 U 6 N 8 q m k R a O d R v t K v J S V N 0 p o L F g N y w G J W Z c t N i m 7 T w c s u C F A l y 2 2 I U C 3 L H Y h Q N c t d i F A 9 y z 2 I U E X L f Y h Q T 8 s B i F B X y 0 G I U F 7 J j s Q 4 b 9 J E F K T j k s c U o P O S J x a j 8 k 1 2 L d Z n D U w t S k M g z i 1 G Y y H O L U a D I C 4 t R W S c v L U a x I q 8 s R t E i r y 1 G 8 S J v L E Y R I 2 8 t x m J G 3 l m w q t N Q z r u F n 4 8 M H d r H 4 M B 5 E A k 3 G E z i C D c Z T M d k u M V g U k i 4 z W A S S b j D Y N J J u M t g k k q 4 x 2 B S S 7 j P Y B J M e M B g 0 k x 4 y O B D W p y w w 3 A 6 V M M j B p N 4 w m M G k 3 7 C E w a T h M I u g + m c D U 8 Z f M q G c s Z w 0 l J 4 z m C S U 3 j B Y F J U e M l g E l V 4 x W D 2 G B x e M 5 y 0 F d 4 w m O Q V 3 j K Y H g j C O w Y b j a n f v s u q f h P 1 e 5 Y + f 8 8 i N g g m f Y l N Q k l e Y o t Q r a 4 X 3 p b + z c d Y g O d 7 A q S H 9 4 5 h 4 G 0 v e 3 0 I f I X L U S S 8 h 2 w c D x D C F n h C / 5 4 E K 8 x x 4 a m P e b I Y H a m P Z e A x x 4 p T / w 6 4 + m W o 2 K E 7 k k L F L q E k U L F H K O l T 7 B P K 3 t K I A 4 J J n u K Q U D r U R I d Q 0 q Y 4 I v S I O T 4 m m K Q p T g g l Z Y o u o S R M c U o o n W / i j F B S p T g n l D 3 D i g u C S Z T i k l D S p L g i l I 4 6 c U 3 o N X N 8 Q z A J U t w S S n o U d 4 R a O W 6 n W A u C f r j w z e u Y x 6 p g p I K g 4 z 4 W q I p x n V o o 1 w 1 q o U w 3 q a x V V e I W k X j q b V M L l b R D L V T Q L r V Q O X v U Q s X s U w u F c k A t F M g h t V A Y H W q h I I 7 Y Y F A I x 0 S i A E 6 o h R v f Z a a 4 4 6 d E 4 k 6 f U Q t 3 + J x a u L E X 1 M I N v a Q W b u Q V t X A D r 6 m F + 3 Z D L d y v W 3 Z 7 3 K g 7 d s O q + K o K L 7 V t w L d N m h p M H S 4 q k P W H e h j N B l 7 2 H i I 5 y s b S w w L I e 8 D 8 l k P h l k h A N Z J T H 1 X 3 l 7 U Q t O F M c Q i 6 g I J G B Q W 6 h I J G D Q W 6 i A K q o i p 0 q 0 b Z 6 z l d R 0 G j k A J d S U G j l A J d S 0 G j m A J d T Q G V U x b e r 2 H 2 A k 6 X V N C o q U A X V d C o q k C X V U B 1 V Y U e 1 S h 7 0 a Y r K 2 i U V q B r K 6 D i y s L d G u 5 y + L S G 2 R s 1 X W J B o 8 Y C X W R B o 8 o C X W Z B o 8 4 C X W h B o 9 I C X W p B o 9 Y C X W x B o 9 o C X W 4 B 1 V s W v q 1 h 9 o Z M l 1 z A a y 5 8 g s A E J I s x e O N 0 A E X 8 p D 5 9 G v j S 9 0 J I o c D c o 9 q R Q L 3 3 x y o R u d r N l e l 0 k r + Z 9 I p k o h s 6 D S q v k O R R E W E C d P r X H y H 2 n 3 T y 0 x + S q J t g t m z 4 t t + Y j H y J D / L u L R z L L r f s T t s G k 2 Q D i D 8 0 E W 1 Q z 8 S 0 p s 2 n m W 5 l 1 W 1 Y N c Y d x Q O o L H u 6 U Q + / 7 o G H h c y C k S / U B 7 T + W G b 6 0 Q o K Z 4 i N j 2 F z Y 1 M P s u o y O 4 A B O H a m 2 W J X I I F H j 7 U z T R R D Y H 5 R 7 F r H f h 7 7 A U z r r 3 I 6 F T D 1 X n j V t b v A b v / t a Z 3 9 t p s j 6 Q j 2 4 U + n y Z 5 N e a Z v u E W V s V V u 9 I y L q X 0 9 5 x I F h N P 6 h V u T C i R N U r W i Y a R W z z U T 2 V A m / i N Z W q B p h 1 k j 0 1 9 C m T d x s 1 7 y e K y m / 1 6 9 J W h M 7 r A z 5 d 9 B H X Z m 9 v D K L 2 g I q t G 8 g c Q f f o H b X 2 T M 8 n x m C z a z k m j V 0 I n u O o u H h Z + o d 1 a j h 6 z A m l X 4 T 8 J 7 3 v n m 1 X P 1 A Z D + M n 2 c m i 9 a R Y 4 S E P o r t O c 9 i G N m Y 9 + c v v A 2 M B V i 3 K f q n y c M e k j U 1 3 C q M D Z O m b X 6 H D U b h z p 7 6 j o 5 k r C s 3 Y v M G 2 S g 3 D 1 E 9 1 E O g 8 h f a X z Q n B V J r N 7 0 T y e d b 1 a n L W S W g u L W 2 j j 5 o P u 9 a u 6 H J n N F 5 S 3 d t B w 6 3 / S i d C i f m l 1 z v 1 C v k f H s 8 F X A n A O e u M I P w Y t S L 8 2 q I l / C 4 4 q 3 O c q E W p 9 M F Y T B y N v C Z + I U P h N e P 8 v u V 1 R Y s r c 9 J 7 k 6 p L P i d 6 j z I t Q j w J + 9 Z X X 1 I U N 1 X B r D S L + z a b H U i k U z / e 8 c i w v U 1 I X 6 V D A G 2 f P 7 G G t x 9 t A v w L 9 f f P N s a a 3 5 / y J m L 6 5 e r a z 9 f u X L 0 y + X v t q o / s / E J w u / W v j 1 w u c L a w t / W P h q Y W + h u 3 C 5 E C z 8 a e E v C 3 9 b + P v a X 9 f + s f b P t X 8 Z 0 4 8 / q v r 8 Y s H 5 s / a f / w E Q 1 1 D y < / l a t e x i t > Q&K < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 H a I A i 1 J f / A G Y M f C q P H w v O d c j x o = " > A A A y F n i c l V v J c t z I E e V 4 H d P b j H 3 0 B W F K M W O H x C A 1 C t v H 4 b 4 1 y e a + T G s U a H Q 2 G i I 2 o a r R p D r a X + G D L / a n + O b w 1 V f / i M / O q k I h s 9 B o T V g R E l H v Z S V q e V m Z A K F + H k d C r q 3 9 5 5 P v f f 8 H P / z R j z / 9 y f J P f / b z X / z y s 8 9 / d S 2 y c R H A V Z D F W X H b 9 w X E U Q p X M p I x 3 O Y F + E k / h p v + w 5 b i b 0 o o R J S l l / I p h z e J H 6 b R M A p 8 i d B 9 L / H l q D + c X s / e f r a y t r q m / 3 j z F + v V x c p S 9 a f 7 9 v O 1 / / Y G W T B O I J V B 7 A v x z f p a L t 9 M / U J G Q Q y z 5 d 5 Y Q O 4 H D 3 4 I 3 + B l 6 i c g 3 k z 1 k G f e c 0 Q G 3 j A r 8 G 8 q P Y 3 y H l M / E e I p 6 a O l G q J o c g p c x C m P 4 k U / a Q x B D v / 0 Z h q l + V h C G p g R D M e x J z N P L Y w 3 i A o I Z P y E F 3 5 Q R D g J L x j 5 h R 9 I X L 7 l 5 e f e s V 8 8 e A J t c O G E l w 2 9 w M / N t Z p G A U M o i i g N l c N B V E b C m g 2 j c F w A j j K F S Z A l i Z 8 O p j 0 E Y x j K 2 X T a g 8 T 7 s o P X v 5 v N l u e M A l x d K K z Z l m 4 p w 6 Z d E Y W j 2 t u 5 a r R Z y S y 3 N p d Z 3 m b R z 6 T M E m u 0 q V t z d t X E f W v m z w + 9 M u l b k / 4 i J 4 G 1 C B Z Z D K z F Q N / m u b e P 0 4 v V F D 3 f Q 3 u 1 Z T B E / Q 8 8 X J z E 9 Y H X C p x 9 s / 4 G v f S H 3 s q 6 c o J e d v W 2 m H 1 D Q c A L L 8 4 m U L w M M J p W l 3 v o U q 8 r D F f W p 2 Y L / 9 z D 1 t Q 4 a O u P 4 4 2 k H 6 9 6 u 6 g H I T E Q 1 P Y L t W f I G 5 e 7 1 u V u 0 6 W m 5 S S z N 1 1 5 V d 1 W e N b I w y l V j V e 2 x / u x P 6 A u K 1 + t v J 7 r 9 q L u Y 6 + + 4 q 5 e o 6 v n 3 o U R 9 q L 1 M D d D 9 Z v R V 2 H A h t / q w a 6 I 6 X 1 h e 1 + 0 9 D 6 3 v X R A T r I 6 0 l b r l T F 3 F 3 p p 6 j h c s D Z N h 6 M C g L n k 0 x F m 4 e Z d 0 r o x 5 1 / N O / d T D 3 A b V O c W E c F 7 M 2 l r s n j W j p 9 x n k P h K T / G z U 7 l Z q f N z Y Z X + B N a + I a z l y 9 f + m U W D b y x U O d T N P T y T I g I U 0 S 1 D n n s Y / h U N 1 g 8 P H U e 5 h h N L Z N U j O l e 2 f z f s 6 w c b d W O t r 7 T E U 4 6 D U G f x M Z W V P P R e D 0 k V I v l r a + X L x c q B Y f n x 2 G G C W C U t E w U O T O 8 2 u i j M 2 W u 5 q a 6 Y V 1 t t L i y m r f 3 w 0 n U v h a K 3 v S 6 d H p t f G e v u W X N 8 e S q p s 4 E q F A z X n X 1 s W 0 x / Z s C 7 t b 9 u 2 5 / O 9 X 6 B j h q d b 1 4 m p X m I I q V X m N 1 g e c 6 W q i r y u E w z r J C 0 / r K 8 P q y M k C q n 0 z X m 0 l H F h g L s 6 k u j A I / n m 4 3 D U o / j g b c 4 K 2 5 L p K p o W Z z L k H I 9 g 6 a m d G U I B c q 1 + U i i r N U w 2 p 1 0 U m W e K V f R D 7 G r K g 0 D t K f K t + P M s 2 K B P 0 + 6 y H 0 b G Z X t G j Q P j F 9 l + k T E 7 h M Q M z A Z Q a z W m 8 F u B R Q p 6 H L D I k J X S Y k Z u Q y I 2 I i l 4 m I e e c y 7 9 j g H l z q g T r F L h P P t J a L x I s E x i 0 W 0 Y M n d e a Z X X z h v R s L 6 Q 2 y 9 A v p q V I W N f m k D i B n a 7 y k 8 p 2 6 v l O 6 a + Y y G T G 5 y + R s E u 9 d 6 j 1 1 K l y m I E a 4 j G D u p E t J 6 j R 2 m T E x p c u U x E x c Z k L M o 8 s 8 E v P k M k 9 s c B 9 c 6 s P M l F s 2 C j B J Z 4 W N g b I K l e n U P s h Q 8 N Q D l y M T K t Z C t 9 k N S 0 a y I C n 7 B L M I K Q O C W X i U A 4 I H D A a C W V y U Q 4 K H f C g h 4 S w k y h H B L B 7 K M c E s G M p 3 B L 9 j 8 A P B L A r K m O C Y w Q n B C R 8 h W 2 u + y B n B T N J l T n D O 4 P c E M y 2 X B c F M y K U g W P B 9 J V j y E b J V 4 f o t C W b i L S c E M + W W j w Q z 2 Z Z P B D 8 x + A P B V q 8 7 M a g n Y / 0 E W L j a r c 5 6 I 7 z W E x q M + l r P a D A S b D 2 l w e i w / Z w G o 8 b W k x q M J F v P a j C y b D 2 t w W i z 9 b x G b u G J D U a l z T P b c g s P b T B 6 b R 7 b l k t c L n E m 3 3 Y m 1 + T C Y x m M i J s H s + X a T u a a X H g 4 g x F 1 8 3 i 2 3 M L z G Y y 6 W 0 9 o M B J v P a P B 6 L z 1 l A Y j 9 t Z z G o z i 2 0 9 q M L p f f F Z j R B R R U B c s y Q Z F y Q Y L 2 G S T 8 E 0 K q m S L 4 C 0 G b x O 8 z e A d g n c Y v E v w L o P 3 C N 5 j 8 D 7 B + w w + I P i A D / y Q 8 E N m f k T w E Y M 7 B H c Y f E z w M Y N P C D 5 h 8 C n B p 3 w o X c K 7 z P y M 4 D M G n x N 8 z u A L g i 8 Y f E n w J Y O v C L 5 i 8 D X B 1 3 y E N 4 T f M P N b g m 8 Z f E f w H Y P v C b 4 3 d X P r S e s q D 4 z 0 m F A 3 m M K 1 / B i 3 y b k t l 9 v i 3 L b L b X N u x + V 2 O L f r c r u c 2 3 O 5 P R 5 t W p 2 M 3 O c d D 1 z u g H O H L n f I u S O X O + J c x + U 6 n D t 2 u W P O n b j c i T O J U 5 c 8 5 R 2 7 L t f l 3 J n L n X H u 3 O X O O X f h c h f O Y C 5 d 8 p J 3 v H K 5 K 8 5 d u 9 w 1 5 2 5 c 7 o Z z t y 5 3 y 7 k 7 l 7 v j 3 L 3 L 1 d K / 5 t V w + Q H 0 0 w U + 1 a 7 V n c s s h a l 9 0 r V Y M j Z Q L 6 H 0 U Z f J C q 9 L Z E N V M E P 6 B u n T a u q i B C E q R n Q p g g i V w 2 U 1 F K o 7 d N W B C F U b u t Z A h G o M X W E g Q p W F r i s Q i d j t 3 x m I y g h d R C B C x Y M u H R C J 2 U o Y J C E k N U j K V t A g V B L o g g A R K g R 0 G Y A I F b M 6 9 y N C O V 9 n f E T Y g 5 h O 9 A h R g i + r n W H 7 U h q E k r l O 5 Y h Q C t c J H B F K 3 D p t I 0 J F q s 7 V i H x o O z a d p y o o / T g f q T 3 X P 2 s Z l v 1 K I F o b F q Q n M H p t U V G x n / Q H q o e 5 I C J L I F S 4 / k m w l q W S p A 3 Q E j 0 i h P 8 y T E R h o j r r n 9 T Z y r e S b j 2 V 6 Z T P Y K o k a 1 u o 2 I B a q N Y B m 9 Z U q d S 2 U K V D a q F C w x m 9 s 1 H 6 H B G L I 4 6 o h b J 8 x w a P m n x g C z R V W q y n P 1 U 6 t C 1 c U b a U q M G M W q i / n F q o v f f U Q t 0 V f L W m S n D 1 I k 2 V 1 m w L l 3 t M L d R Z S S 3 U 2 I R a q K 9 H a q G 2 n q i F u v p g y r 5 j l X U f q 1 v r l I t 6 o 1 S r E y 0 i m x Q A O r 8 i R H l V Z 1 V E K J v q X I o I 5 V C d Q R G h z K n z J i J U y + l k i Q g l S Z 0 i E a H U q B M j I p Q Q d T p E h N K g T o K I U P L T q Q 8 R S n k 6 4 S F C d Z r O c o i w E k 0 n N 4 Q o q e m U h g i l M p 3 I E K E E p t M X I l S O 6 Z y F y C V z f W U g S l E 6 Q S F C i U m n J U Q o H e l k h A g l I Z 2 C E L l j r u 8 N d M 9 2 k d J E n 2 e J p D u q j u U e X r E F t K e A Y j r V S V B P r w p m x V 2 Y g D a n 0 y W k Q v 1 G e B u C 2 C 8 A p T X a U K c R 3 t J U g G I Y q Z e q k A b Z I E p D 9 O a P Y 4 W I Y X 2 d z K Z C v Q + + A L n I Q T + L B 9 / l p v 8 4 w 1 h c b r 7 U T Y X + r a J J p J V D / U a 7 m p w 0 R W c q W A z I T Y t R m S m 3 L L Z F C y + 3 L U i R I H c s R r E g d y 1 G 0 S D 3 L E b x I P c t R h E h D y x G M S E P L U Z R I Y 8 s R n E h O x b r s E E f W 5 C C Q 5 5 Y j M J D n l q M y j / Z t V i X O T y z I A W J P L c Y h Y m 8 s B g F i r y 0 G J V 1 8 s p i F C v y 2 m I U L f L G Y h Q v 8 t Z i F D H y z m I s Z u S 9 B a s 6 D e W 8 V / j 5 y N C h f Q w O n A e R c J P B J I 5 w i 8 F 0 T I b b D C a F h D s M J p G E u w w m n Y R 7 D C a p h P s M J r W E B w w m w Y S H D C b N h E c M P q L F C T s M p 0 M 1 P G Y w i S c 8 Y T D p J z x l M E k o 7 D K Y z t n w j M F n b C j n D C c t h R c M J j m F l w w m R Y V X D C Z R h d c M Z o / B 4 Q 3 D S V v h L Y N J X u E d g + m B I L x n s N G Y + u 2 7 r O o 3 U b 9 n 6 f P 3 L G K T Y N K X 2 C K U 5 C W 2 C d X q e u 5 t 6 9 9 8 j A V 4 v i d A e n j v G A b e z g u v D 4 G v c D m K h D f J x v E A I W y B J / T v S b D C H B e e + p g n i 9 G R + l g G H n O s O P X v g K t f h o p d u i M p V O w R S g I V + 4 S S P s U B o e w t j T g k m O Q p j g i l Q 0 1 0 C C V t i m N C j 5 n j E 4 J J m u K U U F K m 6 B J K w h R n h N L 5 J s 4 J J V W K C 0 L Z M 6 y 4 J J h E K a 4 I J U 2 K a 0 L p q B M 3 h N 4 w x 7 c E k y D F H a G k R 3 F P q J X j T o q 1 I O i H C 9 + 8 j n m s C k Y q C D r u Y 4 G q G D e o h X L d p B b K d I v K W l U l b h O J p 9 4 O t V B J u 9 R C B e 1 R C 5 W z T y 1 U z A G 1 U C i H 1 E K B H F E L h d G h F g r i m A 0 G h X B C J A r g l F q 4 8 V 1 m i j t + R i T u 9 D m 1 c I c v q I U b e 0 k t 3 N A r a u F G X l M L N / C G W r h v t 9 T C / b p j t 8 e N u m c 3 r I q v q v B S 2 w Z 8 2 6 S p w d T h o g J Z f 6 i H 0 W z g F 9 4 k k q N s L D 0 s g L w J 5 r c c C r d E A q q R n P q o u r + s h a A N 5 4 p D 0 A U U N C o o 0 C U U N G o o 0 E U U U B V V o d s 1 y l 7 P 6 T o K G o U U 6 E o K G q U U 6 F o K G s U U 6 G o K q J y y 8 E E N s x d w u q S C R k 0 F u q i C R l U F u q w C q q s q 9 L h G 2 Y s 2 X V l B o 7 Q C X V s B F V c W 7 t Z w l 8 N n N c z e q O k S C x o 1 F u g i C x p V F u g y C x p 1 F u h C C x q V F u h S C x q 1 F u h i C x r V F u h y C 6 j e s v B d D b M 3 Z L r k A l 5 z 4 R M E J i B Z j M E b p w M o 4 i f 1 6 d P A l 7 4 X Q g o F 5 h 7 V j g T q v T 9 W i c j V b q 5 M Z 9 P 8 7 b R X J F P d 0 G l Q e Y U k j 4 o I E 6 D T v / 4 I s f + k k 5 / + k E T d B L N l w 7 f 9 x m T k S 3 y Q d 2 / h W H a 5 Z X f W N p g k G 0 D 8 s Y l o g 3 o m p j V r P s 1 0 K 6 t u w 6 o x 7 i g e Q G X Z 0 4 1 6 + H U P P C x k F o x 8 o T 6 g 9 c c y 0 4 9 W U D h D b H w M m x u b e p B V l / k B D M C x M 8 0 W u w I J P H q s n W m i G A L z i 2 L X O v b z 2 A 9 g V n + V 0 6 m A m f f c q 6 7 d B X b 7 7 8 z q 7 L f T H E l H s A 9 / O k 3 2 f M Y z f c M t q o y t c q N n X M z s 6 z m X K C C c 1 S / c m l Q g a Z K q F Q 0 j t X q u m c i G M v E f y d I C T T v M G p n + E s q 8 i Z v 3 k s d j N f 0 P 6 i 1 B Y 3 J H n R n / D u q o M 7 e H 1 3 5 B Q 1 C N 5 g 0 k / v A L 3 P 4 i Y 5 Y X c 1 u w l Z V E q 4 Z O d D d Z P C z 8 R L 2 z G k 2 y A m t W 4 T 8 J 7 1 n n 2 1 f P 1 A d A + s v 0 c W q + a B U 5 S k D o r 9 C e 9 S C O m Y 1 9 c / r c 2 8 R U i H G f q n + e M O g h U V / D q c L Y O G X W 6 n P U b B z q 7 K n r 5 E j C C + 1 e Z N 4 g A + V u E j 1 E O Q w i f 7 X x Q X N W J L F 6 0 z + b d r 5 d m 7 W Q W Q q K W 2 / j 5 E T 3 e 9 X c D 0 3 m i s p b u m k 5 d L 7 t R e l Q P j W 7 5 n 6 h X i P j 2 e G r g L k A P H G F H 4 I X p V 6 a V U W + h M d V b 2 u U C b U + m S o I g 5 G 3 j c / E K X w h v H 6 W P a y q s G R v e 0 5 z d U h n x e 9 R 5 0 W o R 4 A / e y / U 1 c c M 1 X F p D C P 9 z q b F U i s W z f S / C y w u U V O X 6 l P B G G T P 7 2 O s x d m k X 4 D / s P z 2 s 5 X 1 5 v + L m L + 4 f r W 6 / o f V 1 2 e v V7 7 e r P 7 P x K d L v 1 n 6 7 d K X S + t L f 1 z 6 e m l / q b t 0 t R Q s p U t / W f r b 0 t / X / 7 r + j / V / r v / L m H 7 v k 6 r P r 5 e c P + v / / h 8 / 4 E u 7 < / l a t e x i t > V Normalized Attention Unit < l a t e x i t s h a 1 _ b a s e 6 4 = " J p Q n j I h 7 n 8 R p J C W 2 Y B x I F M 4 w 5 4 g = " > A A A y K n i c l V t Z k 9 v G E a a d y 1 E u O 3 n M C y o r l Z 3 U a m s p q 5 I 8 e u + L u 8 u 9 D 6 + s A s E m C C 0 u Y Y b g r l j M r 8 h r 8 g v y a / L m y m t + S H p m M O g e L C h X V G U J 8 3 0 9 j T m + n m 6 A 8 C C P I y F X V 7 / / 5 N M f / f g n P / 3 Z Z z 9 / 9 o t f / u r X v / n 8 i 9 9 e i m x S B H A R Z H F W X A 9 8 A X G U w o W M Z A z X e Q F + M o j h a n C / o f i r E g o R Z e m 5 f M z h T e K H a T S K A l 8 i d H u X + H I 8 G M 2 O 5 2 8 / X 1 p d W d V / v K c X 3 e p i q V P 9 6 b / 9 o v v s b p g F k w R S G c S + E N 9 2 V 3 P 5 Z u Y X M g p i m D + 7 m w j I / e D e D + F b v E z 9 B M S b m R 7 y 3 H u B y N A b Z Q X + l 0 p P o 7 z H z E + E e E w G a K m G K J q c A h d x y q N Y H i S N I c j R X 9 / M o j S f S E g D M 4 L R J P Z k 5 q m F 8 Y Z R A Y G M H / H C D 4 o I J + E F Y 7 / w A 4 n L 5 9 z i w c z h 2 b M X 3 q F f 3 H s C + + F i C i 8 b e Y G f m 2 s 1 t Q J G U B R R G q q b D K M y E t Z s F I W T A t B t C t M g S x I / H c 7 u E I x h J O e z 2 R 0 k 3 l c 9 v P 7 j H G / T N A p w x a G w Z h u 6 p Q y b d k U U j m t v p 6 r R Z i W z 3 N q c Z 3 m b x S C T M k u s 0 b p u P b G r J u 5 b M / / p 0 C u T g T U Z L H I S W I t g k c X Q W g z 1 b V 5 4 u z i 9 W E 3 R 8 z 2 0 V 9 s I I 4 y J o Y e L k 7 g + 8 F q B 8 2 + 7 b 9 D L Y O Q t d Z U T 9 L K t t 8 X s G 4 o E l r 0 4 m 0 L x M s A I W 0 G 5 w 0 i v K 4 y W u j O z h X + 7 w 9 b M O G j r j + O N p B + v e N u o B y E x O N T 2 C 7 V n y B u X 2 9 b l d t O l p u U 0 s z d d e l X d V n j W y M M p V Y 1 X t s f 7 i T + k L k t f L 7 1 + 0 m 2 5 7 m O v v u a u X q O r F 9 6 Z E f a i 9 T A 3 Q / W b 0 V d h w I b f 6 s G u i O l 9 Z n u f t f Q + t b 1 0 k E 6 z O t J W 6 p U x d x d 6 a e o 4 X L A 2 T Y f j A o C 5 5 N M R Z u G e u q R 1 Y 8 6 / f u r c T z 3 A b V C d W 0 Q E 7 8 2 k r c n i W T t + J n k O h a f 8 G D d b l Z u t N j d r X u F P a e E b z l 6 + f O m X W T T 0 J k K d T 9 H I y z M h I k w b 1 T r k s Y / h U 9 1 g 8 f D U G Z l j N L V M U j G m e 2 X z f 8 + y c r R R O 9 r 4 Q U c 4 6 T Q E f R I b W 1 H N R + P 1 k F A t l r e + X r 5 c q B Q c n h + H G S a F c d I y U e T M 8 G q j j 8 6 U u X o y 1 T X r a q 3 F l d W 8 v R 9 O o v a 1 U P S m 1 7 n T a + 0 H e z 1 Z 1 h x P r m r q T I A K N e N V V x / b F t O / K e B + 3 b / v 9 r d T r W + A o 1 b X i 6 d Z a Q 6 i W O k 1 V h d 4 r q O F u q o c j u I s K z S t r w y v L y s D p A b J r N t M O r L A W J j P d L E U + P F s s 2 l Q + n E 0 5 A Z v z X W R z A w 1 f + I S h G z v o J k 5 T Q l y o X J d L q I 4 S z W s V h e d Z I l X + k X k Y 8 y K S u M g / Z n y / S D T r E j Q 7 / M 7 h J 7 P 7 Y o W D d o n Z u A y A 2 I C l w m I G b r M c F 7 r r Q C X A u o 0 c p k R M a H L h M S M X W Z M T O Q y E T H v X O Y d G 9 y 9 S 9 1 T p 9 h l 4 r n W c p F 4 k c C 4 x c J 6 + K j O P L O L y 9 6 7 i Z D e M E u / l J 4 q b 1 G T j + o A c r b G S y r f q e s 7 p b t m L p M R k 7 t M z i b x 3 q X e U 6 f C Z Q p i h M s I 5 k 6 6 l K R O E 5 e Z E F O 6 T E n M 1 G W m x D y 4 z A M x j y 7 z y A b 3 w a U + z E 2 5 Z a M A k 3 R W 2 B g o q 1 C Z z e z D D Q V P P X A 5 N q F i L X S b 3 b B k J A u S c k A w i 5 A y I J i F R z k k e M h g I J j F R T k i e M S H E h L O Q q I c E 8 z i o Z w Q z I K h f E f w O w b f E 8 y i o I w J j h m c E J z w E b K 1 5 o u c E c w k X e Y E 5 w x + T z D T c l k Q z I R c C o I F 3 1 e C J R 8 h W x W u 3 5 J g J t 5 y S j B T b v l A M J N t + U j w I 4 M / E G z 1 u h W D e l r W T 4 C F q 9 3 q r D f C a z 2 h w a i v 9 Y w G I 8 H W U x q M D t v P a T B q b D 2 p w U i y 9 a w G I 8 v W 0 x q M N l v P a + Q W n t h g V N o 8 s y 2 3 8 N A G o 9 f m s W 2 5 x O U S Z / J t Z 3 J N L j y W w Y i 4 e T B b r u 1 k r s m F h z M Y U T e P Z 8 s t P J / B q L v 1 h A Y j 8 d Y z G o z O W 0 9 p M G J v P a f B K L 7 9 p A a j + 8 V n N U Z E E Q V 1 w Z K s U Z S s s Y B N 1 g l f p 6 B K N g j e Y P A m w Z s M 3 i J 4 i 8 H b B G 8 z e I f g H Q b v E r z L 4 D 2 C 9 / j A 9 w n f Z + Y H B B 8 w u E d w j 8 G H B B 8 y + I j g I w Y f E 3 z M h 9 I n v M / M T w g + Y f A p w a c M P i P 4 j M H n B J 8 z + I L g C w Z f E n z J R 3 h F + B U z vv g 3 K X L X X L u y u W u O H f t c t e c u 3 G 5 G 8 7 d u l w t / U t e D Z c f Q D 9 d 4 F P t a t 2 5 z F K Y 2 S d d i y U T A 9 0 l l D 7 q M l n h d Y l s q A p m y M A g A 1 p N X Z Q g R M W I L k U Q o X K 4 r I Z C d Y e u O h C h a k P X G o h Q j a E r D E S o s t B 1 B S I R u / 0 7 A 1 E Z o Y s I R K h 4 0 K U D I j F b C Y M k h K Q G S d k K G o R K A l 0 Q I E K F g C 4 D E K F i V u d + R C j n 6 4 y P C H s Q 0 4 k e I U r w Z b U z b F 9 K g 1 A y 1 6 k c E U r h O o E j Q o l b p 2 1 E q E j V u R q R D 2 3 H p v N U B a U f 5 2 O 1 5 / r f W o b l o B K I 1 o Y F 6 Q m M X l t U V O w n g 6 H q Y S 6 I y B I I F a 7 / J V j L U k n S B m i J H h H C v x k m o j B R n f W / 1 N n K t 5 J u P Z X Z j M 9 g p i R r W 6 j Y g F q o 1 i G b 1 k y p 1 L Z Q p S N q o U L D O b 2 z U f o c E 4 s j j q i F s n z H B o + a v G c L N F N a r K c / U z q 0 L V x R t p S o w Y x a q L + c W q i 9 9 9 R C 3 R V 8 t W Z K c P U i z Z T W b A u X e 0 I t 1 F l J L d T Y l F q o r w d q o b Y e q Y W 6 + j C v f i D D r P t Q 3 V q n X N Q b p V q d a B F Z p w D Q + R U h y q s 6 q y J C 2 V T n U k Q o h + o M i g h l T p 0 3 E a F a T i d L R C h J 6 h S J C K V G n R g R o Y S o 0 y E i l A Z 1 E k S E k p 9 O f Y h Q y t M J D x G q 0 3 S W Q 4 S V a D q 5 I U R J T a c 0 R C i V 6 U S G C C U w n b 4 Q o X J M 5 y x E z p n r C w N R i t I J C h F K T D o t I U L p S C c j R C g J 6 R S E y A 1 z f W u g W 7 a L l C Y G P E s k / X F 1 L N / h F V t A e w o o p l e d B P X 0 q m B W 3 J k J a H M 6 n U M q 1 K / E m x D E f g E o r f G a O o 3 w l q Y C F K N I v V S F N M i G U R q i N 3 8 S K 0 S M 6 u t k P h P q f f A Z y E U O B l k 8 / C E 3 g 4 c 5 x u K z 5 k v d V O h f F U 0 i r R z q N 9 r V 5 K Q p O l P B Y k C u W 4 z K T L l h s Q 1 a e L l p Q Y o E u W U x i g W 5 b T G K B r l j M Y o H u W s x i g i 5 Z z G K C b l v M Y o K e W A x i g v Z s 1 i P D f r Q g h Q c 8 s h i F B 7 y 2 G J U / s m + x f r M 4 Y k F K U j k q c U o T O S Z x S h Q 5 L n F q K y T F x a j W J G X F q N o k V c W o 3 i R 1 x a j i J E 3 F m M x I 2 8 t W N V p K O e d w s / H h g 7 t Y 3 D g P I i E 6 w w m c Y Q b D K Z j M t x k M C k k 3 G I w i S T c Z j D p J N x h M E k l 3 G U w q S X c Y z A J J t x n M G k m P G D w A S 1 O 2 G M 4 H a r h I Y N J P O E R g 0 k / 4 T G D S U J h n 8 F 0 z o Y n D D 5 h Q z l l O G k p P G M w y S k 8 Z z A p K r x g M I k q v G Q w e w w O r x h O 2 g q v G U z y C m 8 Y T A 8 E 4 S 2 D j c b U r + + y q t 9 E / Z 5 l w N + z i H W C S V 9 i g 1 C S l 9 g k V K v r h b e p f / m Y C P B 8 T 4 D 0 8 N 4 x D L 2 t Z W 8 A g a 9 w O Y 6 E N 8 0 m 8 R A h b I E n 9 O 8 k W G F O C k 9 9 4 J P F 6 E h 9 L A M P O V a c + j f g 6 s d Q s U 1 3 J I W K H U J J o G K X U N K n 2 C O U v a U R + w S T P M U B o X S o i R 6 h p E 1 x S O g h c 3 x E M E l T H B N K y h R 9 Q k m Y 4 o R Q O t / E K a G k S n F G K H u G F e c E k y j F B a G k S X F J K B 1 1 4 o r Q K + b 4 m m A S p L g h l P Q o b g m 1 c t x K s R Y E / X D h m 9 c x D 1 X B S A V B z 3 0 s U B X j G r V Q r u v U Q p l u U F m r q s R N I v H U 2 6 I W K m m b W q i g H W q h c n a p h Y r Z o x Y K Z Z 9 a K J A D a q E w e t R C Q R y y w a A Q j o h E A R x T C z e + z 0 x x x 0 + I x J 0 + p R b u 8 B m 1 c G P P q Y U b e k E t 3 M h L a u E G X l E L 9 + 2 a W r h f N + z 2 u F G 3 7 I Z V 8 V U V X m r b g G + b N D W Y O l x U I O u P 9 z C a D b z s T S M 5 z i b S w w L I m 2 J + y 6 F w S y S g G s m p j 6 r 7 y 1 o I 2 v B J c Q i 6 g I J G B Q W 6 h I J G D Q W 6 i A K q o i p 0 s 0 b Z 6 z l d R 0 G j k A J d S U G j l A J d S 0 G j m A J d T Q G V U x b e q 2 H 2 A k 6 X V N C o q U A X V d C o q k C X V U B 1 V Y U e 1 i h 7 0 a Y r K 2 i U V q B r K 6 D i y s L 9 G u 5 z + K S G 2 R s 1 X W J B o 8 Y C X W R B o 8 o C X W Z B o 8 4 C X W h B o 9 I C X W p B o 9 Y C X W x B o 9 o C X W 4 B 1 V s W v q l h 9 o Z M l 1 z A a y 5 8 g s A E J I s J e J N 0 C E X 8 q D 5 9 G v r S 9 0 J I o c D c o 9 q R Q L 0 P J i o R u d r N l e l 8 l r + d 3 R X J T D d 0 G l R e I c m j I s I E 6 P S v P 0 I c P O r k p z 8 k U T f B b N n w b b 8 x G f s S H + T d W z i W f W 7 Z n 7 c N J s m G E H 9 s I t q g n o l p z Z t P M / 3 K q t + w a o w 7 i o d Q W d 7 p R j 3 8 u g c e F j I L x r 5 Q H 9 X 6 E 5 n p R y s o n C E 2 P o b N j U 0 9 y K r L 0 w E M w b E z z R a 7 A g k 8 e q y d a a I Y A v N D s W s d + 3 n s B z C v v 8 r p V c D c e + F V 1 + 4 C u / 2 3 5 n X 2 2 2 q O p C f Y h z + 9 J n s 6 5 5 m + 4 R Z V x l a 5 0 T M u 5 v b 1 n E s U E M 7 r F 2 5 N K p A 0 S d W K R p F a P d d M Z C O Z + A 9 k a Y G m H W a N T H 8 J Z d 7 E P f W S x x M 1 / Q / q L U F j c g e 9 O f 8 O 6 q D 3 Z A 8 v / Y K G o B r N G 0 j 8 x y 9 w + 4 u M W Z 4 9 2 Y K N r C R a N X S i u 8 r i U e E n 6 p 3 V e J o V W L M K / 1 F 4 z 3 v f v X q u P g D S X 6 t P U v N F q 8 h R A k J / h f b 8 D u K Y 2 d g 3 p y + 8 d U y F G P e p + u s R g x 4 S 9 T W c K o y N U 2 a t P k f N J q H O n r p O j i Q s a / c i 8 4 Y Z K H f T 6 D 7 K Y R j 5 K 4 0 P m r M i i d W b / v m s 9 9 3 q v I X M U l B c t 4 2 T U 9 3 v V X M / N J k r K m / p p u X Q + + 4 u S k f y s d k 1 9 w v 1 G h n P D l 8 F z B n g i S v 8 E L w o 9 d K s K v I l P K x 4 G + N M q P X J V E E Y j L 1 N f C Z O 4 U v h D b L s f k W F J X v b c 5 y r Q z o r / o Q 6 L 0 I 9 A v z 3 b l l d f c x Q H Z f G M N L v b F o s t W L R T P + 9 w O I c N X W u P h W M Q d 7 5 A 4 y 1 O J s O C v D v n 7 3 9 f K n b /H 8 l n l 5 c v l r p / n n l 9 c n r p W / W q / + P 4 r P O 7 z t / 6 H z V 6 X b + 0 v m m s 9 v p d y 4 6 Q S f t / L 3 z j 8 4 / u / / q / r v 7 f f c / x v T T T 6 o + v + s 4 f 7 r / / R + X X F M 7 < / l a t e x i t > O < l a t e x i t s h a 1 _ b a s e 6 4 = " V 3 B e y D s u v e q B p L f r KP b E H 4 B 1 h w Q = " > A A A y H n i c l V v J c t z I E e V 4 H d O b x j 7 6 g j C l m L F D Y p A a h e 3 j c N + a Z H N f p j U K N D o b D R G b U N V o U h 3 t L / H B F / t T f H P 4 a v + I z 8 6 q Q i G z 0 O i Z s C I k o t 7 L S t T y s j I B Q v 0 8 j o R c W / v P J 9 / 7 / g 9 + + K M f f / q T 5 Z / + 7 O e / + O W z z 3 5 1 L b J x E c B V k M V Z c d v 3 B c R R C l c y k j H c 5 g X 4 S T + G m / 7 D l u J v S i h E l K W X 8 i m H t 4 k f p t E w C n y J 0 L t n z 3 o j X 0 5 7 i S 9 H / e F 0 f z Z 7 9 2 x l b X V N / / H m L 9 a r i 5 W l 6 k / 3 3 W d r / + 0 N s m C c Q C q D 2 B f i 6 / W 1 X L 6 d + o W M g h h m y 7 2 x g N w P H v w Q v s b L 1 E 9 A v J 3 q o c + 8 F 4 g M v G F W 4 N 9 U e h r l P a Z + I s R T 0 k d L N U b R 5 B S 4 i F M e x c t + 0 h i C H P 7 p 7 T R K 8 7 G E N D A j G I 5 j T 2 a e W i B v E B U Q y P g J L / y g i H A S X j D y C z + Q u I z L y y + 8 Y 7 9 4 8 A T a 4 A I K L x t 6 g Z + b a z W N A o Z Q F F E a K o e D q I y E N R t G 4 b g A H G U K k y B L E j 8 d T H s I x j C U s + m 0 B 4 n 3 R Q e v f z e b L c 8 Z B b i 6 U F i z L d 1 S h k 2 7 I g p H t b d z 1 W i z k l l u b S 6 z v M 2 i n 0 m Z J d Z o U 7 f m 7 K q J + 9 b M n x 9 6 Z d K 3 J v 1 F T g J r E S y y G F i L g b 7 N C 2 8 f p x e r K X q + h / Z q y 2 C I c T D w c H E S 1 w d e K 3 D 2 9 f p b 9 N I f e i v r y g l 6 2 d X b Y v Y N B Q E v v T i b Q P E q w Kh a X e 6 h S 7 2 u M F x Z n 5 o t / H M P W 1 P j o K 0 / j j e S f r z q 7 a I e h M R A U N s v 1 J 4 h b 1 z u W p e 7 T Z e a l p P M 3 n T l d X V b 4 V k j D 6 d U N V 7 b H h / G / o C 6 r H y 5 8 m a u 2 8 u 6 j 7 3 6 k r t 6 g 6 5 e e B d G 2 I v W w 9 w M 1 W 9 G X 4 U B G 3 6 r B 7 s i p v e F 7 X 3 R 0 v v c 9 t I B O c n q S F u t V 8 b c X e i l q e N w w d o 0 H Y 4 K A O a S T 0 e Y h Z t 3 S e v G n H 8 5 7 9x P P c B t U J 1 b R A Q f z K S t y e J Z O 3 7 G e Q 6 F p / w Y N z u V m 5 0 2 N x t e 4 U 9 o 4 R v O X r 1 6 5 Z d Z N P D G Q p 1 P 0 d D L M y E i T B X V O u S x j + F T 3 W D x 8 N R 5 m G M 0 t U x S M a Z 7 Z f N / z 7 J y t F U 7 2 v p O R z j p N A R 9 E h t b U c 1 H 4 / W Q U C 2 W t 7 5 e v V q o F B y e H 4 c Z J o B R 0 j J R 5 M z w a q N v n S l z N T f V D e t q o 8 W V 1 b y 9 H 0 6 i 9 r V Q 9 K b X p d N r 4 z t 7 z S 1 r j i d X N X U m Q I W a 8 a q r b 9 s W 0 7 8 p 4 G 7 d v + v 2 t 1 O t b 4 C j V t e L p 1 l p D q J Y 6 T V W F 3 i u o 4 W 6 q h w O 4 y w r N K 2 v D K 8 v K w O k + s l 0 v Z l 0 Z I G x M D O V U e D H 0 + 2 m Q e n H 0 Y A b v D P X R T I 1 1 G z O J Q j Z 3 k E z M 5 o S 5 E L l u l x E c Z Z q W K 0 u O s k S r / S L y M e Y F Z X G Q f p T 5 f t R p l m R o N / n P Y S e z + y K F g 3 a J 6 b v M n 1 i A p c J i B m 4 z G B W 6 6 0 A l w L q N H S Z I T G h y 4 T E j F x m R E z k M h E x 7 1 3 m P R v c g 0 s 9 U K f Y Z e K Z 1 n K R e J H A u M V i e v C k z j y z i y + 9 9 2 M h v U G W f i 4 9 V c q i J p / U A e R s j Z d U v l P X d 0 p 3 z V w m I y Z 3 m Z x N 4 o N L f a B O h c s U x A i X E c y d d C l J n c Y u M y a m d J m S m I n L T I h 5 d J l H Y p 5 c 5 o k N 7 q N L f Z y Z c s t G A S b p r L A x U F a h M q 2 f Z C h 4 6 o H L k Q k V a 6 H b 7 I Y l I 1 m Q l H 2 C W Y S U A c E s P M o B w Q M G A 8 E s L s o h w U M + l J B w F h L l i G A W D + W Y Y B Y M 5 X u C 3 z P 4 g W A W B W V M c M z g h O C E j 5 C t N V / k j G A m 6 T I n O G f w B 4 K Z l s u C Y C b k U h A s + L 4 S L P k I 2 a p w / Z Y E M / G W E 4 K Z c s t H g p l s y y e C n x j 8 k W C r 1 5 0 Y 1 J O x f g I s X O 1 W Z 7 0 R X u s J D U Z 9 r W c 0 G A m 2 n t J g d N h + T o N R Y + t J D U a S r W c 1 G F m 2 n t Z g t N l 6 X i O 3 8 M Q G o 9 L m m W 2 5 h Y c 2 G L 0 2 j 2 3 L J S 6 X O J N v O 5 N r c u G x D E b E z Y P Z c m 0 n c 0 0 u P J z B i L p 5 P F t u 4 f k M R t 2 t J z Q Y i b e e 0 W B 0 3 n p K g x F 7 6 z k N R v H t J z U Y 3 S 8 + q z E i i i i o C 5 Z k g 6 J k g w V s s k n 4 J g V V s k X w F o O 3 C d 5 m 8 A 7 B O w z e J X i X w X s E 7 z F 4 n + B 9 B h 8 Q f M A H f k j 4 I T M / I v i I w R 2 C O w w + J v i Y w S c E n z D 4 l O B T P p Q u 4 V 1 m f k b w G Y P P C T 5 n 8 A X B F w y + J P i S w V c E X z H 4 m u B r P s I b w m + Y + S 3 B t w y + I / i O w f c E 3 5 u 6 u f W k d Z U H R n p M q B t M 4 V p + j N v k 3 J b L b X F u 2 + W 2 O b f j c j u c 2 3 W 5 X c 7 t u d w e j z a t T k b u 8 4 4 H L n f A u U O X O + T c k c s d c a 7 j c h 3 O H b v c M e d O XO 7 E m c S p S 5 7 y j l 2 X 6 3 L u z O X O O H f u c u e c u 3 C 5 C 2 c w l y 5 5 y T t e u d w V 5 6 5 d 7 p p z N y 5 3 w 7 l b l 7 v l 3 J 3 L 3 X H u 3 u V q 6 V / z a r j 8 C P r p A p 9 q 1+ r O Z Z b C 1 D 7 p W i w Z G 6 i X U P q o y 2 S F 1 y W y o S q Y I X 2 D 9 G k 1 d V G C E B U j u h R B h M r h s h o K 1 R 2 6 6 k C E q g 1 d a y B C N Y a u M B C h y k L X F Y h E 7 P b v D U R l h C 4 i E K H i Q Z c O i M R s J Q y S E J I a J G U r a B A q C X R B g A g V A r o M Q I S K W Z 3 7 E a G c r z M + I u x B T C d 6 h C j B l 9 X O s H 0 p D U L J X K d y R C i F 6 w S O C C V u n b Y R o S J V 5 2 p E P r Y d m 8 5 T F Z R + n I / U n u u f t Q z L f i U Q r Q 0 L 0 h M Y v b a o q N h P + g P V w 1 w Q k S U Q K l z / J F j L U k n S B m i J H h H C f x k m o j B R n f V P 6 m z l W 0 m 3 n s p 0 y m c w V Z K 1 L V R s Q C 1 U 6 4 B N a 6 p U a l u o 0 i G 1 U K H h j N 7 Z K H 2 O i M U R R 9 R C W b 5 n g 0 d N P r A F m i o t 1 t O f K h 3 a F q 4 o W 0 r U Y E Y t 1 F 9 O L d T e B 2 q h 7 g q + W l M l u H q R p k p r t o X L P a Y W 6 q y k F m p s Q i 3 U 1 y O 1 U F t P 1 E J d f T R l 3 7 H K u o / V r X X K R b 1 R q t W J F p F N C g C d X x G i v K q z K i K U T X U u R Y R y q M 6 g i F D m 1 H k T E a r l d L J E h J K k T p G I U G r U i R E R S o g 6 H S J C a V A n Q U Q o + e n U h w i l P J 3 w E K E 6 T W c 5 R F i J p p M b Q p T U d E p D h F K Z T m S I U A L T 6 Q s R K s d 0 z k L k k r m + M h C l K J 2 g E K H E p N M S I p S O d D J C h J K Q T k G I 3 D H X 9 w a 6 Z 7 t I a a L P s 0 T S H V X H c g + v 2 A L a U 0 A x n e o k q K d X B b P i L k x A m 9 P p E l K h f i O 8 D U H s F 4 D S G m 2 o 0 w h v a S p A M Y z U S 1 V I g 2 w Q p S F 6 8 8 e x Q s S w v k 5 m U 6 H e B 1 + A X O S g n 8 W D 7 3 L T f 5 x h L C 4 3 X + q m Q v 9 W 0 S T S y q F + o 1 1 N T p q i M x U s B u S m x a j M l F s W 2 6 K F l 9 s W p E i Q O x a j W J C 7 F q N o k H s W o 3 i Q + x a j i J A H F q O Y k I c W o 6 i Q R x a j u J A d i 3 X Y o I 8 t S M E h T y x G 4 S F P L U b l n + x a r M s c n l m Q g k S e W 4 z C R F 5 Y j A J F X l q M y j p 5 Z T G K F X l t M Y o W e W M x i h d 5 a z G K G H l n M R Y z 8 t 6 C V Z 2 G c t 4 r / H x k 6 N A + B g f O g 0 i 4 y W A S R 7 j F Y D o m w 2 0 G k 0 L C H Q a T S M J d B p N O w j 0 G k 1 T C f Q a T W s I D B p N g w k M G k 2 b C I w Y f 0 e K E H Y b T o R o e M 5 j E E 5 4 w m P Q T n j K Y J B R 2 G U z n b H j G 4 D M 2 l H O G k 5 b C C w a T n M J L B p O i w i s G k 6 j C a w a z x + D w h u G k r f C W w S S v 8 I 7 B 9 E A Q 3 j P Y a E z 9 9 l 1 W 9 Z u o 3 7 P 0 + X s W s U k w 6 U t s E U r y E t u E a n W 9 8 L b 1 b z 7 G A j z f E y A 9 v H c M A 2 / n p d e H w F e 4 H E X C m 2 T j e I A Q t s A T + v c k W G G O C 0 9 9 z J P F 6 E h 9 L A O P O V a c + n f A 1 S 9 D x S 7 d k R Q q 9 g g l g Y p 9 Q k m f 4 o B Q 9 p Z G H B J M 8 h R H h N K h J j q E k j b F M a H H z P E J w S R N c U o o K V N 0 C S V h i j N C 6 X w T 5 4 S S K s U F o e w Z V l w S T K I U V 4 S S J s U 1 o X T U i R t C b 5 j j W 4 J J k O K O U N K j u C f U y n E n x V o Q 9 M O F b 1 7 H P F Y F I x U E H f e x Q F W M G 9 R C u W 5 S C 2 W 6 R W W t q h K 3 i c R T b 4 d a q K R d a q G C 9 q i F y t m n F i r m g F o o l E N q o U C O q I X C 6 F A L B X H M B o N C O C E S B X B K L d z 4 L j P F H T 8 j E n f 6 n F q 4 w x f U w o 2 9 p B Z u 6 B W 1 c C O v q Y U b e E M t 3 L d b a u F + 3 b H b 4 0 b d s x t W x V d V e K l t A 7 5 t 0 t R g 6 n B R g a w / 1 M N o N v B L b x L J U T a W H h Z A 3 g T z W w 6 F W y I B 1 U h O f V T d X 9 Z C 0 I Z z x S H o A g o a F R T o E g o a N R T o I g q o i q r Q 7 R p l r + d 0 H Q W N Q g p 0 J Q W N U g p 0 L Q W N Y g p 0 N Q V U T l n 4 o I b Z C z h d U k G j p g J d V E G j q g J d V g H V V R V 6 X K P s R Z u u r K B R W o G u r Y C K K w t 3 a 7 j L 4 b M a Z m / U d I k F j R o L d J E F j S o L d J k F j T o L d K E F j U o L d K k F j V o L d L E F j W o L d L k F V G 9 Z + K 6 G 2 R s y X X I B r 7 n w C Q I T k C z G 4 I 3 T A R T x k / r 0 a e B L 3 w s h h Q J z j 2 p H A v X e H 6 t E 5 G o 3 V 6 a z a f 5 u 2 i u S q W 7 o N K i 8 Q p J H R Y Q J 0 O l f f 4 T Y f 9 L J T 3 9 I o m 6 C 2 b L h 2 3 5 j o j 7 B z W f u L R z L L r f s z t o G k 2 Q D i L 9 t I t q g n o l p z Z p P M 9 3 K q t u w a o w 7 i g d Q W f Z 0 o x 5 + 3 Q M P C 5 k F I 1 + o D 2 j 9 s c z 0 o x U U z h A b H 8 P m x q Y e Z N V l f g A D c O x M s 8 W u Q A K P H m t n m i i G w P y i 2 L W O / T z 2 A 5 j V X + V 0 K m D m v f C q a 3 e B 3 f 4 7 s z r 7 7 T R H 0 h H s w 5 9 O k z 2 f 8 U z f c I s q Y 6 v c 6 B k X M / t 6 z i U K C G f 1 C 7 c m F U i a p G p F w 0 i t n m s m s q F M / E e y t E D T D r N G p r + E M m / i 5 r 3 k 8 V h N / 6 N 6 S 9 C Y 3 F F n x r + D O u r M 7 e G 1 X 9 A Q V K N 5 A 4 k / / A K 3 v 8 i Y 5 c X c F m x l J d G q o R P d T R Y P C z 9 R 7 6 x G k 6 z A m l X 4 T 8 J 7 3 v n m 9 X P 1 A Z D + M n 2 c m i 9 a R Y 4 S E P o r t O c 9 i G N m Y 9 + c v v A 2 M R V i 3 K f q n y c M e k j U 1 3 C q M D Z O m b X 6 H D U b h z p 7 6 j o 5 k v B S u x e Z N 8 h A u Z t E D 1 E O g 8 h f b X z Q n B V J r N 7 0 z 6 a d b 9 Z m L W S W g u L W 2 z g 5 0 f 1 e N / d D k 7 m i 8 p Z u W g 6 d b 3 p R O p R P z a 6 5 X 6 j X y H h 2 + C p g L g B P X O G H 4 E W p l 2 Z V k S / h c d X b G m V C r U + m C s J g 5 G 3 j M 3 E K n w u v n 2 U P q y o s 2 d u e 0 1 w d 0 l n x e 9 R 5 E e o R 4 M / e S 3 X 1 b Y b q u D S G k X 5 n 0 2 K p F Y t m + t 8 F F p e o q U v 1 q W A M s u f 3 M d b i b N I v w H 9 Y f v d s Z b 3 5 / y L m L 6 5 f r 6 7 / Y f X N 2 Z u Vr z a r / z P x 6 d J v l n 6 7 9 M X S + t I f l 7 5 a 2 l / q L l 0 t B U v l 0 l + W / r b 0 9 / W / r v 9 j / Z / r / z K m 3 / u k 6 v P r J e f P + r / / B 2 z v T q s = < / l a t e x i t > Ĥ Output < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 O 3 + 1 Q M u d 3 z y 0 / z U m P N h W e D M D U g = " > A A A B 8 X i c b V D L S g M x F L 1 T X 7 W + q i 7 d B I v g q s y I q M u i G 5 c V 7 E P b U j J p p g 3 N Z I b k j l C G / o U b F 4 q 4 9 W / c + T d m 2 l l o 6 4 H A 4 Z x 7 y b n H j 6 U w 6 L r f T m F l d W 1 9 o 7 h Z 2 t r e 2 d 0 r 7 x 8 0 TZ R o x h s s k p F u + 9 R w K R R v o E D J 2 7 H m N P Q l b / n j m 8 x v P X F t R K T u c R L z X k i H S g S C U b T S Y z e k O P K D 9 G H a L 1 f c q j s D W S Z e T i q Q o 9 4 v f 3 U H E U t C r p B J a k z H c 2 P s p V S j Y J J P S 9 3 E 8 J i y M R 3 y j q W K h t z 0 0 l n i K T m x y o A E k b Z P I Z m p v z d S G h o z C X 0 7 m S U 0 i 1 4 m / u d 1 E g y u e q l Q c Y J c s f l H Q S I J R i Q 7 n w y E 5 g z l x B L K t L B Z C R t R T R n a k k q 2 B G / x 5 G X S P K t 6 F 9 X z u / N K 7 T q v o w h H c A y n 4 M E l 1 O A W 6 t A A B g q e 4 R X e H O O 8 O O / O x 3 y 0 4 O Q 7 h / A H z u c P 0 Q y R B g = = < / l a t e x i t > Y < l a t e x i t s h a 1 _ b a s e 6 4 = " A P c Q O U h v W 0 4 3 g o G B / 0 d 6 c U R q + a o = " > A A A y G n i c l V v J c t z I E e V 4 H d P b j H 3 0 B W F K M W O H x C A 1 C t v H 4 b 4 1 y e a + T E s K N D o b D R G b U N V o U h 3 t 7 / D B F / t T f H P 4 6 o t / x G d n V a G Q W W j 0 T F g R E l H v Z S V q e V m Z A K F + H k d C r q 3 9 5 5 P v f f 8 H P / z R j z / 9 y f J P f / b z X / z y s 8 9 / d S 2 y c R H A V Z D F W X H b 9 w X E U Q p X M p I x 3 O Y F + E k / h p v + w 5 b i b 0 o o R J S l l / I p h z e J H 6 b R M A p 8 i d D b X u L L U X 8 4 7 W W D T M 7 e f b a y t r q m / 3 j z F + v V x c p S 9 a f 7 7 v O 1 / / Y G W T B O I J V B 7 A v x z f p a L t 9 M / U J G Q Q y z 5 d 5 Y Q O 4 H D 3 4 I 3 + B l 6 i c g 3 k z 1 s G f e c 0 Q G 3 j A r 8 G 8 q P Y 3 y H l M / E e I p 6 a O l G q Z o c g p c x C m P 4 k U / a Q x B D v / 0 Z h q l + V h C G p g R D M e x J z N P L Y 4 3 i A o I Z P y E F 3 5 Q R D g J L x j 5 h R 9 I X M L l 5 e f e s V 8 8 e A J t c P G E l w 2 9 w M / N t Z p G A U M o i i g N l c N B V E b C m g 2 j c F w A j j K F S Z A l i Z 8 O p j 0 E Y x j K 2 X T a g 8 T 7 s o P X v 5 v N l u e M A l x d K K z Z l m 4 p w 6 Z d E Y W j 2 t u 5 a r R Z y S y 3 N p d Z 3 m b R z 6 T M E m u 0 q V t z d t X E f W v m z w + 9 M u l b k / 4 i J 4 G 1 C B Z Z D K z F Q N / m u b e P 0 4 v V F D 3 f Q 3 u 1 Z T D E G B h 4 u D i J 6 w O v F T j 7 Z v 0 N e u k P v Z V 1 5 Q S 9 7 O p t M f u G g o A X X p x N o H g Z Y E S t L v f Q p V 5 X G K 6 s T 8 0 W / rm H r a l x 0 N Y f x x t J P 1 7 1 d l E P Q m I g q O 0 X a s + Q N y 5 3 r c v d p k t N y 0 l m b 7 r y q r q t 8 K y R h 1 O q G q 9 s j w 9 j f 0 B d V r 5 a e T 3 X 7 U X d x 1 5 9 x V 2 9 R l f P v Q s j 7 E X r Y W 6 G 6 j e j r 8 K A D b / V g 1 0 R 0 / v C 9 r 5 o 6 X 1 u e + m A n G R 1 p K 3 W K 2 P u L v T S 1 H G 4 Y G 2 a D k c F A H P J p y P M w s 2 7 p H V j z r + a d + 6 n H u A 2 q M 4 t I o I P Z t L W Z P G s H T / j P I f C U 3 6 M m 5 3 K z U 6 b m w 2 v 8 C e 0 8 A 1 n L 1 + + 9 M s s G n h j o c 6 n a O j l m R A R p o l q H f L Y x / C p b r B 4 e O o 8 z D G a W i a p G N O 9 s v m / Z 1 k 5 2 q o d b X 2 n I 5 x 0 G o I + i Y 2 t q O a j 8 X p I q B b L W 1 8 v X y 5 U C g 7 P j 8 M M E 8 A o a Z k o c m Z 4 t d G 3 z p S 5 m p v q h n W 1 0 e L K a t 7 e D y d R + 1 o o e t P r 0 u m 1 8 Z 2 9 5 p Y 1 x 5 O r m j o T o E L N e N X V t 2 2 L 6 d 8 U c L f u 3 3 X 7 2 6 n W N 8 B R q + v F 0 6 w 0 B 1 G s 9 B q r C z z X 0 U J d V Q 6 H c Z Y V m t Z X h t e X l Q F S / W S 6 3 k w 6 s s B Y m E 1 1 c R T 4 8 X S 7 a V D 6 c T T g B u / M d Z F M D T W b c w l C t n f Q z I y m B L l Q u S 4 X U Z y l G l a r i 0 6 y x C v 9 I v I x Z k W l c Z D + V P l + l G l W J O j 3 W Q + h Z z O 7 o k W D 9 o n p u 0 y f m M B l A m I G L j O Y 1 X o r w K W A O g 1 d Z k h M 6 D I h M S O X G R E T u U x E z H u X e c 8 G 9 + B S D 9 Q p d p l 4 p r V c J F 4 k M G 6 x k B 4 8 q T P P 7 O I L 7 / 1 Y S G + Q p V 9 I T 5 W y q M k n d Q A 5 W + M l l e / U 9 Z 3 S X T O X y Y j J X S Z n k / j g U h + o U + E y B T H C Z Q R z J 1 1 K U q e x y 4 y J K V 2 m J G b i M h N i H l 3 m k Z g n l 3 l i g / v o U h 9 n p t y y U Y B J O i t s D J R V q E y n 9 c N M H T z 1 w O X I h I q 1 0 G 1 2 w 5 K R L E j K P s E s Q s q A Y B Y e 5 Y D g A Y O B Y B Y X 5 Z D g I R 9 K S D g L i X J E M I u H c k w w C 4 b y P c H v G f x A M I u C M i Y 4 Z n B C c M J H y N a a L 3 J G M J N 0 m R O c M / g D w U z L Z U E w E 3 I p C B Z 8 X w m W f I R s V b h + S 4 K Z e M s J w U y 5 5 S P B T L b l E 8 F P D P 5 I s N X r T g z q y V g / A R a u d q u z 3 g i v 9 Y Q G o 7 7 W M x q M B F t P a T A 6 b D + n w a i x 9 a Q G I 8 n W s x q M L F t P a z D a b D 2 v k V t 4 Y o N R a f P M t t z C Q x u M X p v H t u U S l 0 u c y b e d y T W 5 8 F g G I + L m w W y 5 t p O 5 J h c e z m B E 3 T y e L b f w f A a j 7 t Y T G o z E W 8 9 o M D p v P a X B i L 3 1 n A a j + P a T G o z u F 5 / V G B F F F N Q F S 7 J B U b L B A j b Z J H y T g i r Z I n i L w d s E b z N 4 h + A d B u 8 S v M v g P Y L 3 G L x P 8 D 6 D D w g + 4 A M / J P y Q m R 8 R f M T g D s E d B h 8 T f M z g E 4 J P G H x K 8 C k f S p f w L j M / I / i M w e c E n z P 4 g u A L B l 8 S f M n g K 4 K v G H x N 8 D U f 4 Q 3 h N 8 z 8 l u B b B t 8 R f M f g e 4 L vl 7 v h 3 K 3 L 3 X L u z u X u O H f v c r X 0 r 3 k 1 X H 4 E / X S B T 7 V r d e c y S 2 F q n 3 Q t l o w N 1 E s o f d R l s s L r E t l Q F c y Q v k H 6 t J q 6 K E G I i h F d i i B C 5 X B Z D Y X q D l 1 1 I E L V h q 4 1 E K E a Q 1 c Y i F B l o e s K R C J 2 + / c G o j J C F x G I U P G g S w d E Y r Y S B k k I S Q 2 S s h U 0 C J U E u i B A h A o B X Q Y g Q s W s z v 2 I U M 7 X G R 8 R 9 i C m E z 1 C l O D L a m f Y v p Q G o W S u U z k i l M J 1 A k e E E r d O 2 4 h Q k a p z N S I f 2 4 5 N 5 6 k K S j / O R 2 r P 9 c 9 a h m W / E o j W h g X p C Y x e W 1 R U 7 C f 9 g e p h L o j I E g g V r n 8 S r G W p J G k D t E S P C O G / D B N R m K j O + i d 1 t v K t p F t P Z T r l M 5 g q y d o W K j a g F q p 1 w K Y 1 V S q 1 L V T p k F q o 0 H B G 7 2 y U P k f E 4 o g j a q E s 3 7 P B o y Y f 2 A J N l R b r 6 U + V D m 0 L V 5 Q t J W o w o x b q L 6 c W a u 8 D t V B 3 B V + t q R J c v U h T p T X b w u U e U w t 1 V l I L N T a h F u r r k V q o r S d q o a 4 + m r L v W G X d x + r W O u W i 3 i j V 6 k S L y C Y F g M 6 v C F F e 1 V k V E c q m O p c i Q j l U Z 1 B E K H P q v I k I 1 X I 6 W S J C S V K n S E Q o N e r E i A g l R J 0 O E a E 0 q J M g I p T 8 d O p D h F K e T n i I U J 2 m s x w i r E T T y Q 0 h S m o 6 p S F C q U w n M k Q o g e n 0 h Q i V Y z p n I X L J X F 8 Z i F K U T l C I U G L S a Q k R S k c 6 G S F C S U i n I E T u m O t 7 A 9 2 z X a Q 0 0 e d Z I u m O q m O 5 h 1 d s A e 0 p o J h O d R L U 0 6 u C W X E X J q D N 6 X Q J q V C / E d 6 G I P Y L Q G m N N t R p h L c 0 F a A Y R u q l K q R B N o j S E L 3 5 4 1 g h Y l h f J 7 O p U O + D L 0 A u c t D P 4 s F 3 u e k / z j A W l 5 s v d V O h f 6 t o E m n l U L / R r i Y n T d G Z C h Y D c t N i V G b K L Y t t 0 c L L b Q t S J M g d i 1 E s y F 2 L U T T I P Y t R P M h 9 i 1 F E y A O L U U z I Q 4 t R V M g j i 1 F c y I 7 F O m z Q x x a k 4 J A n F q P w k K c W o / J P d i 3 W Z Q 7 P L E h B I s 8 t R m E i L y x G g S I v L U Z l n b y y G M W K v L Y Y R Y u 8 s R j F i 7 y 1 G E W M v L M Y i x l 5 b 8 G q T k M 5 7 x V + P j J 0 a B + D A + d B J N x k M I k j 3 G I w H Z P h N o N J I e E O g 0 k k 4 S 6 D S S f h H o N J K u E + g 0 k t 4 Q G D S T D h I Y N J M + E R g 4 9 o c c I O w + l Q D Y 8 Z T O I J T x h M + g l P G U w S C r s M p n M 2 P G P w G R v K O c N J S + E F g 0 l O 4 S W D S V H h F Y N J V O E 1 g 9 l j c H j D c N J W e M t g k l d 4 x 2 B 6 I A j v G W w 0 p n 7 7 L q v 6 T d T v W f r 8 P Y v Y J J j 0 J b Y I J X m J b U K 1 u p 5 7 2 / o 3 H 2 M B n u 8 J k B 7 e O 4 a B t / P C 6 0 P g K 1 y O I u F N s n E 8 Q A h b 4 A n 9 e x K s M M e F p z 7 m y W J 0 p D 6 W g c c c K 0 7 9 O + D q l 6 F i l + 5 I C h V 7 h J J A x T 6 h p E 9 x Q C h 7 S y M O C S Z 5 i i N C 6 V A T H U J J m + K Y 0 G P m + I R g k q Y 4 J Z S U K b q E k j D F G a F 0 v o l z Q k m V 4 o J Q 9 g w r L g k m U Y o r Q k m T 4 p p Q O u r E D a E 3 z P E t w S R I c U c o 6 V H c E 2 r l u J N i L Q j 6 4 c I 3 r 2 M e q 4 K R C o K O + 1 i g K s Y N a q F c N 6 m F M t 2 i s l Z V i d t E 4 q m 3 Q y 1 U 0 i 6 1 U E F 7 1 E L l 7 F M L F X N A L R T K I b V Q I E f U Q m F 0 q I W C O G a D Q S G c E I k C O K U W b n y X m e K O n x G J O 3 1 O L d z h C 2 r h x l 5 S C z f 0 i l q 4 k d f U w g 2 8 o R b u 2 y 2 1 c L / u 2 O 1 x o + 7 Z D a v i q y q 8 1 L Y B 3 z Z p a j B 1 u K h A 1 h / q Y T Q b + I U 3 i e Q o G 0 s P C y B v g v k t h 8 I t k Y B q J K c + q u 4 v a y F o w 7 n i E H Q B B Y 0 K C n Q J B Y 0 a C n Q R B V R F V e h 2 j b L X c 7 q O g k Y h B b q S g k Y p B b q W g k Y x B b q a A i q n L H x Q w + w F n C 6 p o F F T g S 6 q o F F V g S 6 r g O q q C j 2 u U f a i T V d W 0 C i t Q N d W Q M W V h b s 1 3 O X w W Q 2 z N 2 q 6 x I J G j Q W 6 y I J G l Q W 6 z I J G n Q W 6 0 I J G p Q W 6 1 I J G r Q W 6 2 I J G t Q W 6 3 A K q t y x 8 V 8 P s D Z k u u Y D X X P g E g Q l I F m P w x u k A i v h J f f o 0 8 K X v h Z B C g b l H t S O B e u + P V S J y t Z s r 0 9 k 0 f z f t F c l U N 3 Q a V F 4 h y a M i w g T o 9 K 8 / Q u w / 6 e S n P y R R N 8 F s 2 f B t v z E Z + R I f 5 N 1 b O J Z d b t m d t Q 0 m y Q Y Q f 9 t E t E E 9 E 9 O a N Z 9 m u p V V t 2 H V G H c U D 6 C y 7 O l G P f y 6 B x 4 W M g t G v l A f 0 P p j m e l H K y i c I T Y + h s 2 N T T 3 I q s v 8 A A b g 2 J l m i 1 2 B B B 4 9 1 s 4 0 U Q y B + U W x a x 3 7 e e w H M K u / y u l U w M x 7 7 l X X 7 g K 7 / X d m d f b b a Y 6 k I 9 i H P 5 0 m e z 7 j m b 7 h F l X G V r n R M y 5 m 9 v W c S x Q Q z u o X b k 0 q k D R J 1 Y q G k V o 9 1 0 x k Q 5 n 4 j 2 R p g a Y d Z o 1 M f w l l 3 s T N e 8 n j s Z r + R / W W o D G 5 o 8 6 M f w d 1 1 J n b w 2 u / o C G o R v M G E n / 4 B W 5 / k T H L i 7 k t 2 M p K o l V D J 7 q b L B 4 W f q L e W Y 0 m W Y E 1 q / C f h P e s 8 / b V M / U B k P 4 y f Z y a L 1 p F j h I Q + i u 0 Z z 2 I Y 2 Z j 3 5 w + 9 z Y x F W L c p + q f J w x 6 S N T X c K o w N k 6 Z t f o c N R u H O n v q O j m S 8 E K 7 F 5 k 3 y E C 5 m 0 Q P U Q 6 D y F 9 t f N C c F U m s 3 v T P p p 2 3 a 7 M W M k t B c e t t n J z o f q + a + 6 H J X F F 5 S z c t h 8 7 b X p Q O 5 V O z a + 4 X 6 j U y n h 2 + C p g L w B N X + C F 4 U e q l W V X k S 3 h c 9 b Z G m V D r k 6 m C M B h 5 2 / h M n M I X w u t n 2 c O q C k v 2 t u c 0 V 4 d 0 V v w e d V 6 E e g T 4 s / d C X X 2 b o T o u j W G k 3 9 m 0 W G r F o p n + d 4 H F J W r q U n 0 q G I P s + X 2 M t T i b 9 A v w H 5 b f f b a y 3 v x / E f M X 1 6 9 W 1 / + w + v r s 9 c r X m 9 X / m f h 0 6 T d L v 1 3 6 c m l 9 6 Y 9 L X y / t L 3 W X r p a C p W L p L 0 t / W / r 7 + l / X / 7 H + z / V / G d P v f V L 1 + f W S 8 2 f 9 3 / 8 D z W F N n w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 g + g L f V o a w Q t 1 9 v U f Z I 7 Y m K 0 5 f I = " > A A A y K n i c l V t Z k 9 v G E a a d y 1 E u O 3 n M C y o r l Z 3 U a m s p q 5 I 8 e u + L u 8 u 9 D 6 + s A s E m C C 0 u Y Y b g r l j M r 8 h r 8 g v y a / L m y m t + S H p m M O g e L C h X V G U J 8 3 0 9 j T m + n m 6 A 8 C C P I y F X V 7 / / 5 N M f / f g n P / 3 Z Z z 9 / 9 o t f / u r X v / n 8 i 9 9 e i m x S B H A R Z H F W X A 9 8 A X G U w o W M Z A z X e Q F + M o j h a n C / o f i r E g o R Z e m 5 f M z h T e K H a T S K A l 8 i d H u X + H I 8 G M 0 e 5 m 8 / X 1 p d W d V / v K c X 3 e p i q V P 9 6 b / 9 o v v s b p g F k w R S G c S + E N 9 2 V 3 P 5 Z u Y X M g p i m D + 7 m w j I / e D e D + F b v E z 9 B M S b m R 7 y 3 H u B y N A b Z Q X + l 0 p P o 7 z H z E + E e E w G a K m G K J q c A h d x y q N Y H i S N I c j R X 9 / M o j S f S E g D M 4 L R J P Z k 5 q m F 8 Y Z R A Y G M H / H C D 4 o I J + E F Y 7 / w A 4 n L 5 9 z i w c z h 2 b M X 3 q F f 3 H s C + + F i C i 8 b e Y G f m 2 s 1 t Q J G U B R R G q q b D K M y E t Z s F I W T A t B t C t M g S x I / H c 7 u E I x h J O e z 2 R 0 k 3 l c 9 v P 7 j H G / T N A p w x a G w Z h u 6 p Q y b d k U U j m t v p 6 r R Z i W z 3 N q c Z 3 m b x S C T M k u s 0 b p u P b G r J u 5 b M / / p 0 C u T g T U Z L H I S W I t g k c X Q W g z 1 b V 5 4 u z i 9 W E 3 R 8 z 2 0 V 9 s I I 4 y J o Y e L k 7 g + 8 F q B 8 2 + 7 b 9 D L Y O Q t d Z U T 9 L K t t 8 X s G 4 o E l r 0 4 m 0 L x M s A I W 0 G 5 w 0 i v K 4 y W u j O z h X + 7 w 9 b M O G j r j + O N p B + v e N u o B y E x O N T 2 C 7 V n y B u X 2 9 b l d t O l p u U 0 s z d d e l X d V n j W y M M p V Y 1 X t s f 7 i T + k L k t f L 7 1 + 0 m 2 5 7 m O v v u a u X q O r F 9 6 Z E f a i 9 T A 3 Q / W b 0 V d h w I b f 6 s G u i O l 9 Z n u f t f Q + t b 1 0 k E 6 z O t J W 6 p U x d x d 6 a e o 4 X L A 2 T Y f j A o C 5 5 N M R Z u G e u q R 1 Y 8 6 / f u r c T z 3 A b V C d W 0 Q E 7 8 2 k r c n i W T t + J n k O h a f 8 G D d b l Z u t N j d r X u F P a e E b z l 6 + f O m X W T T 0 J k K d T 9 H I y z M h I k w b 1 T r k s Y / h U 9 1 g 8 f D U G Z l j N L V M U j G m e 2 X z f 8 + y c r R R O 9 r 4 Q U c 4 6 T Q E f R I b W 1 H N R + P 1 k F A t l r e + X r 5 c q B Q c n h + H G S a F c d I y U e T M 8 G q j j 8 6 U u X o y 1 T X r a q 3 F l d W 8 v R 9 O o v a 1 U P S m 1 7 n T a + 0 H e z 1 Z 1 h x P r m r q T I A K N e N V V x / b F t O / K e B + 3 b / v 9 r d T r W + A o 1 b X i 6 d Z a Q 6 i W O k 1 V h d 4 r q O F u q o c j u I s K z S t r w y v L y s D p A b J r N t M O r L A W J j P d L E U + P F s s 2 l Q + n E 0 5 A Z v z X W R z A w 1 f + I S h G z v o J k 5 T Q l y o X J d L q I 4 S z W s V h e d Z I l X + k X k Y 8 y K S u M g / Z n y / S D T r E j Q 7 / M 7 h J 7 P 7 Y o W D d o n Z u A y A 2 I C l w m I G b r M c F 7 r r Q C X A u o 0 c p k R M a H L h M S M X W Z M T O Q y E T H v X O Y d G 9 y 9 S 9 1 T p 9 h l 4 r n W c p F 4 k c C 4 x c J 6 + K j O P L O L y 9 6 7 i Z D e M E u / l J 4 q b 1 G T j + o A c r b G S y r f q e s 7 p b t m L p M R k 7 t M z i b x 3 q X e U 6 f C Z Q p i h M s I 5 k 6 6 l K R O E 5 e Z E F O 6 T E n M 1 G W m x D y 4 z A M x j y 7 z y A b 3 w a U + z E 2 5 Z a M A k 3 R W 2 B g o q 1 C Z z e z D D Q V P P X A 5 N q F i L X S b 3 b B k J A u S c k A w i 5 A y I J i F R z k k e M h g I J j F R T k i e M S H E h L O Q q I c E 8 z i o Z w Q z I K h f E f w O w b f E 8 y i o I w J j h m c E J z w E b K 1 5 o u c E c w k X e Y E 5 w x + T z D T c l k Q z I R c C o I F 3 1 e C J R 8 h W x W u 3 5 J g J t 5 y S j B T b v l A M J N t + U j w I 4 M / E G z 1 u h W D e l r W T 4 C F q 9 3 q r D f C a z 2 h w a i v 9 Y w G I 8 H W U x q M D t v P a T B q b D 2 p w U i y 9 a w G I 8 v W 0 x q M N l v P a + Q W n t h g V N o 8 s y 2 3 8 N A G o 9 f m s W 2 5 x O U S Z / J t Z 3 J N L j y W w Y i 4 e T B b r u 1 k r s m F h z M Y U T e P Z 8 s t P J / B q L v 1 h A Y j 8 d Y z G o z O W 0 9 p M G J v P a f B K L 7 9 p A a j + 8 V n N U Z E E Q V 1 w Z K s U Z S s s Y B N 1 g l f p 6 B K N g j e Y P A m w Z s M 3 i J 4 i 8 H b B G 8 z e I f g H Q b v E r z L 4 D 2 C 9 / j A 9 w n f Z + Y H B B 8 w u E d w j 8 G H B B 8 y + I j g I w Y f E 3 z M h 9 I n v M / M T w g + Y f A p w a c M P i P 4 j M H n B J 8 z + I L g C w Z f E n z J R 3 h F + B U z vv g 3 K X L X X L u y u W u O H f t c t e c u 3 G 5 G 8 7 d u l w t / U t e D Z c f Q D 9 d 4 F P t a t 2 5 z F K Y 2 S d d i y U T A 9 0 l l D 7 q M l n h d Y l s q A p m y M A g A 1 p N X Z Q g R M W I L k U Q o X K 4 r I Z C d Y e u O h C h a k P X G o h Q j a E r D E S o s t B 1 B S I R u / 0 7 A 1 E Z o Y s I R K h 4 0 K U D I j F b C Y M k h K Q G S d k K G o R K A l 0 Q I E K F g C 4 D E K F i V u d + R C j n 6 4 y P C H s Q 0 4 k e I U r w Z b U z b F 9 K g 1 A y 1 6 k c E U r h O o E j Q o l b p 2 1 E q E j V u R q R D 2 3 H p v N U B a U f 5 2 O 1 5 / r f W o b l o B K I 1 o Y F 6 Q m M X l t U V O w n g 6 H q Y S 6 I y B I I F a 7 / J V j L U k n S B m i J H h H C v x k m o j B R n f W / 1 N n K t 5 J u P Z X Z j M 9 g p i R r W 6 j Y g F q o 1 i G b 1 k y p 1 L Z Q p S N q o U L D O b 2 z U f o c E 4 s j j q i F s n z H B o + a v G c L N F N a r K c / U z q 0 L V x R t p S o w Y x a q L + c W q i 9 9 9 R C 3 R V 8 t W Z K c P U i z Z T W b A u X e 0 I t 1 F l J L d T Y l F q o r w d q o b Y e q Y W 6 + j C v f i D D r P t Q 3 V q n X N Q b p V q d a B F Z p w D Q + R U h y q s 6 q y J C 2 V T n U k Q o h + o M i g h l T p 0 3 E a F a T i d L R C h J 6 h S J C K V G n R g R o Y S o 0 y E i l A Z 1 E k S E k p 9 O f Y h Q y t M J D x G q 0 3 S W Q 4 S V a D q 5 I U R J T a c 0 R C i V 6 U S G C C U w n b 4 Q o X J M 5 y x E z p n r C w N R i t I J C h F K T D o t I U L p S C c j R C g J 6 R S E y A 1 z f W u g W 7 a L l C Y G P E s k / X F 1 L N / h F V t A e w o o p l e d B P X 0 q m B W 3 J k J a H M 6 n U M q 1 K / E m x D E f g E o r f G a O o 3 w l q Y C F K N I v V S F N M i G U R q i N 3 8 S K 0 S M 6 u t k P h P q f f A Z y E U O B l k 8 / C E 3 g 4 c 5 x u K z 5 k v d V O h f F U 0 i r R z q N 9 r V 5 K Q p O l P B Y k C u W 4 z K T L l h s Q 1 a e L l p Q Y o E u W U x i g W 5 b T G K B r l j M Y o H u W s x i g i 5 Z z G K C b l v M Y o K e W A x i g v Z s 1 i P D f r Q g h Q c 8 s h i F B 7 y 2 G J U / s m + x f r M 4 Y k F K U j k q c U o T O S Z x S h Q 5 L n F q K y T F x a j W J G X F q N o k V c W o 3 i R 1 x a j i J E 3 F m M x I 2 8 t W N V p K O e d w s / H h g 7 t Y 3 D g P I i E 6 w w m c Y Q b D K Z j M t x k M C k k 3 G I w i S T c Z j D p J N x h M E k l 3 G U w q S X c Y z A J J t x n M G k m P G D w A S 1 O 2 G M 4 H a r h I Y N J P O E R g 0 k / 4 T G D S U J h n 8 F 0 z o Y n D D 5 h Q z l l O G k p P G M w y S k 8 Z z A p K r x g M I k q v G Q w e w w O r x h O 2 g q v G U z y C m 8 Y T A 8 E 4 S 2 D j c b U r + + y q t 9 E / Z 5 l w N + z i H W C S V 9 i g 1 C S l 9 g k V K v r h b e p f / m Y C P B 8 T 4 D 0 8 N 4 x D L 2 t Z W 8 A g a 9 w O Y 6 E N 8 0 m 8 R A h b I E n 9 O 8 k W G F O C k 9 9 4 J P F 6 E h 9 L A M P O V a c + j f g 6 s d Q s U 1 3 J I W K H U J J o G K X U N K n 2 C O U v a U R + w S T P M U B o X S o i R 6 h p E 1 x S O g h c 3 x E M E l T H B N K y h R 9 Q k m Y 4 o R Q O t / E K a G k S n F G K H u G F e c E k y j F B a G k S X F J K B 1 1 4 o r Q K + b 4 m m A S p L g h l P Q o b g m 1 c t x K s R Y E / X D h m 9 c x D 1 X B S A V B z 3 0 s U B X j G r V Q r u v U Q p l u U F m r q s R N I v H U 2 6 I W K m m b W q i g H W q h c n a p h Y r Z o x Y K Z Z 9 a K J A D a q E w e t R C Q R y y w a A Q j o h E A R x T C z e + z 0 x x x 0 + I x J 0 + p R b u 8 B m 1 c G P P q Y U b e k E t 3 M h L a u E G X l E L 9 + 2 a W r h f N + z 2 u F G 3 7 I Z V 8 V U V X m r b g G + b N D W Y O l x U I O u P 9 z C a D b z s T S M 5 z i b S w w L I m 2 J + y 6 F w S y S g G s m p j 6 r 7 y 1 o I 2 v B J c Q i 6 g I J G B Q W 6 h I J G D Q W 6 i A K q o i p 0 s 0 b Z 6 z l d R 0 G j k A J d S U G j l A J d S 0 G j m A J d T Q G V U x b e q 2 H 2 A k 6 X V N C o q U A X V d C o q k C X V U B 1 V Y U e 1 i h 7 0 a Y r K 2 i U V q B r K 6 D i y s L 9 G u 5 z + K S G 2 R s 1 X W J B o 8 Y C X W R B o 8 o C X W Z B o 8 4 C X W h B o 9 I C X W p B o 9 Y C X W x B o 9 o C X W 4 B 1 V s W v q l h 9 o Z M l 1 z A a y 5 8 g s A E J I s J e J N 0 C E X 8 q D 5 9 G v r S 9 0 J I o c D c o 9 q R Q L 0 P J i o R u d r N l e l 8 l r + d 3 R X J T D d 0 G l R e I c m j I s I E 6 P S v P 0 I c P O r k p z 8 k U T f B b N n w b b 8 x G f s S H + T d W z i W f W 7 Z n 7 c N J s m G E H 9 s I t q g n o l p z Z t P M / 3 K q t + w a o w 7 i o d Q W d 7 p R j 3 8 u g c e F j I L x r 5 Q H 9 X 6 E 5 n p R y s o n C E 2 P o b N j U 0 9 y K r L 0 w E M w b E z z R a 7 A g k 8 e q y d a a I Y A v N D s W s d + 3 n s B z C v v 8 r p V c D c e + F V 1 + 4 C u / 2 3 5 n X 2 2 2 q O p C f Y h z + 9 J n s 6 5 5 m + 4 R Z V x l a 5 0 T M u 5 v b 1 n E s U E M 7 r F 2 5 N K p A 0 S d W K R p F a P d d M Z C O Z + A 9 k a Y G m H W a N T H 8 J Z d 7 E P f W S x x M 1 / Q / q L U F j c g e 9 O f 8 O 6 q D 3 Z A 8 v / Y K G o B r N G 0 j 8 x y 9 w + 4 u M W Z 4 9 2 Y K N r C R a N X S i u 8 r i U e E n 6 p 3 V e J o V W L M K / 1 F 4 z 3 v f v X q u P g D S X 6 t P U v N F q 8 h R A k J / h f b 8 D u K Y 2 d g 3 p y + 8 d U y F G P e p + u s R g x 4 S 9 T W c K o y N U 2 a t P k f N J q H O n r p O j i Q s a / c i 8 4 Y Z K H f T 6 D 7 K Y R j 5 K 4 0 P m r M i i d W b / v m s 9 9 3 q v I X M U l B c t 4 2 T U 9 3 v V X M / N J k r K m / p p u X Q + + 4 u S k f y s d k 1 9 w v 1 G h n P D l 8 F z B n g i S v 8 E L w o 9 d K s K v I l P K x 4 G + N M q P X J V E E Y j L 1 N f C Z O 4 U v h D b L s f k W F J X v b c 5 y r Q z o r / o Q 6 L 0 I 9 A v z 3 b l l d f c x Q H Z f G M N L v b F o s t W L R T P + 9 w O I c N X W u P h W M Q d 7 5 A 4 y 1 O J s O C v D v n 7 3 9 f K n b / H 8 l n l 5 c v l r p / n n l 9 c n r p W / W q / + P 4 r P O 7 z t / 6 H z V 6 X b + 0 v m m s 9 v p d y 4 6 Q S f t / L 3 z j 8 4 / u / / q / r v 7 f f c / x v T T T 6 o + v + s 4 f 7 r / / R + P Q l N k < / l a t e x i t > x < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 O 3 + 1 Q M u d 3 z y 0 / z U m P N h W e D M D U g = " > A A A B 8 X i c b V D L S g M x F L 1 T X 7 W + q i 7 d B I v g q s y I q M u i G 5 c V 7 E P b U j J p p g 3 N Z I b k j l C G / o U b F 4 q 4 9 W / c + T d m 2 l l o 6 4 H A 4 Z x 7 y b n H j 6 U w 6 L r f T m F l d W 1 9 o 7 h Z 2 t r e 2 d 0 r 7 x 8 0 T Z R o x h s s k p F u + 9 R w K R R v o E D J 2 7 H m N P Q l b / n j m 8 x v P X F t R K T u c R L z X k i H S g S C U b T S Y z e k O P K D 9 G H a L 1 f c q j s D W S Z e T i q Q o 9 4 v f 3 U H E U t C r p B J a k z H c 2 P s p V S j Y J J P S 9 3 E 8 J i y M R 3 y j q W K h t z 0 0 l n i K T m x y o A E k b Z P I Z m p v z d S G h o z C X 0 7 m S U 0 i 1 4 m / u d 1 E g y u e q l Q c Y J c s f l H Q S I J R i Q 7 n w y E 5 g z l x B L K t L B Z C R t R T R n a k k q 2 B G / x 5 G X S P K t 6 F 9 X z u / N K 7 T q v o w h H c A y n 4 M E l 1 O A W 6 t A A B g q e 4 R X e H O O 8 O O / O x 3 y 0 4 O Q 7 h / A H z u c P 0 Q y R B g = = < / l a t e x i t > Y Layer Norm Attention + + Layer Norm FFN + < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 g + g L f V o a w Q t 1 9 v U f Z I 7 Y m K 0 5 f I = " > A A A y K n i c l V t Z k 9 v G E a a d y 1 E u O 3 n M C y o r l Z 3 U a m s p q 5 I 8 e u + L u 8 u 9 D 6 + s A s E m C C 0 u Y Y b g r l j M r 8 h r 8 g v y a / L m y m t + S H p m M O g e L C h X V G U J 8 3 0 9 j T m + n m 6 A 8 C C P I y F X V 7 / / 5 N M f / f g n P / 3 Z Z z 9 / 9 o t f / u r X v / n 8 i 9 9 e i m x S B H A R Z H F W X A 9 8 A X G U w o W M Z A z X e Q F + M o j h a n C / o f i r E g o R Z e m 5 f M z h T e K H a T S K A l 8 i d H u X + H I 8 G M 0 e 5 m 8 / X 1 p d W d V / v K c X 3 e p i q V P 9 6 b / 9 o v v s b p g F k w R S G c S + E N 9 2 V 3 P 5 Z u Y X M g p i m D + 7 m w j I / e D e D + F b v E z 9 B M S b m R 7 y 3 H u B y N A b Z Q X + l 0 p P o 7 z H z E + E e E w G a K m G K J q c A h d x y q N Y H i S N I c j R X 9 / M o j S f S E g D M 4 L R J P Z k 5 q m F 8 Y Z R A Y G M H / H C D 4 o I J + E F Y 7 / w A 4 n L 5 9 z i w c z h 2 b M X 3 q F f 3 H s C + + F i C i 8 b e Y G f m 2 s 1 t Q J G U B R R G q q b D K M y E t Z s F I W T A t B t C t M g S x I / H c 7 u E I x h J O e z 2 R 0 k 3 l c 9 v P 7 j H G / T N A p w x a G w Z h u 6 p Q y b d k U U j m t v p 6 r R Z i W z 3 N q c Z 3 m b x S C T M k u s 0 b p u P b G r J u 5 b M / / p 0 C u T g T U Z L H I S W I t g k c X Q W g z 1 b V 5 4 u z i 9 W E 3 R 8 z 2 0 V 9 s I I 4 y J o Y e L k 7 g + 8 F q B 8 2 + 7 b 9 D L Y O Q t d Z U T 9 L K t t 8 X s G 4 o E l r 0 4 m 0 L x M s A I W 0 G 5 w 0 i v K 4 y W u j O z h X + 7 w 9 b M O G j r j + O N p B + v e N u o B y E x O N T 2 C 7 V n y B u X 2 9 b l d t O l p u U 0 s z d d e l X d V n j W y M M p V Y 1 X t s f 7 i T + k L k t f L 7 1 + 0 m 2 5 7 m O v v u a u X q O r F 9 6 Z E f a i 9 T A 3 Q / W b 0 V d h w I b f 6 s G u i O l 9 Z n u f t f Q + t b 1 0 k E 6 z O t J W 6 p U x d x d 6 a e o 4 X L A 2 T Y f j A o C 5 5 N M R Z u G e u q R 1 Y 8 6 / f u r c T z 3 A b V C d W 0 Q E 7 8 2 k r c n i W T t + J n k O h a f 8 G D d b l Z u t N j d r X u F P a e E b z l 6 + f O m X W T T 0 J k K d T 9 H I y z M h I k w b 1 T r k s Y / h U 9 1 g 8 f D U G Z l j N L V M U j G m e 2 X z f 8 + y c r R R O 9 r 4 Q U c 4 6 T Q E f R I b W 1 H N R + P 1 k F A t l r e + X r 5 c q B Q c n h + H G S a F c d I y U e T M 8 G q j j 8 6 U u X o y 1 T X r a q 3 F l d W 8 v R 9 O o v a 1 U P S m 1 7 n T a + 0 H e z 1 Z 1 h x P r m r q T I A K N e N V V x / b F t O / K e B + 3 b / v 9 r d T r W + A o 1 b X i 6 d Z a Q 6 i W O k 1 V h d 4 r q O F u q o c j u I s K z S t r w y v L y s D p A b J r N t M O r L A W J j P d L E U + P F s s 2 l Q + n E 0 5 A Z v z X W R z A w 1 f + I S h G z v o J k 5 T Q l y o X J d L q I 4 S z W s V h e d Z I l X + k X k Y 8 y K S u M g / Z n y / S D T r E j Q 7 / M 7 h J 7 P 7 Y o W D d o n Z u A y A 2 I C l w m I G b r M c F 7 r r Q C X A u o 0 c p k R M a H L h M S M X W Z M T O Q y E T H v X O Y d G 9 y 9 S 9 1 T p 9 h l 4 r n W c p F 4 k c C 4 x c J 6 + K j O P L O L y 9 6 7 i Z D e M E u / l J 4 q b 1 G T j + o A c r b G S y r f q e s 7 p b t m L p M R k 7 t M z i b x 3 q X e U 6 f C Z Q p i h M s I 5 k 6 6 l K R O E 5 e Z E F O 6 T E n M 1 G W m x D y 4 z A M x j y 7 z y A b 3 w a U + z E 2 5 Z a M A k 3 R W 2 B g o q 1 C Z z e z D D Q V P P X A 5 N q F i L X S b 3 b B k J A u S c k A w i 5 A y I J i F R z k k e M h g I J j F R T k i e M S H E h L O Q q I c E 8 z i o Z w Q z I K h f E f w O w b f E 8 y i o I w J j h m c E J z w E b K 1 5 o u c E c w k X e Y E 5 w x + T z D T c l k Q z I R c C o I F 3 1 e C J R 8 h W x W u 3 5 J g J t 5 y S j B T b v l A M J N t + U j w I 4 M / E G z 1 u h W D e l r W T 4 C F q 9 3 q r D f C a z 2 h w a i v 9 Y w G I 8 H W U x q M D t v P a T B q b D 2 p w U i y 9 a w G I 8 v W 0 x q M N l v P a + Q W n t h g V N o 8 s y 2 3 8 N A G o 9 f m s W 2 5 x O U S Z / J t Z 3 J N L j y W w Y i 4 e T B b r u 1 k r s m F h z M Y U T e P Z 8 s t P J / B q L v 1 h A Y j 8 d Y z G o z O W 0 9 p M G J v P a f B K L 7 9 p A a j + 8 V n N U Z E E Q V 1 w Z K s U Z S s s Y B N 1 g l f p 6 B K N g j e Y P A m w Z s M 3 i J 4 i 8 H b B G 8 z e I f g H Q b v E r z L 4 D 2 C 9 / j A 9 w n f Z + Y H B B 8 w u E d w j 8 G H B B 8 y + I j g I w Y f E 3 z M h 9 I n v M / M T w g + Y f A p w a c M P i P 4 j M H n B J 8 z + I L g C w Z f E n z J R 3 h F + B U z v y b 4 m s E 3 B N 8 w + J b g W 1 M 3 t 5 6 0 r v L A S I 8 J d Y 0 p X M u P cv g 3 K X L X X L u y u W u O H f t c t e c u 3 G 5 G 8 7 d u l w t / U t e D Z c f Q D 9 d 4 F P t a t 2 5 z F K Y 2 S d d i y U T A 9 0 l l D 7 q M l n h d Y l s q A p m y M A g A 1 p N X Z Q g R M W I L k U Q o X K 4 r I Z C d Y e u O h C h a k P X G o h Q j a E r D E S o s t B 1 B S I R u / 0 7 A 1 E Z o Y s I R K h 4 0 K U D I j F b C Y M k h K Q G S d k K G o R K A l 0 Q I E K F g C 4 D E K F i V u d + R C j n 6 4 y P C H s Q 0 4 k e I U r w Z b U z b F 9 K g 1 A y 1 6 k c E U r h O o E j Q o l b p 2 1 E q E j V u R q R D 2 3 H p v N U B a U f 5 2 O 1 5 / r f W o b l o B K I 1 o Y F 6 Q m M X l t U V O w n g 6 H q Y S 6 I y B I I F a 7 / J V j L U k n S B m i J H h H C v x k m o j B R n f W / 1 N n K t 5 J u P Z X Z j M 9 g p i R r W 6 j Y g F q o 1 i G b 1 k y p 1 L Z Q p S N q o U L D O b 2 z U f o c E 4 s j j q i F s n z H B o + a v G c L N F N a r K c / U z q 0 L V x R t p S o w Y x a q L + c W q i 9 9 9 R C 3 R V 8 t W Z K c P U i z Z T W b A u X e 0 I t 1 F l J L d T Y l F q o r w d q o b Y e q Y W 6 + j C v f i D D r P t Q 3 V q n X N Q b p V q d a B F Z p w D Q + R U h y q s 6 q y J C 2 V T n U k Q o h + o M i g h l T p 0 3 E a F a T i d L R C h J 6 h S J C K V G n R g R o Y S o 0 y E i l A Z 1 E k S E k p 9 O f Y h Q y t M J D x G q 0 3 S W Q 4 S V a D q 5 I U R J T a c 0 R C i V 6 U S G C C U w n b 4 Q o X J M 5 y x E z p n r C w N R i t I J C h F K T D o t I U L p S C c j R C g J 6 R S E y A 1 z f W u g W 7 a L l C Y G P E s k / X F 1 L N / h F V t A e w o o p l e d B P X 0 q m B W 3 J k J a H M 6 n U M q 1 K / E m x D E f g E o r f G a O o 3 w l q Y C F K N I v V S F N M i G U R q i N 3 8 S K 0 S M 6 u t k P h P q f f A Z y E U O B l k 8 / C E 3 g 4 c 5 x u K z 5 k v d V O h f F U 0 i r R z q N 9 r V 5 K Q p O l P B Y k C u W 4 z K T L l h s Q 1 a e L l p Q Y o E u W U x i g W 5 b T G K B r l j M Y o H u W s x i g i 5 Z z G K C b l v M Y o K e W A x i g v Z s 1 i P D f r Q g h Q c 8 s h i F B 7 y 2 G J U / s m + x f r M 4 Y k F K U j k q c U o T O S Z x S h Q 5 L n F q K y T F x a j W J G X F q N o k V c W o 3 i R 1 x a j i J E 3 F m M x I 2 8 t W N V p K O e d w s / H h g 7 t Y 3 D g P I i E 6 w w m c Y Q b D K Z j M t x k M C k k 3 G I w i S T c Z j D p J N x h M E k l 3 G U w q S X c Y z A J J t x n M G k m P G D w A S 1 O 2 G M 4 H a r h I Y N J P O E R g 0 k / 4 T G D S U J h n 8 F 0 z o Y n D D 5 h Q z l l O G k p P G M w y S k 8 Z z A p K r x g M I k q v G Q w e w w O r x h O 2 g q v G U z y C m 8 Y T A 8 E 4 S 2 D j c b U r + + y q t 9 E / Z 5 l w N + z i H W C S V 9 i g 1 C S l 9 g k V K v r h b e p f / m Y C P B 8 T 4 D 0 8 N 4 x D L 2 t Z W 8 A g a 9 w O Y 6 E N 8 0 m 8 R A h b I E n 9 O 8 k W G F O C k 9 9 4 J P F 6 E h 9 L A M P O V a c + j f g 6 s d Q s U 1 3 J I W K H U J J o G K X U N K n 2 C O U v a U R + w S T P M U B o X S o i R 6 h p E 1 x S O g h c 3 x E M E l T H B N K y h R 9 Q k m Y 4 o R Q O t / E K a G k S n F G K H u G F e c E k y j F B a G k S X F J K B 1 1 4 o r Q K + b 4 m m A S p L g h l P Q o b g m 1 c t x K s R Y E / X D h m 9 c x D 1 X B S A V B z 3 0 s U B X j G r V Q r u v U Q p l u U F m r q s R N I v H U 2 6 I W K m m b W q i g H W q h c n a p h Y r Z o x Y K Z Z 9 a K J A D a q E w e t R C Q R y y w a A Q j o h E A R x T C z e + z 0 x x x 0 + I x J 0 + p R b u 8 B m 1 c G P P q Y U b e k E t 3 M h L a u E G X l E L 9 + 2 a W r h f N + z 2 u F G 3 7 I Z V 8 V U V X m r b g G + b N D W Y O l x U I O u P 9 z C a D b z s T S M 5 z i b S w w L I m 2 J + y 6 F w S y S g G s m p j 6 r 7 y 1 o I 2 v B J c Q i 6 g I J G B Q W 6 h I J G D Q W 6 i A K q o i p 0 s 0 b Z 6 z l d R 0 G j k A J d S U G j l A J d S 0 G j m A J d T Q G V U x b e q 2 H 2 A k 6 X V N C o q U A X V d C o q k C X V U B 1 V Y U e 1 i h 7 0 a Y r K 2 i U V q B r K 6 D i y s L 9 G u 5 z + K S G 2 R s 1 X W J B o 8 Y C X W R B o 8 o C X W Z B o 8 4 C X W h B o 9 I C X W p B o 9 Y C X W x B o 9 o C X W 4 B 1 V s W v q l h 9 o Z M l 1 z A a y 5 8 g s A E J I s J e J N 0 C E X 8 q D 5 9 G v r S 9 0 J I o c D c o 9 q R Q L 0 P J i o R u d r N l e l 8 l r + d 3 R X J T D d 0 G l R e I c m j I s I E 6 P S v P 0 I c P O r k p z 8 k U T f B b N n w b b 8 x G f s S H + T d W z i W f W 7 Z n 7 c N J s m G E H 9 s I t q g n o l p z Z t P M / 3 K q t + w a o w 7 i o d Q W d 7 p R j 3 8 u g c e F j I L x r 5 Q H 9 X 6 E 5 n p R y s o n C E 2 P o b N j U 0 9 y K r L 0 w E M w b E z z R a 7 A g k 8 e q y d a a I Y A v N D s W s d + 3 n s B z C v v 8 r p V c D c e + F V 1 + 4 C u / 2 3 5 n X 2 2 2 q O p C f Y h z + 9 J n s 6 5 5 m + 4 R Z V x l a 5 0 T M u 5 v b 1 n E s U E M 7 r F 2 5 N K p A 0 S d W K R p F a P d d M Z C O Z + A 9 k a Y G m H W a N T H 8 J Z d 7 E P f W S x x M 1 / Q / q L U F j c g e 9 O f 8 O 6 q D 3 Z A 8 v / Y K G o B r N G 0 j 8 x y 9 w + 4 u M W Z 4 9 2 Y K N r C R a N X S i u 8 r i U e E n 6 p 3 V e J o V W L M K / 1 F 4 z 3 v f v X q u P g D S X 6 t P U v N F q 8 h R A k J / h f b 8 D u K Y 2 d g 3 p y + 8 d U y F G P e p + u s R g x 4 S 9 T W c K o y N U 2 a t P k f N J q H O n r p O j i Q s a / c i 8 4 Y Z K H f T 6 D 7 K Y R j 5 K 4 0 P m r M i i d W b / v m s 9 9 3 q v I X M U l B c t 4 2 T U 9 3 v V X M / N J k r K m / p p u X Q + + 4 u S k f y s d k 1 9 w v 1 G h n P D l 8 F z B n g i S v 8 E L w o 9 d K s K v I l P K x 4 G + N M q P X J V E E Y j L 1 N f C Z O 4 U v h D b L s f k W F J X v b c 5 y r Q z o r / o Q 6 L 0 I 9 A v z 3 b l l d f c x Q H Z f G M N L v b F o s t W L R T P + 9 w O I c N X W u P h W M Q d 7 5 A 4 y 1 O J s O C v D v n 7 3 9 f K n b /H 8 l n l 5 c v l r p / n n l 9 c n r p W / W q / + P 4 r P O 7 z t / 6 H z V 6 X b + 0 v m m s 9 v p d y 4 6 Q S f t / L 3 z j 8 4 / u / / q / r v 7 f f c / x v T T T 6 o + v + s 4 f 7 r / / R + P Q l N k < / l a t e x i t >x < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 O 3 + 1 Q M u d 3 z y 0 / z U m P N h W e D M D U g = " > A A A B 8 X i c b V D L S g M x F L 1 T X 7 W + q i 7 d B I v g q s y I q M u i G 5 c V 7 E P b U j J p p g 3 N Z I b k j l C G / o U b F 4 q 4 9 W / c + T d m 2 l l o 6 4 H A 4 Z x 7 y b n H j 6 U w 6 L r f T m F l d W 1 9 o 7 h Z 2 t r e 2 d 0 r 7 x 8 0 T Directly inspired from these normalized attention mechanisms, we propose the normalized attention mechanism specifically defined for MEGA to improve its stability. Formally,Z R o x h s s k p F u + 9 R w K R R v o E D J 2 7 H m N P Q l b / n j m 8 x v P X F t R K T u c R L z X k i H S g S C U b T S Y z e k O P K D 9 G H a L 1 f c q j s D W S Z e T i q Q o 9 4 v f 3 U H E U t C r p B J a k z H c 2 P s p V S j Y J J P S 9 3 E 8 J i y M R 3 y j q W K h t z 0 0 l n i K T m x y o A E k b Z P I Z m p v z d S G h o z C X 0 7 m S U 0 i 1 4 m / u d 1 E g y u e q l Q c Y J c s f l H Q S I J R i Q 7 n w y E 5 g z l x B L K t L B Z C R t R T R n a k k q 2 B G / x 5 G X S P K t 6 F 9 X z u / N K 7 T q v o w h H c A y n 4 M E l 1 O A W 6 t A A B g q e 4 R XEQUATIONwhere Q and K are computed by using the normalized shared representation Z ′ instead of Z. Note that we remove the SiLU (Ramachandran et al., 2017) activation function ϕ silu in (13), because the normalization on Z has incorporated non-linearity into Z ′ . Then the attention operation in (17) has been changed to:EQUATIONAs we use learnable κ q , κ k in ( 7) and ( 8), we can remove the scaled term τ (X). In addition, we found that with the normalized attention, the softmax function f softmax obtains the best or at least comparable performance on different tasks and data modalities (see Appendix C). Hence, throughout this paper we use softmax as the default attention function.Normalization configurations are crucial in stably training deep architectures, and prenormalization (Xiong et al., 2020) has become the default normalization configuration because of its better convergence properties than post-normalization in the original Transformer architecture (Vaswani et al., 2017) . However, extensive studies have investigated the instability issue of pre-normalization when scaling up model size (Davis et al., 2021; Liu et al., 2022) . Formally, a Transformer-based block in pre-noromalization can be formulated as (shown in Figure 3 (b)):Ŷ = Attention(Norm(X)) + X Y = FFN(Norm( Ŷ )) + Ŷ = FFN(Norm( Ŷ )) + Attention(Norm(X)) + X (10)where the output Y is the sum of the input X and the output of each component in one block. Hence, the range and/or variance of Y keeps increasing for deeper blocks, causing the instability issue. In the original MEGA architecture, the update gate φ (19) is used for a gated residual connection (21) to mitigate this problem (Parisotto et al., 2020; Xu et al., 2020) . However, the update gate φ introduces more model parameters and the instability issue still exists when scaling up model size to 7 billion.MEGALODON introduces a new configuration named pre-norm with two-hop residual, which simply re-arranges the residual connections in each block (shown in Figure 3 (c):Ŷ = Attention(Norm(X)) + X Y = FFN(Norm( Ŷ )) + X (11)where the input X is reused as the residual connection of the FFN layer. Since Ŷ is directly followed by a normalization layer, we remove the update gate φ and use standard residual connection. The graphical architecture of a MEGALODON sub-layer is visualized in Figure 3 (a). Note that the Timestep Normalization is only applied before the attention layer. Before the FFN layer, we still use Layer Normalization. The reasons are two-fold: i) Layer Normalization is faster than Timestep Normalization; ii) the output vector of each token from the attention layer is a mixture of vectors from contextual tokens via attention weights. Hence, normalizing the attention output along the feature dimension is similar to indirectly normalize along the timestep dimension.Efficient distributed training algorithm is essential to train a large-scale language model, and several parallelization mechanisms have been introduced. The three most commonly used parallelism strategies are data, tensor (Shoeybi et al., 2019) and pipeline parallelism (Huang et al., 2019) . However, the 3-dimensional parallelism is still insufficient to scale up the context length of LLMs (Li et al., 2023b; Liu et al., 2024) .Benefiting from the chunk-wise attention in MEGALODON, we can efficiently parallelize it along the new timestep/sequence dimension, which is orthogonal to all the aforementioned three parallelism dimensions. In MEGALODON, the only communications between devices in one chunk-parallel group are the last hidden state of CEMA and the cumulative mean and variance of Timestep Normalization in each block. Using asynchronous communication, we can minimize the overhead of chunk parallelization by hiding the communication costs in the computation of other components inside the same block and/or other blocks. 
experiments: 
 To evaluate the scalability and efficiency of MEGALODON on long-context sequence modeling, we scale up MEGALODON to 7-billion model size and apply it to large-scale language model pretraining on 2 trillion tokens. We also conduct experiments on small/medium-scale sequence modeling benchmarks, including Long Range Arena (LRA) (Tay et al., 2021) , raw speech classification on Speech Commands (Warden, 2018) , image classification on ImageNet-1K (Deng et al., 2009) , and language-modeling on WikiText-103 (Merity et al., 2017) and PG19 (Rae et al., 2019) .2 Empirically, MEGALODON significantly outperforms all the state-of-the-art baseline models on these tasks across various data modalities.Architectural Details In our MEGALODON-7B model, we adopt most of architectural hyperparameters from LLAMA2-7B to ensure fair comparison: MEGALODON-7B consists of 32 blocks, with feature dimension d = 4096. Following LLAMA2, we use the SwiGLU activation function (Shazeer, 2020) in the feed-forward layer, and rotary positional embedding (RoPE, Su et al. (2021) ). We set the attention chunk size c = 4096, which is the same as the pretraining context length in LLAMA2.Benefiting from the attention gate (γ in ( 18)), we use a much smaller number of attention heads h = 4 in MEGALODON-7B, comparing to h = 32 in LLAMA2-7B. In addition, we apply pre-norm with two-hop residual ( §3.4), using Timestep Normalization ( §3.2) and Layer Normalization (Ba et al., 2016) , while LLAMA2 models apply pre-normalization with RMSNorm (Zhang and Sennrich, 2019) .We use the same mix of publicly available data from LLAMA2, ensuring that the model are trained on exactly the same 2-trillion tokens. We also use the same tokenizer as LLAMA2, whose vocabulary size is 32K.We trained MEGALODON-7B using the AdamW optimizer (Loshchilov and Hutter, 2019) , with β 1 = 0.9, β 2 = 0.95, ϵ = 1e -8. The learning rate is 3.5e -4 and cosine learning rate schedule is applied with warmup of 2500 steps. We use a weight decay of 0.1 and gradient clipping of 1.0, and no dropout is applied during training. The context length in pretraining is 32K (4 attention chunks).The global batch size is 4M tokens, and is distributed on 256 NVIDIA A100 GPUs (16K tokens per A100). We set data parallel size to 128, chunk parallel size to 2 and tensor parallel size to 1.We evaluate the efficiency of MEGALODON w.r.t both the data and computation perspectives. For data efficiency, we display the negative log-likelihood (NLL) for MEGALODON-7B, LLAMA2-7B and LLAMA2-13B w.r.t processed tokens during training in Figure 1 . MEGALODON-7B obtains significantly better (lower) NLL than LLAMA2-7B under the same amount of training tokens, demonstrating better data efficiency. Moreover, MEGALODON suffers less training spikes than the Transformer-based architecture in LLAMA2. Note that at the first 1/4 of the pretraining process (< 500B tokens), the NLL of MEGALODON-7B is slightly worse than LLAMA2-7B. We found that the main reason is that we increased the base θ of RoPE from 10, 000 in LLAMA2 to 100, 000 in MEGALODON, which slows down model convergence at the beginning of the pretraining process. At the end, MEGALODON reaches a training loss of 1.70, landing mid-way between LLAMA2-7B (1.75) and LLAMA2-13B (1.67). E n C / Y g O l Q g F o 2 i l V g 9 F x E 2 / X H G r 7 h x k l X g 5 q U C O R r / 8 1 R v E L I 2 4 Q i a p M V 3 P T d D P q E b B J J + W e q n h C W V j O u R d S x W 1 S / x s f u 2 U n F l l Q M J Y 2 1 J I 5 u r v i Y x G x k y i w H Z G F E d m 2 Z u J / 3 n d F M N r P x M q S Z E r t l g U p p J g T G a v k 4 H Q n K G c W E K Z F v Z W w k Z U U 4 Y 2 o J I N w V t + e Z W 0 L q r e Z b VEQUATIONLlama2-7B 32K For computation efficiency, we conduct experiments of running LLAMA2-7B and MEGA-LODON-7B using the same amount of computational resources and comparing their training speed under various context lengths. Specifically, we execute each experiment to train a model with global batch size 4M tokens distributed on 256 NVIDIA A100 GPUs (16K tokens per A100) and calculate the word/token per second (WPS) to measure the training speed. Figure 4 illustrates the average WPS per device of LLAMA2-7B and MEGALODON-7B using 4K and 32K context lengths, respectively. For LLAMA2 models, we accelerate the computation of full attention with Flash-Attention V2 (Dao, 2024) . Under 4K context length, MEGALODON-7B is slightly slower (about 6%) than LLAMA2-7B, due to the introduction of CEMA and Timestep Normalization. When we scale up context length to 32K, MEGALODON-7B is significantly faster (about 32%) than LLAMA2-7B, demonstrating the computation efficiency of MEGALODON for long-context pretraining. In addition, MEGALODON-7B-32K, which utilizes chunk parallelism ( §3.5), achieves about 94% utilization of MEGALODON-7B-4K.We compare MEGALODON-7B to LLAMA2 models on standard academic benchmarks with short contexts (< 4K tokens), closely following the settings in LLAMA2 (Touvron et al., 2023) . The benchmarks are grouped into the categories listed below:• Commonsense Reasoning (0-shot): HellaSwag (Zellers et al., 2019) , PIQA (Bisk et al., 2020) , SIQA (Sap et al., 2019) , WinoGrande (Sakaguchi et al., 2021) , ARC-e and -c (Clark et al., 2018) . • World Knowledge (5-shot): NaturalQuestions (NQ, Kwiatkowski et al. (2019) ) and TriviaQA (TQA, Joshi et al. (2017) ). • Reading Comprehension (0-shot): BoolQ (Clark et al., 2019) .• Popular aggregated results (5-shot): MMLU (Hendrycks et al., 2020) . Table 1 summarizes the results of MEGALODON and LLAMA2 on these academic benchmarks, together with other open-source base models, including MPT (MosaicML, 2023) , RWKV (Peng et al., 2023) , Mamba (Gu and Dao, 2023), Mistral (Jiang et al., 2023) and Gemma (Mesnard et al., 2024) . Pretrained on the same 2T tokens, MEGALODON-7B surpasses LLAMA2-7B across all the (Hutchins et al., 2022) and MEGABYTE (Yu et al., 2024) . MEGALODON significantly outperforms all the baselines. 
conclusion: 
 We have introduced MEGALODON, an improved MEGA architecture with multiple novel technical components, including complex exponential moving average (CEMA), the timestep normalization layer, normalized attention and pre-norm with two-hop residual configuration, to improve its capability, efficiency and scalability. Through a direct comparison with LLAMA2, MEGALODON achieves impressive improvements on both training perplexity and across downstream benchmarks. Importantly, experimental results on long-context modeling demonstrate MEGALODON's ability to model sequences of unlimited length. Additional experiments on small/medium-scale benchmarks across different data modalities illustrate the robust improvements of MEGALODON, which lead to a potential direction of future work to apply MEGALODON for large-scale multi-modality pretraining. Timestep Normalization For the TimestepNorm operator, we have an efficient implementation to improve both its speed and numerical stability. To compute the cumulative mean and variance for each of the timesteps, our implementation distributed the threads in each CUDA block in both the timestep/sequence dimension and the feature dimension to balance the parallelism of the algorithm and the performance of the global memory access. To improve numerical stability, we used the Welford algorithm (Welford, 1962) to compute the cumulative mean and variance and the Kahan Summation (Kahan, 1965) to reduce the numerical error from summation. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [NA] Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: see experiments and appendix Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In the normalization methods, two learnable parameters γ and β are introduced to scale and shift the normalized value:EQUATIONwhere µ and σ 2 are the mean and variance of the input x across the pre-defined dimensions. Initialization of γ and β is crucial for model performance and stability. The standard implementation of normalization layers, such as PyTorch (Paszke et al., 2019) , initializes γ and β to vectors of ones and zeros, respectively, to preserve the mean and variance of the normalized inputs at the beginning of training.This standard implementation, however, suffers a problem when weight decay regularization is applied to prevent overfitting (Hanson and Pratt, 1988) . Technically, the weight decay regularization pushes the values of model parameters towards smaller magnitudes. In the context of normalization methods, weight decay pushes the values in γ towards zero, which diverges from its initialization of one. This may prevent the model from learning the true scale of the data distribution, and may cause numerical stability issues as well.To address this problem, we used the plus 1 reparameterization5 of the scale parameter γ:EQUATION)where γ is initialized to zero. Under weight decay, γ remains centered around zero, resulting in a desirable scale of γ + 1 around one.We conducted small-scale experiments on five benchmarks across various data modalities, including text, audio and image. To demonstrate the robustness of the MEGALODON architecture on different tasks and data types, we used a single unified architecture with minimal architectural divergence in What is the central dogma of molecular biology? What processes are involved? Who named this?The central dogma of molecular biology describes the flow of genetic information from DNA to RNA to proteins. The three processes involved in the central dogma are transcription, translation, and translocation. The central dogma was first proposed by Francis Crick, James Watson, and Maurice Wilkins in the 1950s, and was later formalized by Crick in a 1958 paper titled "On the Central Dogma of Molecular Biology". The term "central dogma" was coined by Crick to describe the flow of genetic information from DNA to RNA to proteins.Identify and fix one incorrect fact in your previous response.The correct name of the person who proposed the central dogma of molecular biology is James Watson, not Maurice Wilkins. James Watson, along with Francis Crick, first proposed the central dogma in the 1950s, and Maurice Wilkins was a co-recipient of the Nobel Prize in Physiology or Medicine in 1962 for his contributions to the discovery of the double helix structure of DNA.Act as a math teacher. I will provide some mathematical equations or concepts, and it will be your job to explain them in easy-to-understand terms. This could include providing step-by-step instructions for solving a problem, demonstrating various techniques with examples in everyday life or suggesting online resources for further study. My first request is "I need help understanding how probability works."Probability is a branch of mathematics that deals with the likelihood of an event occurring. It is often expressed as a number between 0 and 1, where 0 represents an impossible event and 1 represents a certain event.For example, if you were to roll a fair six-sided die, the probability of rolling a 1 would be 1/6, or about 17%. This is because there is only one way for the die to land on a 1 out of the six possible outcomes.To calculate probability, you can use the formula: Probability = Number of ways the event can occur / Total number of possible outcomes For example, if you were to ask "What is the probability of flipping heads on a fair coin toss?", the answer would be 1/2, or 50%. This is because there is only one way for the coin to land heads-up out of the two possible outcomes (heads or tails).There are many different ways to interpret probability, and it can be used in a variety of real-world applications such as gambling, insurance, and statistics. I hope this helps to give you a basic understanding of probability. Let me know if you have any questions or need further clarification.What are the differences between Riemannian geometry and euclidean geometry?Riemannian geometry and Euclidean geometry are both branches of mathematics that study the properties of space, but they differ in their assumptions and methods.Euclidean geometry is based on the assumption that the space is flat and flat Euclidean space is the simplest non-empty connected flat metric space. It studies the properties of space using concepts such as distance, angle, and parallelism. It is named after the Greek mathematician Euclid, who developed the foundations of the field in his book "Elements". Riemannian geometry, on the other hand, is based on the assumption that the space is curved. It studies the properties of space using concepts such as distance, angle, and parallelism, but in a more general setting, and it is named after the German mathematician Bernhard Riemann.In Riemannian geometry, the concept of a metric is used to measure the distance between two points in a curved space. A metric is a mathematical function that assigns a distance to each segment in a space. The most common metric in Riemannian geometry is the distance function induced by a Riemannian metric, which is a mathematical function that assigns a distance to each segment in a space. In summary, Euclidean geometry is a special case of Riemannian geometry, where the metric is the flat Euclidean metric. Riemannian geometry is a more general field that studies the properties of curved spaces, and it is used in many areas of mathematics, physics and other sciences. I hope this helps to give you a basic understanding of the differences between Riemannian geometry and Euclidean geometry. Let me know if you have any questions or need further clarification.Evaluate the following movie reviews on a scale of 1 to 5, with 1 being very negative, 3 being neutral, and 5 being very positive: 1. This movie released on Nov. 18, 2019, was phenomenal. The cinematography, the acting, the plot -everything was top-notch. 2. Never before have I been so disappointed with a movie. The plot was predictable and the characters were one-dimensional. In my opinion, this movie is the worst one to have been released in 2022. 3. The movie was okay. There were some parts I enjoyed, but there were also parts that felt lackluster. This is a movie that was released in Feb 2018 and seems to be quite ordinary. Return the answer as a JSON array of integers.[5, 1, 3]Update your previous reply by including the release date as part of the JSON content.[ {"release_date": "Nov 18, 2019", "rating": 5}, {"release_date": "2022", "rating": 1}, {"release_date": "Feb 2018", "rating": 3} ] • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.Some results are provided in Appendix C, due to space limits.Geometric mean of ROUGE-1, 2 and L.https://klu.ai/glossary/mt-bench-evalSimilar idea in the blog: https://medium.com/@ohadrubin/exploring-weight-decay-in-layer-normalizationchallenges-and-a-reparameterization-solution-ad4d12c24950 


Decision: This paper addresses long context length issue by introducing new architecture. 

Intially, three reviewers voted acceptance and one reviewer gave a borderline rejection with lack of experiments.

However, the authors successfuly addresss main concerns that reviewers raised, thus all reviewers gave positive scores.

AC also agree with the reviewers opinion, and moreover AC would like to encourage research on new architecture apporach for topic diversity in LLM regimes. 

So, AC recommens accepting this paper.

Reviews: review: paper_summary: paper_summary: The paper proposes Megalodon, which introduces three advancements over Mega: complex EMA, timestep normalization, and normalized attention. These advancements address the limitations of chunk-wise attention and architecture divergence across different tasks and data types. The new model is evaluated alongside Llama-2, both trained on the same public dataset, and demonstrates competitive and superior performance across a wide range of benchmarks.
strength_weakness: strength_weakness: Recent theoretical work [1] has shown that efficient versions of attention (like the chunk-based method used in this paper) can limit the expressiveness of the model, particularly for reasoning tasks that involve long-range information. The paper evaluates Megalodon with long-context open-book QA tasks. How will Megalodon perform on PhoneBook lookup [2] with ICL, especially for phonebook lengths longer than 4K tokens? How will it perform on complex reasoning tasks requiring long context, such as 8-shot or 16-shot math problems with GSM8K or coding tasks on HumanEval?

Empirically, it is not clear how CEMA improves expressiveness. It would be most direct to compare using CEMA versus using EMA on these tasks.

The reviewer understands that the rebuttal period is short and is therefore not requiring most of these experiments to be added.

[1] Yang, Kai, et al. "Do Efficient Transformers Really Save Computation?" Forty-first International Conference on Machine Learning.

[2] Jelassi, Samy, David Brandfonbrener, and Sham M. Kakade. "Repeat After Me: Transformers are Better than State Space Models at Copying." Forty-first International Conference on Machine Learning.
questions: questions: Have the authors conducted ablation studies for small-scale models before moving to 7B, similar to the results in Appendix C? Including these ablation studies, if already performed, would help readers understand how the three designs impact performance.
limitations: limitations: The paper does not discuss its limitations. The authors believe there is no negative societal impact. In the paper checklist, justifications are required for answers marked "Yes," but the authors have deleted the justification for several items. For limitations, the authors claim they are discussed, but there is no justification provided.

The anonymous link is not working.
paper_summary: paper_summary: This is an empirincal paper. The paper presents MEGALODON, a neural architecture designed to overcome the quadratic complexity and weak length extrapolation of Transformers. By extensive experiments, this  paper demonstrates MEGALODON's ability to efficiently handle unlimited context lengths and its superior performance across different tasks and modalities. The claimed key contributions of this paper is as follows::
1) improves upon the MEGA architecture by adding the complex exponential moving average (CEMA), timestep normalization, normalized attention, and a pre-norm with two-hop residuals.
2. achieves better pretraining efficiency and downstream task accuracy than Transformers, specifically in handling long sequences with 7 billion parameters and 2 trillion training tokens.
3. outperforms Transformers across various benchmarks, including long-document comprehension, multi-turn conversation, and video generation.
strength_weakness: strength_weakness: 1. The training curves of MEGALODON and LLAMA2 7B in Figure 1 cross at 750 billion training tokens. It would be interesting to see if they cross again with more tokens, such as at 6 trillion.
2. Section 3.2 is difficult to understand due to the lack of background and intuitive explanations.
3. Section 3.5 makes overclaims about 4D parallelism. This topic is well-explored and relevant, as discussed in [1, 2, 3], but these references are ignored. If the paper aims to highlight the benefits of 4D parallelism, it should include comparative experiments.
4. The code link provided in the abstract does not work.
5. The long context tasks in Section 4.3, as shown in Tables 2 and 6, are not fairly set up. In Table 2, the only fair baseline is LLAMA2-L, which performs better. This raises doubts about the proposed method's long-context ability since other models (Yarn, MPT, Xgen) use different training datasets, potentially limiting their long-context capability. For Table 5, the baselines (Transformer, Reformer, ...) also use different training datasets, making the comparisons misleading. Compared to MEGA, the improvements are marginal. Thue the compression of the long-ctx ability is not convincing to me.


[1] Sequence Parallelism: Making 4D Parallelism Possible\
[2] Lightseq: Sequence level parallelism for distributed training of long context transformers\
[3] USP: A Unified Sequence Parallelism Approach for Long Context Generative AI
questions: questions: see Weaknesses
limitations: limitations: see Weaknesses
paper_summary: paper_summary: Megalodon i.e. Mega2 improves over Mega by using (1) complex-valued EMA; (2) improved normalization schemes (e.g. timestep normalization, attention normalization, pre-norm with 2-hop residuals).
strength_weakness: strength_weakness: - There are no ablation studies in moderate-scale language modeling at all

- Figure1: the training sequence length is not the same, thus the perplexity comparison might be unfair. 

- Lacking discussions & comparison to recent hybrid (local) attention and RNN models: e.g. Griffin, Jamba

- Long context evaluation is not comprehensive: the paper claims unlimited context length, while it is unclear about the effective context length. It is better to provide the Needle in the Haystack results
questions: questions: - Complex-valued linear recurrence is replaced by gated linear recurrence for many recent works:  e.g. LRU (complex) -> RG-LRU (real gated). HGRN1 (complex) -> HGRN2 (real gated).  S4 (complex) -> Mamba1 & 2 (real gated).  Did you try real-valued gated linear recurrence layers?

- What's your opinion on Sliding Window Attention vs. Chunk Attention?
limitations: limitations: NA
paper_summary: paper_summary: The paper introduces Megaladon, an improvement on the existing technique Mega. This technique uses a diagonalizable complex moving average to allow for integration of information across a longer context efficiently.
strength_weakness: strength_weakness: The evaluation is lacking in some key aspects. If these are remedied, the paper is strong.

- There needs to be more comparison against MEGA, this seems critical to establish technical novelty beyond operating at a larger scale than the Mega paper.
At scale, for the core NLP benchmarks, how does Megaladon compare Mega and Mega chunk? What changes actually make an improvement? Ablations in particualar around these design choices seem key to establishing the technical novelty, since the changes are somewhat small and it's not clear they buy you a large gap in improvement, or which pieces do. 

I have looked at the Imagenet Mega comparison, and those in the appendix, but this is not at scale, which is a critical element of the paper.

- The paper spends much time comparing against models of shorter context lengths, while it is clear the unlimited inference context length is an advantage, Mega had this too. A compute matched setting should still be compared against. In particular, How does Llama trained with let's say a ~28k context (or whatever context yields the same 1.3x speed improvement of Megaladon) perform in comparison at the scale of the main experiments on the main benchmarks.  The monotonic perplexity in Figure 5 is nice to see, but the perplexity other methods at their native or extended context lengths should be included, the minimum perplexity by megaladon might be higher than those achieved by other methods at shorter context lengths.

- A commonly targeted use case of models like this is direct byte level modeling, it would be interesting to see how this method performs on that task. Many of the methods in the long context section were designed with this purpose explicitly in mind. This seems like an application which is important to test on, particularly if context extension is viable in this domain using this method, that would be very interesting.

- Mamba state space models seem to exhibit naturally good context extension, since this method can be seen as a state space model it might be good to compare in the long sequence domain at a similar scale to the 3b Mamba-[1,2] models.


While it might be infeasible to include all of the above, the first 2-3 seem of particular importance to establish the usefulness and technical novelty of this technique.
questions: questions: Why do not start to modeling improvements until well over 1T training tokens, is this a byproduct of the learning schedule or of something structural within the model?

How much compute is added over Mega/Mega-Chunk to incorporate the necessary changes?

How is the complex moving average implemented?
limitations: limitations: Limitations should be addressed further and more directly in the paper.

Comments: title: Re: Nearly There
Re: Nearly There
Re: Nearly There
Re: Official Comment by Reviewer 6wcE
Nearly There
comment: I appreciate the large-scale training done in this work and will maintain my positive score, leaning towards acceptance. Here are some suggestions for the camera-ready version:

- I understand that it may be challenging to incorporate many results during the rebuttal period. However, I highly recommend that the authors conduct some moderate-scale experiments for ablation in the camera-ready version, which I believe would significantly benefit the paper. I suggest using the setting from Samba [1], specifically a 1.3B model trained on Slimpajama with 100Bn tokens.

- I would also suggest revising the claim of "unlimited context length" in the camera-ready version. This is definitely an overstatement. While I understand that many works, such as [1], make similar claims, from a more scientific perspective, the ability to process arbitrary sequence lengths differs from effectively utilizing the entire context. I can easily imagine that these RNN + local attention models still struggle with retrieval tasks like Needle in the Haystack or Phonebook Lookup. Please include related discussions in the camera-ready version.

[1] Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling
Thanks for the rebuttal and additional ablation studies. After reviewing the new information, I am maintaining my original scores and continue to recommend acceptance.
> I do still think the perplexities should be comparable if we expect the downstream performance numbers to be comparable, we should also expect the perplexities to be comparable as well correct?

Unfortunately, neither the PPL nor the downstream performance are directly comparable when the models are in different scales (#parameters) and/or trained on different datasets. That is why we trained our Megalodon-7B on exactly the same 2T training tokens with LlaMa2-7B. Therefor, the only comparable downstream results in Table 1 are LlaMa2-7B and Megalodon-7B. And that is why we put them together in the last two rows. Conducting a direct comparison between Megalodon and the Transformer architecture in LlaMa2 in large-scale pretraining is one of the main motivations and contributions of this work.

> At the very least the y-xis should be labeled on that plot, is the perplexity monotonically decreasing from 4.98 to 4.97 or down much further for example. It is unclear.

As mentioned in Section 4.3, the validation data in Figure 5 are selected books, each of which contains at least 2M tokens. We removed the scale in y-xis in Fiture 5 to protect the privacy of the data. What we can say is that the PPL at 4K is $4.98$ and the PPL at 2M is $4.82$
I do still think the perplexities should be comparable if we expect the downstream performance numbers to be comparable, we should also expect the perplexities to be comparable as well correct?

At the very least the y-xis should be labeled on that plot, is the perplexity monotonically decreasing from 4.98 to 4.97 or down much further for example. It is unclear.
We appreciate your positive response!

> For your suggestion on reporting PPLs of other architectures at their native context length. 

The problem is that the PPLs from different models on the held-out validation dataset are not directly comparable, since these models were pre-trained on different training data. The only comparable model/architecture is LlaMa2-7B because it was trained on exactly the same 2T data with Megalodon. However, LlaMa2 only supports 4K context length. On 4K context length, the PPL of LlaMa2-7B on the validation data in Figure 5 is $5.09$ while the corresponding PPL of Megalodon is $4.98$. Thus, even at a relatively short context length of 4K, Megalodon slightly outperforms LlaMa2 on PPL.
We appreciate your suggestion on conducting ablation studies on Slimpajama. However, due to the limits of time and computation resources, we cannot provide the results during rebuttal period. We will consider to add this study in our final version.

In addition, though the performance on WikiText-103 might be impacted by regularization, we want to say that we have fully swept hyper-parameters for the models in our ablation study to ensure fair comparison. The results in our table are better than previous state-of-the-art numbers, demonstrating the reliability of our ablation study.
Hi,
Thank you for the thoughtful response.

The ablations help, and knowing that the changes were critical to allowing Mega to achieve larger scales is good. Knowing that there are byte level classification present.

For the monotonic perplexity plot, can you please add in a y-axis scale and the the perplexities of the other techniques at their native context length. The moving average makes it clear that generalization is possible, but it's critical to know the range of improvement we can actually expect. The actual improvement may not be that much though.

With this final change, I will raise my score.

Thank you
Regarding ablations, it is well known that performance on wikitext-103 can be sensitive to regularization since the dataset is relatively small, can we have some experiments on Slimpajama?