Title: ON GENERALIZATION BOUNDS OF A FAMILY OF RECURRENT NEURAL NETWORKS

Abstract: Recurrent Neural Networks (RNNs) have been widely applied to sequential data analysis. Due to their complicated modeling structures, however, the theory behind is still largely missing. To connect theory and practice, we study the generalization properties of vanilla RNNs as well as their variants, including Minimal Gated Unit (MGU) and Long Short Term Memory (LSTM) RNNs. Specifically, our theory is established under the PAC-Learning framework. The generalization bound is presented in terms of the spectral norms of the weight matrices and the total number of parameters. We also establish refined generalization bounds with additional norm assumptions, and draw a comparison among these bounds. We remark: (1) Our generalization bound for vanilla RNNs is significantly tighter than the best of existing results; (2) We are not aware of any other generalization bounds for MGU and LSTM in the exiting literature; (3) We demonstrate the advantages of these variants in generalization.
Under review as a conference paper at ICLR 2019 In this paper, we propose to understand the generalization ability of RNNs and their variants. We aim to answer two questions from a theoretical perspective: Q.1) Do RNNs suffer from significant curse of dimensionality? Q.2) What are the advantages of MGU and LSTM over vanilla RNNs?
The investigation of generalization properties of neural networks including RNNs has a long history. Most of these works adopt a layer wise analysis and establish their results by induction. For example, Haussler (1992) establishes a complexity bound for feedforward neural networks. The result assumes average Lipschitz constant of each layer to be greater than 1 and involves the product of these Lipschitz constants. Thus, the resulting bound is inevitably exponential in the depth of the network. Later, Dasgupta and Sontag (1996) and Koiran (1998) adopt a VC-dimension argument to show complexity bounds of RNNs that are polynomial in the size of the network. Both of the works, however, are based on oversimplified assumptions: Dasgupta and Sontag (1996) only consider linear RNNs for binary classification tasks; Koiran (1998) assumes RNNs take the first coordinate of their hidden states as outputs. More recently, Bartlett et al. (2017) propose a new technique for developing generalization bounds for feedforward neural networks based on empirical Rademacher complexity under the PAC-Learning framework. Neyshabur et al. (2017) further adapt the technique to establish their generalization bound using the PAC-Bayes approach. Then the follow-up work Zhang et al. (2018) use the PAC-Bayes approach to establish a generalization bound for vanilla RNNs.
Our theory is partially motivated by Bartlett et al. (2017) , but is quite different from Bartlett et al. (2017) and Zhang et al. (2018) . In particular, our analysis exploits the compositional nature of RNNs, and decouples the spectral norms of weight matrices and the number of weight parameters. This makes our analysis conceptually much simpler (e.g. avoid layer wise analysis), and also yields better generalization bound than Zhang et al. (2018) .

1. INTRODUCTION:

Recurrent Neural Networks (RNNs) have successfully revolutionized sequential data analysis, and been widely applied to many real world problems, such as natural language processing (Cho et al., 2014; Bahdanau et al., 2014; Sutskever et al., 2014) , speech recognition (Graves et al., 2006; Mikolov et al., 2010; Graves, 2012; Graves et al., 2013) , computer vision (Gregor et al., 2015; Xu et al., 2015; Donahue et al., 2015; Karpathy and Fei-Fei, 2015) , healthcare (Lipton et al., 2015; Choi et al., 2016a; b) , and robot control (Ku and Lee, 1995; Lee and Teng, 2000; Yoo et al., 2006) . Quite a few of these applications can be approached easily in our daily life, such as Google Translate, Google Now, Apple Siri, etc.

The sequential modeling nature of RNNs is significantly different from feedforward neural networks, though they both have neurons as the basic components. RNNs exploit the internal state (also known as hidden unit) to process the sequence of inputs, which naturally captures the dependence of the sequence. RNNs can also be viewed as nonlinear dynamical systems, and reduced to linear dynamical systems given identity activation operators. Besides the vanilla version, RNNs have many other variants. A large class of variants incorporate the so-called "gated" units to trim RNNs for different tasks. Typical examples include Long Short-Term Memory (LSTM, Hochreiter and Schmidhuber (1997)), Gated Recurrent Unit (GRU, Jozefowicz et al. (2015) ) and Minimal Gated Unit (MGU, Zhou et al. (2016) ).

The success of RNNs owes not only to their special network structures and the ability to fit training data, but also to their good generalization property: They can provide accurate predictions on unseen data. For example, Graves et al. (2013) report that after training with merely 462 speech samples, deep LSTM RNNs achieve a test set error of 17.7% on TIMIT phoneme recognition benchmark, which is the best recorded score. Mikolov et al. (2010) also show that RNNs outperform significantly state-of-the-art backoff models for speech recognition. When using RNNs in Wall Street Journal task to predict the next word in textual data given the context, word error rate is reduced around 18% compared to backoff models trained on the same amount of data, and 12% when backoff model is trained on 5 times more data. Despite of the popularity of RNNs in applications, their theory is less studied than other feedforward neural networks (Bartlett et al., 2017; Neyshabur et al., 2017; Golowich et al., 2017; Li et al., 2018) . There are still several long lasting fundamental questions regarding the approximation, trainability, and generalization of RNNs. Taking a sequence to sequence multiclass classification problem as an example, we observe m sequences of data points (x i,t , z i,t ) T t=1 , where x i,t ∈ R dx and the class label z i,t ∈ {1, . . . , K} for all t = 1, ..., T and i = 1, ..., m. Each sequence is drawn independently from some underlying distribution over R dx×T × {1, . . . , K}. The vanilla RNNs compute h i,t and y i,t iteratively as follows, h i,t = σ h (U h i,t-1 + W x i,t ) , and y i,t = σ y (V h i,t ) , where σ y and σ h are activation operators, h i,t ∈ R d h is the hidden state with h i,0 = 0, and U ∈ R d h ×d h , V ∈ R dy×d h , and W ∈ R d h ×dx are weight matrices. The activation operators σ h and σ y are entrywise, i.e., σ h ([v 1 , . . . , v d ] ) = [σ h (v 1 ), . . . , σ h (v d )] , and Lipschitz with parameters ρ h and ρ y respectively. For simplicity, we assume σ h (•) = tanh(•), σ y (0) = 0, and ρ y = 1. Extensions to general activation operators are given in Section 2. For a new testing sequence (x t , z t ) T t=1 , we predict the label sequence using z t = argmax j [y t ] j , for all t = 1, . . . , T.

To establish the generalization bound, we need to define the "model complexity" of vanilla RNNs. In this paper, we adopt the empirical Rademacher complexity (ERC, see more details in Section 2), which has been widely used in the existing literature on PAC-Learning. For many nonparametric function classes, we often need complicated argument to upper bound their ERC. Our analysis, however, shows that we can upper bound the ERC of vanilla RNNs in a very simple manner by exploiting their Lipschitz continuity with respect to (w.r.t) the model parameters, since they are essentially in parametric forms. More specifically, denote F t = {f t : {x 1 , ..., x t } → y t } as the class of mappings from the first t inputs to the t-th output computed by vanilla RNNs. For a matrix A, A 2 denotes the spectral norm, and for a vector v, v 2 denotes the Euclidean norm. Define

x t -1

x-1 = t for x = 1. Then, informally speaking, the "model complexity" of vanilla RNNs satisfies

Complexity = O d min √ d, W 2 U t 2 -1 U 2 -1 V 2 log t √ d U t 2 -1 U 2 -1

,

where d = d x d h + d 2 h + d h d y .

We then give the generalization bound in the following statement.

Theorem 1 (informal). Given a collection of samples S = (x i,t , z i,t ) T t=1 , i = 1, ..., m with x i,t 2 ≤ 1 and a new testing sequence (x t , z t ) T t=1 , with probability at least 1δ over S, for every margin value γ > 0 and f t ∈ F for integer t ≤ T , we have,

EQUATION

where R γ,t =1 m m i=1 1([y i,t ] zi,t ≤ max j =zi,t [y i,t ] j + γ).

Please refer to Section 2 for a complete statement. The generalization bound in Theorem 1 can be interpreted under three different scenarios 1 : (I) When U 2 < 1, the generalization bound is We theoretically justify that vanilla RNNs do not suffer from significant curse of dimensionality.

O d √ mγ ,

Because they compute outputs y t recursively using the same weight matrices, and their hidden states h t are entrywise bounded.

Compared with the generalization bound in Zhang et al. (2018) , which is of the order

O dt 2 W 2 V 2 max{1, U t 2 } √ mγ ,

our bound is tighter by a factor of t2 for case (I), a factor of t for case (II). Additionally, Zhang et al. (2018) fail to incorporate the boundedness condition of hidden state into their analysis, thus the generalization bound is exponential in t for case (III). Our generalization bound, however, is still polynomial in d and t for case (III).

Moreover, (II) is closely related to a few recent results on imposing orthogonal constraints on weight matrices to stabilize the training of RNNs (Saxe et al., 2013; Le et al., 2015; Arjovsky et al., 2016; Vorontsov et al., 2017; Zhang et al., 2018) . We remark that from a learning theory perspective, (II) also implies that the orthogonal constraints can potentially help generalization.

We also present refined generalization bounds with additional matrix norm assumptions. These assumptions allow us to derive norm-based generalization bounds. We draw a comparison among these bounds and highlight their advantage under different scenarios.

Our theory can be further extended to several variants, including MGU and LSTM RNNs. Specifically, we show that the gated units in MGU and LSTM RNNs can introduce extra decaying factors to further reduce the dependence on d and t in generalization. Such an advantage in generalization make these RNNs do not suffer from significant curse of dimensionality either. To the best of our knowledge, these are the first results on generalization guarantees for these neural networks.

The rest of the paper is organized as follows: Section 2 presents the generalization bound of vanilla RNNs; Section 3 presents the proof outline of the generalization bound; Section 4 presents refined generalization bounds and their comparison; Section 5 presents the generalization bound of MGU and LSTM RNNs; Section 7 discusses related works and collects open problems.

Notations: Given a vector v ∈ R d , we denote the vector Euclidean norm by v 2 2 = d i=1 |v i | 2 , and the infinity norm by v ∞ = max j |v j |. Given a matrix M ∈ R m×n , we denote the spectral norm by M 2 as the largest singular value of M , the Frobenius norm by M 2 F = trace(M M ), and the (2, 1) norm by M 2,1 = n i=1 M :,i 2 . Given a function f , we denote the function infinity norm by f ∞ = sup |f |. We adopt the standard O(•) notation, which is defined as f (x) = O (g(x)) for x → ∞ if and only if there exists M > 0 and x 0 , such that |f (x)| ≤ M g(x) for x ≥ x 0 . We use O(•) to denote O(•) with hidden log factors.

To establish the generalization bound, we start with imposing some mild assumptions.

Assumption 1. Input data are bounded, i.e., x i,t 2 ≤ B x for all i = 1, . . . , m and t = 1, . . . , T .

Assumption 2. The spectral norms of weight matrices are bounded respectively, i.e., U 2 ≤ B U , V 2 ≤ B V , and W 2 ≤ B W .

Assumption 3. Activation operators σ h and σ y are Lipschitz with parameters ρ h and ρ y respectively, and σ h (0) = σ y (0) = 0. Additionally, σ h is entrywise bounded by b.

Assumptions 1 and 2 are moderate assumptions. Moreover, Assumption 3 holds for most commonly used activation operators, such as σ h (•) = tanh(•) and σ y (•) = ReLU(•) = max{•, 0} (1-Lipschitz).

Recall that vanilla RNNs compute h i,t and y i,t as follows, h i,t = σ h (U h i,t-1 + W x i,t ) and y i,t = σ y (V h i,t ) , where U ∈ R d h ×d h , V ∈ R dy×d h , and W ∈ R d h ×dx . Given a sequence (x t , z t ) T t=1 , we define X t ∈ R dx×t by concatenating x 1 , . . . , x t as columns of X t . Recall that we denote F t = {f t : X t → y t } as the class of mappings from the first t inputs to the t-th output computed by vanilla RNNs. Then we define the functional margin for the t-th output in vanilla RNNs as

M(f t (X t ), z t ) = [f t (X t )] zt -max j =zt [f t (X t )] j .

We further define a ramp loss γ (-M(f t (X t ), z t )) : R → R + to each margin, where γ is a piecewise linear function defined as

γ (a) = 1{a > 0} + (1 + a γ )1{-γ ≤ a ≤ 0},

where 1{A} denotes the indicator function of a set A. Accordingly, the ramp risk is defined as R γ (f t ) = E [ γ (-M(f t (X t ), z t ))] , and its empirical counterpart is defined as R γ (f t ) = 1 m m i=1 γ (-M(f t (X i,t ), z i,t )) . We then present the formal statement of Theorem 1. Theorem 2. Let activation operators σ h and σ y be given, and Assumptions 1-3 hold. Then for (x t , z t ) T t=1 and S = (x i,t , z i,t ) T t=1 , i = 1, . . . , m drawn i.i.d. from any underlying distribution over R dx×T × {1, . . . , K}, with probability at least 1δ over S, for every margin value γ > 0 and every f t ∈ F t for integer t ≤ T , we have

P ( z t = z t ) ≤ R γ (f t ) + O dρ y B V min b √ d,ρ h B x B W β t -1 β-1 log t √ dm β t -1 β-1 √ mγ + log 1 δ m , where d = d x d h + d 2 h + d h d y and β = ρ h B U .

We remark that the generalization bound depends on the total number of weights, and the range of ρ h B U in three cases as indicated in Section 1. More precisely, if ρ h B U (1 + 1 t α ) for constant α > 0 bounded away from zero, the generalization bound is of the order O dt α √ mγ , which has a polynomial dependence on d and t. As can be seen, with proper normalization on model parameters, vanilla RNNs do not suffer from significant curse of dimensionality.

We also highlight a tradeoff between generalization and representation of vanilla RNNs. As can be seen, when ρ h B U is strictly smaller than 1, the generalization bound is nearly independent on t. The hidden state, however, only has limited representation ability, since its magnitude diminishes as t grows large. On the contrary, when ρ h B U is strictly greater than 1, the representation ability of hidden state is amplified but the generalization becomes worse. As a consequence, recent empirical results show that imposing extra constraints or regularization, such as U U = I or U 2 ≤ 1 (Saxe et al., 2013; Le et al., 2015; Arjovsky et al., 2016; Vorontsov et al., 2017; Zhang et al., 2018) , helps balance the generalization and representation of RNNs.

3. PROOF OF MAIN RESULTS:

Our analysis is based on the PAC-learning framework. Due to space limit, we only present an outline of our proof. More technical details are deferred to Appendix A. Before we proceed, we first define the empirical Rademacher complexity as follows. Definition 1 (Empirical Rademacher Complexity). Let H be a function class and S = {s 1 , . . . , s m } be a collection of samples. The empirical Rademacher complexity of H given S is defined as Empirical Rademacher Complexity: R S (H) = E sup h∈H 1 m m i=1 i h(s i ) , where i 's are i.i.d. Rademacher random variables, i.e., P( i = 1) = P( i = -1) = 0.5.

We then proceed with our analysis. Recall that Mohri et al. (2012) give an empirical Rademacher complexity (ERC)-based generalization bound, which is restated in the following lemma with F γ,t = {(X t , z t ) → γ (-M(f t (X t ), z t )) : f t ∈ F t } .

Lemma 1. Given a testing sequence (x t , z t ) T t=1 , with probability at least 1δ over samples S = (x i,t , z i,t ) T t=1 , i = 1, . . . , m , for every margin value γ > 0 and any f t ∈ F t , we have

EQUATION

Note that Lemma 1 adapts the original version (Theorem 3.1, Chapter 3.1, Mohri et al. (2012) ) for the multiclass ramp loss, and we have P( z t = z t ) ≤ R γ (f t ) by definition.

Now we only need to bound the ERC R S (F γ,t ). Our analysis consists of three steps. First, we characterize the Lipschitz continuity of vanilla RNNs w.r.t model parameters. Next, we bound the covering number of function class F t . At last, we derive an upper bound on R S (F γ,t ) via the standard machinery in the PAC-learning framework. Specifically, consider two different sets of weight matrices (U, V, W ) and (U , V , W ). Given the same activation operators and input data, denote the t-th output as y t and y t respectively. We characterize the Lipschitz property of y t 2 w.r.t model parameters in the following lemma. Lemma 2. Under Assumptions 1-3, given input (x t ) T t=1 and for any integer t ≤ T , y t 2 is Lipschitz in U , V and W , i.e., y ty t 2 ≤ L U,t U -

U F + L V,t V -V F + L W,t W -W F ,

where L U,t = ρ h B V B W ta t , L V,t = B W a t , and L W,t = B V a t with a t = ρ y ρ h B x

(ρ h B U ) t -1 ρ h B U -1 .

The detailed proof is provided in Appendix A.2. We give a simple example to illustrate the proof technique. Specifically, we consider a single layer network that outputs y = σ(W x), where x is the input, σ is an activation operator with Lipschitz parameter ρ, and W is a weight matrix. Such a network is Lipschitz in both x and W as follows. Given weight matrices W and W , we have y -

y 2 = σ(W x) -σ(W x) 2 ≤ ρ x 2 W -W F .

Additionally, given inputs x and x , we have y -

y 2 = σ(W x) -σ(W x ) 2 ≤ ρ W 2 x -x 2 .

Since vanilla RNNs are multilayer networks, Lemma 2 can be obtained by telescoping.

We remark that Lemma 2 is the key to the proof of our generalization bound, which separates the spectral norms of weight matrices and the total number of parameters.

Next, we bound the covering number of F t . Denote by N (F t , , dist(•, •)) the minimal cardinality of a subset C ⊂ F t that covers F t at scale w.r.t the metric dist(•, •), such that for any f t ∈ F t , there exists

f t ∈ C satisfying dist(f t , f t ) = sup Xt f t (X t ) -f t (X t ) 2 ≤ .

The following lemma gives an upper bound on N (F t , , dist(•, •)). Lemma 3. Under Assumptions 1-3, given any > 0, the covering number of F t satisfies

N (F t , , dist(•, •)) ≤ 1 + 6c √ dt ((ρ h B U ) t -1) (ρ h B U -1) 3d 2

,

where c = ρ y ρ h B V B W B x max {1, ρ h B U }.

The detailed proof is provided in Appendix A.3. We briefly explain the proof technique. Given activation operators, since vanilla RNNs are in parametric forms, f t has a one-to-one correspondence to its weight matrices U, V , and W . Lemma 2 implies that dist(•, •) is controlled by the Frobenius norms of the difference of weight matrices. Thus, it suffices to bound the covering numbers of three weight matrices, which can be obtained by the standard machinery. The product of covering numbers of three weight matrices gives us Lemma 3.

Lastly, we give an upper bound on R S (F γ,t ) in the following lemma. Lemma 4. Under Assumptions 1-3, given activation operators and samples S = {(x i,t , z i,t ) T t=1 , i = 1, . . . , m}, the empirical Rademacher complexity R S (F γ,t ) satisfies

R S (F γ,t ) = O dρ y B V min b √ d, ρ h B x B W (ρ h B U ) t -1 ρ h B U -1 log t √ dm (ρ h B U ) t -1 ρ h B U -1 √ mγ

.

The detailed proof is provided in Appendix A.4. Our proof exploits the Lipschitz continuity of M and γ , and uses Dudley's entropy integral as the standard machinery to establish Lemma 4.

Combining Lemma 1 and Lemma 4, we complete the proof.

4. REFINED GENERALIZATION BOUNDS:

When additional norm constraints on weight matrices U, V and W are available, we can further refine generalization bounds. Specifically, we consider the following assumptions. Assumption 4. The weight matrices satisfy

U 2,1 ≤ M U , V 2,1 ≤ M V , and W 2,1 ≤ M W . Assumption 5. The weight matrices satisfy U F ≤ B U,F , V F ≤ B V,F , and W F ≤ B W,F .

Note that Assumption 4 appears in Bartlett et al. (2017) and Assumption 5 appears in Neyshabur et al. (2017) . We have an equivalent relation between matrix norms, i.e.,

• 2 ≤ • 2,1 ≤ √ d • F ≤ d • 2 .

Comparing to Assumption 2, Assumptions 4 and 5 further restrict the model class. We then establish the following generalization bounds. Theorem 3. Let activation operators σ h and σ y be given, and Assumptions 1-3 hold. Then for (x t , z t ) T t=1 and S = (x i,t , z i,t ) T t=1 , i = 1, . . . , m drawn i.i.d. from any underlying distribution over R dx×T × {1, . . . , K}, with probability at least 1δ over S, for every margin value γ > 0 and every f t ∈ F t for integer t ≤ T , the following two bounds hold:

• Suppose Assumption 4 also holds. We have

EQUATION

where

α = ρ 2 h ρ y B V B W B x , d = d x d h + d 2 h + d h d y , and β = ρ h B U .

• Suppose Assumption 5 also holds. We have

EQUATION

where

λ t = min b √ d, ρ h B x B W β t -1 β-1 , d = d x d h + d 2 h + d h d y , and β = ρ h B U .

The detailed proof is provided in Appendix B.1. The first bound (3) adapts the matrix covering lemma in Bartlett et al. (2017) . The second bound (4) adapts the PAC-Bayes approach (Neyshabur et al., 2017) by analyzing the divergence when imposing small perturbations on the weight matrices.

We compare generalization bounds in Table 1 by differentiating ranges2 of β. It can be seen that when β > 1, both bounds (3) and ( 4) involve an exponential term in β. Thus, Theorem 2 yields a better result. When β ≤ 1, we distinguish two cases: the extreme case and the approximately low rank case. Specifically, remember that in the extreme case, we have

• 2,1 = d • 2 and • F = √ d • 2 , e.

g., orthogonal weight matrices. Therefore, bound (4) meets Theorem 2 for β < 1 and is worse for β = 1. Bound (3) is worse than Theorem 2 for β ≤ 1. On the other hand, if the weight matrices are approximately low rank, we have

• 2,1 d • 2 and • F √ d • 2 .

In this case, bound (4) improves Theorem 2 for β < 1 by reducing dependence on d. Additionally, if t (M U + M V + M W ) < d, bound (3) yields tighter results both for β < 1 and β = 1. Note that t (M U + M V + M W ) < d also implies that the input sequence is relatively short.

Table 1 : Generalization bounds in Theorems 2 and 3 with respect to different ranges of β.

Theorem 2:

Theorem 3 Bound (3) Bound ( 4)

β < 1 O d √ mγ O t(M U +M V +M W ) √ mγ h t 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " v 4 Z Z + 3 n 2 7 Q M 0 F D e M D B f x A n l v Q n Y = " > A A A B 8 H i c b V A 9 S w N B E J 3 z M 8 a v q K X N Y i L Y G O 7 S a C U B G 8 s I 5 k O S I + x t 9 p I l u 3 f H 7 p w Q j v w K G w t F b P 0 5 d v 4 b N 8 k V m v h g 4 P H e D D P z g k Q K g 6 7 7 7 a y t b 2 x u b R d 2 i r t 7 + w e H p a P j l o l T z X i T x T L W n Y A a L k X E m y h Q 8 k 6 i O V W B 5 O 1 g f D v z 2 0 9 c G x F H D z h J u K / o M B K h Y B S t 9 F g Z 9 T O 8 9 K a V f q n s V t 0 5 y C r x c l K G H I 1 + 6 a s 3 i F m q e I R M U m O 6 n p u g n 1 G N g k k + L f Z S w x P K x n T I u 5 Z G V H H j Z / O D p + T c K g M S x t p W h G S u / p 7 I q D J m o g L b q S i O z L I 3 E / / z u i m G 1 3 4 m o i R F H r H F o j C V B G M y + 5 4 M h O Y M 5 c Q S y r S w t x I 2 o p o y t B k V b Q j e 8 s u r p F W r e m 7 V u 6 + V 6 z d 5 H A U 4 h T O 4 A A + u o A 5 3 0 I A m M F D w D K / w 5 m j n x X l 3 P h a t a 0 4 + c w J / 4 H z + A L L b j 6 c = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " v 4 Z Z + 3 n 2 7 Q M 0 F D e M D B f x A n l v Q n Y = " > A A A B 8 H i c b V A 9 S w N B E J 3 z M 8 a v q K X N Y i L Y G O 7 S a C U B G 8 s I 5 k O S I + x t 9 p I l u 3 f H 7 p w Q j v w K G w t F b P 0 5 d v 4 b N 8 k V m v h g 4 P H e D D P z g k Q K g 6 7 7 7 a y t b 2 x u b R d 2 i r t 7 + w e H p a P j l o l T z X i T x T L W n Y A a L k X E m y h Q 8 k 6 i O V W B 5 O 1 g f D v z 2 0 9 c G x F H D z h J u K / o M B K h Y B S t 9 F g Z 9 T O 8 9 K a V f q n s V t 0 5 y C r x c l K G H I 1 + 6 a s 3 i F m q e I R M U m O 6 n p u g n 1 G N g k k + L f Z S w x P K x n T I u 5 Z G V H H j Z / O D p + T c K g M S x t p W h G S u / p 7 I q D J m o g L b q S i O z L I 3 E / / z u i m G 1 3 4 m o i R F H r H F o j C V B G M y + 5 4 M h O Y M 5 c Q S y r S w t x I 2 o p o y t B k V b Q j e 8 s u r p F W r e m 7 V u 6 + V 6 z d 5 H A U 4 h T O 4 A A + u o A 5 3 0 I A m M F D w D K / w 5 m j n x X l 3 P h a t a 0 4 + c w J / 4 H z + A L L b j 6 c = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " v 4 Z Z + 3 n 2 7 Q M 0 F D e M D B f x A n l v Q n Y = " > A A A B 8 H i c b V A 9 S w N B E J 3 z M 8 a v q K X N Y i L Y G O 7 S a C U B G 8 s I 5 k O S I + x t 9 p I l u 3 f H 7 p w Q j v w K G w t F b P 0 5 d v 4 b N 8 k V m v h g 4 P H e D D P z g k Q K g 6 7 7 7 a y t b 2 x u b R d 2 i r t 7 + w e H p a P j l o l T z X i T x T L W n Y A a L k X E m y h Q 8 k 6 i O V W B 5 O 1 g f D v z 2 0 9 c G x F H D z h J u K / o M B K h Y B S t 9 F g Z 9 T O 8 9 K a V f q n s V t 0 5 y C r x c l K G H I 1 + 6 a s 3 i F m q e I R M U m O 6 n p u g n 1 G N g k k + L f Z S w x P K x n T I u 5 Z G V H H j Z / O D p + T c K g M S x t p W h G S u / p 7 I q D J m o g L b q S i O z L I 3 E / / z u i m G 1 3 4 m o i R F H r H F o j C V B G M y + 5 4 M h O Y M 5 c Q S y r S w t x I 2 o p o y t B k V b Q j e 8 s u r p F W r e m 7 V u 6 + V 6 z d 5 H A U 4 h T O 4 A A + u o A 5 3 0 I A m M F D w D K / w 5 m j n x X l 3 P h a t a 0 4 + c w J / 4 H z + A L L b j 6 c = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " v 4 Z Z + 3 n 2 7 Q M 0 F D e M D B f x A n l v Q n Y = " > A A A B 8 H i c b V A 9 S w N B E J 3 z M 8 a v q K X N Y i L Y G O 7 S a C U B G 8 s I 5 k O S I + x t 9 p I l u 3 f H 7 p w Q j v w K G w t F b P 0 5 d v 4 b N 8 k V m v h g 4 P H e D D P z g k Q K g 6 7 7 7 a y t b 2 x u b R d 2 i r t 7 + w e H p a P j l o l T z X i T x T L W n Y A a L k X E m y h Q 8 k 6 i O V W B 5 O 1 g f D v z 2 0 9 c G x F H D z h J u K / o M B K h Y B S t 9 F g Z 9 T O 8 9 K a V f q n s V t 0 5 y C r x c l K G H I 1 + 6 a s 3 i F m q e I R M U m O 6 n p u g n 1 G N g k k + L f Z S w x P K x n T I u 5 Z G V H H j Z / O D p + T c K g M S x t p W h G S u / p 7 I q D J m o g L b q S i O z L I 3 E / / z u i m G 1 3 4 m o i R F H r H F o j C V B G M y + 5 4 M h O Y M 5 c Q S y r S w t x I 2 o p o y t B k V b Q j e 8 s u r p F W r e m 7 V u 6 + V 6 z d 5 H A U 4 h T O 4 A A + u o A 5 3 0 I A m M F D w D K / w 5 m j n x X l 3 P h a t a 0 4 + c w J / 4 H z + A L L b j 6 c = < / l a t e x i t > x t < l a t e x i t s h a 1 _ b a s e 6 4 = " w C 5 N f Z J u E u z b w i 8 I A C S S O v 3 l O A c = " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a r Q y J j S U m H p D A h e w t e 7 B h b + + y O 2 c k h N 9 g Y 6 E x t v 4 g O / + N C 1 y h 4 E

s m e X l v J j P z w l Q K g 6 7 7 7 R Q 2 N r e 2 d 4 q 7 p b 3 9 g 8 O j 8 v F J y y S Z Z t x n i U x 0 J 6 S G S 6 G 4 j w I l 7 6 S a 0

z i U v B 2 O b + d + + 5 F r I x L 1 g J O U B z E d K h E J R t F K f v W p j 9 V + u e L W 3 A X I O v F y U o E c z X 7 5 q z d I W B Z z h U x S Y 7 q e m 2 I w p R o F k 3 x W 6 m W G p 5 S N 6 Z B 3 L V U 0 5 i a Y L o 6 d k Q u r D E i U a F s K y U L 9 P T G l s T G T O L S d M c W R W f X m 4 n 9 e N 8 P o O p g K l W b I F V s u i j J J M C H z z 8 l A a M 5 Q T i y h T A t 7 K 2 E j q i l D m 0 / J h u C t v r x O W v W a 5 9 a 8 + 3 q l c Z P H U Y Q z O I d L 8 O A K G n A H T f C B g Y B n e I U 3 R z k v z r v z s W w t O P n M K f y B 8 / k D K E S O O Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " w C 5 N f Z J u E u z b w i 8 I A C S S O v 3 l O A c = " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a r Q y J j S U m H p D A h e w t e 7 B h b + + y O 2 c k h N 9 g Y 6 E x t v 4 g O / + N C 1 y h 4 E

s m e X l v J j P z w l Q K g 6 7 7 7 R Q 2 N r e 2 d 4 q 7 p b 3 9 g 8 O j 8 v F J y y S Z Z t x n i U x 0 J 6 S G S 6 G 4 j w I l 7 6 S a 0

z i U v B 2 O b + d + + 5 F r I x L 1 g J O U B z E d K h E J R t F K f v W p j 9 V + u e L W 3 A X I O v F y U o E c z X 7 5 q z d I W B Z z h U x S Y 7 q e m 2 I w p R o F k 3 x W 6 m W G p 5 S N 6 Z B 3 L V U 0 5 i a Y L o 6 d k Q u r D E i U a F s K y U L 9 P T G l s T G T O L S d M c W R W f X m 4 n 9 e N 8 P o O p g K l W b I F V s u i j J J M C H z z 8 l A a M 5 Q T i y h T A t 7 K 2 E j q i l D m 0 / J h u C t v r x O W v W a 5 9 a 8 + 3 q l c Z P H U Y Q z O I d L 8 O A K G n A H T f C B g Y B n e I U 3 R z k v z r v z s W w t O P n M K f y B 8 / k D K E S O O Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " w C 5 N f Z J u E u z b w i 8 I A C S S O v 3 l O A c = " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a r Q y J j S U m H p D A h e w t e 7 B h b + + y O 2 c k h N 9 g Y 6 E x t v 4 g O / + N C 1 y h 4 E

s m e X l v J j P z w l Q K g 6 7 7 7 R Q 2 N r e 2 d 4 q 7 p b 3 9 g 8 O j 8 v F J y y S Z Z t x n i U x 0 J 6 S G S 6 G 4 j w I l 7 6 S a 0

z i U v B 2 O b + d + + 5 F r I x L 1 g J O U B z E d K h E J R t F K f v W p j 9 V + u e L W 3 A X I O v F y U o E c z X 7 5 q z d I W B Z z h U x S Y 7 q e m 2 I w p R o F k 3 x W 6 m W G p 5 S N 6 Z B 3 L V U 0 5 i a Y L o 6 d k Q u r D E i U a F s K y U L 9 P T G l s T G T O L S d M c W R W f X m 4 n 9 e N 8 P o O p g K l W b I F V s u i j J J M C H z z 8 l A a M 5 Q T i y h T A t 7 K 2 E j q i l D m 0 / J h u C t v r x O W v W a 5 9 a 8 + 3 q l c Z P H U Y Q z O I d L 8 O A K G n A H T f C B g Y B n e I U 3 R z k v z r v z s W w t O P n M K f y B 8 / k D K E S O O Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " w C 5 N f Z J u E u z b w i 8 I A C S S O v 3 l O A c = " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a r Q y J j S U m H p D A h e w t e 7 B h b + + y O 2 c k h N 9 g Y 6 E x t v 4 g O / + N C 1 y h 4 E

s m e X l v J j P z w l Q K g 6 7 7 7 R Q 2 N r e 2 d 4 q 7 p b 3 9 g 8 O j 8 v F J y y S Z Z t x n i U x 0 J 6 S G S 6 G 4 j w I l 7 6 S a 0

z i U v B 2 O b + d + + 5 F r I x L 1 g J O U B z E d K h E J R t F K f v W p j 9 V + u e L W 3 A X I O v F y U o E c z X 7 5 q z d I W B Z z h U x S Y 7 q e m 2 I w p R o F k 3 x W 6 m W G p 5 S N 6 Z B 3 L V U 0 5 i a Y L o 6 d k Q u r D E i U a F s K y U L 9 P T G l s T G T O L S d M c W R W f X m 4 n 9 e N 8 P o O p g K l W b I F V s u i j J J M C H z z 8 l A a M 5 Q T i y h T A t 7 K 2 E j q i l D m 0 / J h u C t v r x O W v W a 5 9 a 8 + 3 q l c Z P H U Y Q z O I d L 8 O A K G n A H T f C B g Y B n e I U 3 R z k v z r v z s W w t O P n M K f y B 8 / k D K E S O O Q = = < / l a t e x i t > h t < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 p 2 S f 4 E V 1 f R Y c 7 3 8 z 6 O O N c 5 0 2 D o = " > A A A B 7 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L L a C p 5 L 0 o i c p e P F Y w b R C G 8 p m u 2 m X b j Z h d y K U 0 N / g x Y M i X v 1 B 3 v w 3 b t s c t P X B w O O 9 G W b m h a k U B l 3 3 2 y l t b G 5 t 7 5 R 3 K 3 v 7 B 4 d H 1 e O T j k k y z b j P E p n o x 5 A a L o X i P g q U / D H V n M a h 5 N 1 w c j v 3 u 0 9 c G 5 G o B 5 y m P I j p S I l I M I p W 8 u v j A d Y H 1 Z r b c B c g 6 8 Q r S A 0 K t A f V r / 4 w Y V n M F T J J j e l 5 b o p B T j U K J v m s 0 s 8 M T y m b 0 B H v W a p o z E 2 Q L 4 6 d k Q u r D E m U a F s K y U L 9 P Z H T 2 J h p H N r O m O L Y r H p z 8 T + v l 2 F 0 H e R C p R l y x Z a L o k w S T M j 8 c z I U m j O U U 0 s o 0 8 L e S t i Y a s r Q 5 l O x I X i r L 6 + T T r P h u Q 3 v v l l r 3 R R x l O E M z u E S P L i C F t x B G 3 x g I O A Z X u H N U c 6 L 8 + 5 8 L F t L T j F z C n / g f P 4 A D 9 S O K Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 p 2 S f 4 E V 1 f R Y c 7 3 8 z 6 O O N c 5 0 2 D o = " > A A A B 7 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L L a C p 5 L 0 o i c p e P F Y w b R C G 8 p m u 2 m X b j Z h d y K U 0 N / g x Y M i X v 1 B 3 v w 3 b t s c t P X B w O O 9 G W b m h a k U B l 3 3 2 y l t b G 5 t 7 5 R 3 K 3 v 7 B 4 d H 1 e O T j k k y z b j P E p n o x 5 A a L o X i P g q U / D H V n M a h 5 N 1 w c j v 3 u 0 9 c G 5 G o B 5 y m P I j p S I l I M I p W 8 u v j A d Y H 1 Z r b c B c g 6 8 Q r S A 0 K t A f V r / 4 w Y V n M F T J J j e l 5 b o p B T j U K J v m s 0 s 8 M T y m b 0 B H v W a p o z E 2 Q L 4 6 d k Q u r D E m U a F s K y U L 9 P Z H T 2 J h p H N r O m O L Y r H p z 8 T + v l 2 F 0 H e R C p R l y x Z a L o k w S T M j 8 c z I U m j O U U 0 s o 0 8 L e S t i Y a s r Q 5 l O x I X i r L 6 + T T r P h u Q 3 v v l l r 3 R R x l O E M z u E S P L i C F t x B G 3 x g I O A Z X u H N U c 6 L 8 + 5 8 L F t L T j F z C n / g f P 4 A D 9 S O K Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 p 2 S f 4 E V 1 f R Y c 7 3 8 z 6 O O N c 5 0 2 D o = " > A A A B 7 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L L a C p 5 L 0 o i c p e P F Y w b R C G 8 p m u 2 m X b j Z h d y K U 0 N / g x Y M i X v 1 B 3 v w 3 b t s c t P X B w O O 9 G W b m h a k U B l 3 3 2 y l t b G 5 t 7 5 R 3 K 3 v 7 B 4 d H 1 e O T j k k y z b j P E p n o x 5 A a L o X i P g q U / D H V n M a h 5 N 1 w c j v 3 u 0 9 c G 5 G o B 5 y m P I j p S I l I M I p W 8 u v j A d Y H 1 Z r b c B c g 6 8 Q r S A 0 K t A f V r / 4 w Y V n M F T J J j e l 5 b o p B T j U K J v m s 0 s 8 M T y m b 0 B H v W a p o z E 2 Q L 4 6 d k Q u r D E m U a F s K y U L 9 P Z H T 2 J h p H N r O m O L Y r H p z 8 T + v l 2 F 0 H e R C p R l y x Z a L o k w S T M j 8 c z I U m j O U U 0 s o 0 8 L e S t i Y a s r Q 5 l O x I X i r L 6 + T T r P h u Q 3 v v l l r 3 R R x l O E M z u E S P L i C F t x B G 3 x g I O A Z X u H N U c 6 L 8 + 5 8 L F t L T j F z C n / g f P 4 A D 9 S O K Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 p 2 S f 4 E V 1 f R Y c 7 3 8 z 6 O O N c 5 0 2 D o = " > A A A B 7 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L L a C p 5 L 0 o i c p e P F Y w b R C G 8 p m u 2 m X b j Z h d y K U 0 N / g x Y M i X v 1 B 3 v w 3 b t s c t P X B w O O 9 G W b m h a k U B l 3 3 2 y l t b G 5 t 7 5 R 3 K 3 v 7 B 4 d H 1 e O T j k k y z b j P E p n o x 5 A a L o X i P g q U / D H V n M a h 5 N 1 w c j v 3 u 0 9 c G 5 G o B 5 y m P I j p S I l I M I p W 8 u v j A d Y H 1 Z r b c B c g 6 8 Q r S A 0 K t A f V r / 4 w Y V n M F T J J j e l 5 b o p B T j U K J v m s 0 s 8 M T y m b 0 B H v W a p o z E 2 Q L 4 6 d k Q u r D E m U a F s K y U L 9 P Z H T 2 J h p H N r O m O L Y r H p z 8 T + v l 2 F 0 H e R C p R l y x Z a L o k w S T M j 8 c z I U m j O U U 0 s o 0 8 L e S t i Y a s r Q 5 l O x I X i r L 6 + T T r P h u Q 3 v v l l r 3 R R x l O E M z u E S P L i C F t x B G 3 x g I O A Z X u H N U c 6 L 8 + 5 8 L F t L T j F z C n / g f P 4 A D 9 S O K Q = = < / l a t e x i t > h t 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " v 4 Z Z + 3 n 2 7 Q M 0 F D e M D B f x A n l v Q n Y = " > A A A B 8 H i c b V A 9 S w N B E J 3 z M 8 a v q K X N Y i L Y G O 7 S a C U B G 8 s I 5 k O S I + x t 9 p I l u 3 f H 7 p w Q j v w K G w t F b P 0 5 d v 4 b N 8 k V m v h g 4 P H e D D P z g k Q K g 6 7 7 7 a y t b 2 x u b R d 2 i r t 7 + w e H p a P j l o l T z X i T x T L W n Y A a L k X E m y h Q 8 k 6 i O V W B 5 O 1 g f D v z 2 0 9 c G x F H D z h J u K / o M B K h Y B S t 9 F g Z 9 T O 8 9 K a V f q n s V t 0 5 y C r x c l K G H I 1 + 6 a s 3 i F m q e I R M U m O 6 n p u g n 1 G N g k k + L f Z S w x P K x n T I u 5 Z G V H H j Z / O D p + T c K g M S x t p W h G S u / p 7 I q D J m o g L b q S i O z L I 3 E / / z u i m G 1 3 4 m o i R F H r H F o j C V B G M y + 5 4 M h O Y M 5 c Q S y r S w t x I 2 o p o y t B k V b Q j e 8 s u r p F W r e m 7 V u 6 + V 6 z d 5 H A U 4 h T O 4 A A + u o A 5 3 0 I A m M F D w D K / w 5 m j n x X l 3 P h a t a 0 4 + c w J / 4 H z + A L L b j 6 c = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " v 4 Z Z + 3 n 2 7 Q M 0 F D e M D B f x A n l v Q n Y = " > A A A B 8 H i c b V A 9 S w N B E J 3 z M 8 a v q K X N Y i L Y G O 7 S a C U B G 8 s I 5 k O S I + x t 9 p I l u 3 f H 7 p w Q j v w K G w t F b P 0 5 d v 4 b N 8 k V m v h g 4 P H e D D P z g k Q K g 6 7 7 7 a y t b 2 x u b R d 2 i r t 7 + w e H p a P j l o l T z X i T x T L W n Y A a L k X E m y h Q 8 k 6 i O V W B 5 O 1 g f D v z 2 0 9 c G x F H D z h J u K / o M B K h Y B S t 9 F g Z 9 T O 8 9 K a V f q n s V t 0 5 y C r x c l K G H I 1 + 6 a s 3 i F m q e I R M U m O 6 n p u g n 1 G N g k k + L f Z S w x P K x n T I u 5 Z G V H H j Z / O D p + T c K g M S x t p W h G S u / p 7 I q D J m o g L b q S i O z L I 3 E / / z u i m G 1 3 4 m o i R F H r H F o j C V B G M y + 5 4 M h O Y M 5 c Q S y r S w t x I 2 o p o y t B k V b Q j e 8 s u r p F W r e m 7 V u 6 + V 6 z d 5 H A U 4 h T O 4 A A + u o A 5 3 0 I A m M F D w D K / w 5 m j n x X l 3 P h a t a 0 4 + c w J / 4 H z + A L L b j 6 c = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " v 4 Z Z + 3 n 2 7 Q M 0 F D e M D B f x A n l v Q n Y = " > A A A B 8 H i c b V A 9 S w N B E J 3 z M 8 a v q K X N Y i L Y G O 7 S a C U B G 8 s I 5 k O S I + x t 9 p I l u 3 f H 7 p w Q j v w K G w t F b P 0 5 d v 4 b N 8 k V m v h g 4 P H e D D P z g k Q K g 6 7 7 7 a y t b 2 x u b R d 2 i r t 7 + w e H p a P j l o l T z X i T x T L W n Y A a L k X E m y h Q 8 k 6 i O V W B 5 O 1 g f D v z 2 0 9 c G x F H D z h J u K / o M B K h Y B S t 9 F g Z 9 T O 8 9 K a V f q n s V t 0 5 y C r x c l K G H I 1 + 6 a s 3 i F m q e I R M U m O 6 n p u g n 1 G N g k k + L f Z S w x P K x n T I u 5 Z G V H H j Z / O D p + T c K g M S x t p W h G S u / p 7 I q D J m o g L b q S i O z L I 3 E / / z u i m G 1 3 4 m o i R F H r H F o j C V B G M y + 5 4 M h O Y M 5 c Q S y r S w t x I 2 o p o y t B k V b Q j e 8 s u r p F W r e m 7 V u 6 + V 6 z d 5 H A U 4 h T O 4 A A + u o A 5 3 0 I A m M F D w D K / w 5 m j n x X l 3 P h a t a 0 4 + c w J / 4 H z + A L L b j 6 c = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " v 4 Z Z + 3 n 2 7 Q M 0 F D e M D B f x A n l v Q n Y = " > A A A B 8 H i c b V A 9 S w N B E J 3 z M 8 a v q K X N Y i L Y G O 7 S a C U B G 8 s I 5 k O S I + x t 9 p I l u 3 f H 7 p w Q j v w K G w t F b P 0 5 d v 4 b N 8 k V m v h g 4 P H e D D P z g k Q K g 6 7 7 7 a y t b 2 x u b R d 2 i r t 7 + w e H p a P j l o l T z X i T x T L W n Y A a L k X E m y h Q 8 k 6 i O V W B 5 O 1 g f D v z 2 0 9 c G x F H D z h J u K / o M B K h Y B S t 9 F g Z 9 T O 8 9 K a V f q n s V t 0 5 y C r x c l K G H I 1 + 6 a s 3 i F m q e I R M U m O 6 n p u g n 1 G N g k k + L f Z S w x P K x n T I u 5 Z G V H H j Z / O D p + T c K g M S x t p W h G S u / p 7 I q D J m o g L b q S i O z L I 3 E / / z u i m G 1 3 4 m o i R F H r H F o j C V B G M y + 5 4 M h O Y M 5 c Q S y r S w t x I 2 o p o y t B k V b Q j e 8 s u r p F W r e m 7 V u 6 + V 6 z d 5 H A U 4 h T O 4 A A + u o A 5 3 0 I A m M F D w D K / w 5 m j n x X l 3 P h a t a 0 4 + c w J / 4 H z + A L L b j 6 c = < / l a t e x i t > x t < l a t e x i t s h a 1 _ b a s e 6 4 = " w C 5 N f Z J u E u z b w i 8 I A C S S O v 3 l O A c = " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a r Q y J j S U m H p D A h e w t e 7 B h b + + y O 2 c k h N 9 g Y 6 E x t v 4 g O / + N C 1 y h 4 E s m e X l v J j P z w l Q K g 6 7 7 7 R Q 2 N r e 2 d 4 q 7 p b 3 9 g 8 O j 8 v F J y y S Z Z t x n i U x 0 J 6 S G S 6 G 4 j w I l 7 6 S a 0 z i U v B 2 O b + d + + 5 F r I x L 1 g J O U B z E d K h E J R t F K f v W p j 9 V + u e L W 3 A X I O v F y U o E c z X 7 5 q z d I W B Z z h U x S Y 7 q e m 2 I w p R o F k 3 x W 6 m W G p 5 S N 6 Z B 3 L V U 0 5 i a Y L o 6 d k Q u r D E i U a F s K y U L 9 P T G l s T G T O L S d M c W R W f X m 4 n 9 e N 8 P o O p g K l W b I F V s u i j J J M C H z z 8 l A a M 5 Q T i y h T A t 7 K 2 E j q i l D m 0 / J h u C t v r x O W v W a 5 9 a 8 + 3 q l c Z P H U Y Q z O I d L 8 O A K G n A H T f C B g Y B n e I U 3 R z k v z r v z s W w t O P n M K f y B 8 / k D K E S O O Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " w C 5 N f Z J u E u z b w i 8 I A C S S O v 3 l O A c = " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a r Q y J j S U m H p D A h e w t e 7 B h b + + y O 2 c k h N 9 g Y 6 E x t v 4 g O / + N C 1 y h 4 E s m e X l v J j P z w l Q K g 6 7 7 7 R Q 2 N r e 2 d 4 q 7 p b 3 9 g 8 O j 8 v F J y y S Z Z t x n i U x 0 J 6 S G S 6 G 4 j w I l 7 6 S a 0 z i U v B 2 O b + d + + 5 F r I x L 1 g J O U B z E d K h E J R t F K f v W p j 9 V + u e L W 3 A X I O v F y U o E c z X 7 5 q z d I W B Z z h U x S Y 7 q e m 2 I w p R o F k 3 x W 6 m W G p 5 S N 6 Z B 3 L V U 0 5 i a Y L o 6 d k Q u r D E i U a F s K y U L 9 P T G l s T G T O L S d M c W R W f X m 4 n 9 e N 8 P o O p g K l W b I F V s u i j J J M C H z z 8 l A a M 5 Q T i y h T A t 7 K 2 E j q i l D m 0 / J h u C t v r x O W v W a 5 9 a 8 + 3 q l c Z P H U Y Q z O I d L 8 O A K G n A H T f C B g Y B n e I U 3 R z k v z r v z s W w t O P n M K f y B 8 / k D K E S O O Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " w C 5 N f Z J u E u z b w i 8 I A C S S O v 3 l O A c = " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a r Q y J j S U m H p D A h e w t e 7 B h b + + y O 2 c k h N 9 g Y 6 E x t v 4 g O / + N C 1 y h 4 E s m e X l v J j P z w l Q K g 6 7 7 7 R Q 2 N r e 2 d 4 q 7 p b 3 9 g 8 O j 8 v F J y y S Z Z t x n i U x 0 J 6 S G S 6 G 4 j w I l 7 6 S a 0 z i U v B 2 O b + d + + 5 F r I x L 1 g J O U B z E d K h E J R t F K f v W p j 9 V + u e L W 3 A X I O v F y U o E c z X 7 5 q z d I W B Z z h U x S Y 7 q e m 2 I w p R o F k 3 x W 6 m W G p 5 S N 6 Z B 3 L V U 0 5 i a Y L o 6 d k Q u r D E i U a F s K y U L 9 P T G l s T G T O L S d M c W R W f X m 4 n 9 e N 8 P o O p g K l W b I F V s u i j J J M C H z z 8 l A a M 5 Q T i y h T A t 7 K 2 E j q i l D m 0 / J h u C t v r x O W v W a 5 9 a 8 + 3 q l c Z P H U Y Q z O I d L 8 O A K G n A H T f C B g Y B n e I U 3 R z k v z r v z s W w t O P n M K f y B 8 / k D K E S O O Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " w C 5 N f Z J u E u z b w i 8 I A C S S O v 3 l O A c = " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a r Q y J j S U m H p D A h e w t e 7 B h b + + y O 2 c k h N 9 g Y 6 E x t v 4 g O / + N C 1 y h 4 E s m e X l v J j P z w l Q K g 6 7 7 7 R Q 2 N r e 2 d 4 q 7 p b 3 9 g 8 O j 8 v F J y y S Z Z t x n i U x 0 J 6 S G S 6 G 4 j w I l 7 6 S a 0 z i U v B 2 O b + d + + 5 F r I x L 1 g J O U B z E d K h E J R t F K f v W p j 9 V + u e L W 3 A X I O v F y U o E c z X 7 5 q z d I W B Z z h U x S Y 7 q e m 2 I w p R o F k 3 x W 6 m W G p 5 S N 6 Z B 3 L V U 0 5 i a Y L o 6 d k Q u r D E i U a F s K y U L 9 P T G l s T G T O L S d M c W R W f X m 4 n 9 e N 8 P o O p g K l W b I F V s u i j J J M C H z z 8 l A a M 5 Q T i y h T A t 7 K 2 E j q i l D m 0 / J h u C t v r x O W v W a 5 9 a 8 + 3 q l c Z P H U Y Q z O I d L 8 O A K G n A H T f C B g Y B n e I U 3 R z k v z r v z s W w t O P n M K f y B 8 / k D K E S O O Q = = < / l a t e x i t > h < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 P 0 M Q b 1 + i p K 1 v h U u U q N x b R C j p A = " > A A A B 8 X i c b V D L S g N B E O y N r x h f U Y 9 e B h P B U 9 j N R U 8 S 8 O I x g n l g s o T Z y W w y Z B 7 L z K w Q l v y F F w + K e P V v v P k 3 T p I 9 a G J B Q 1 H V T X d X l H B m r O 9 / e 4 W N z a 3 t n e J u a W / / 4 P C o f H z S N i r V h L a I 4 k p 3 I 2 w o Z 5 K 2 L L O c d h N N s Y g 4 7 U S T 2 7 n f e a L a M C U f 7 D S h o c A j y W J G s H X S Y 7 V v 2 E j g w b g 6 K F f 8 m r 8 A W i d B T i q Q o z k o f / W H i q S C S k s 4 N q Y X + I k N M 6 w t I 5 z O S v 3 U 0 A S T C R 7 R n q M S C 2 r C b H H x D F 0 4 Z Y h i p V 1 J i x b q 7 4 k M C 2 O m I n K d A t u x W f X m 4 n 9 e L 7 X x d Z g x m a S W S r J c F K c c W Y X m 7 6 M h 0 5 R Y P n U E E 8 3 c r Y i M s c b E u p B K L o R g 9 e V 1 0 q 7 X A r 8 W 3 N c r j Z s 8 j i K c w T l c Q g B X 0 I A 7 a E I L C E h 4 h l d 4 8 4 z 3 4 r 1 7 H 8 v W g p f P n M I f e J 8 / 0 t q Q V A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 P 0 M Q b 1 + i p K 1 v h U u U q N x b R C j p A = " > A A A B 8 X i c b V D L S g N B E O y N r x h f U Y 9 e B h P B U 9 j N R U 8 S 8 O I x g n l g s o T Z y W w y Z B 7 L z K w Q l v y F F w + K e P V v v P k 3 T p I 9 a G J B Q 1 H V T X d X l H B m r O 9 / e 4 W N z a 3 t n e J u a W / / 4 P C o f H z S N i r V h L a I 4 k p 3 I 2 w o Z 5 K 2 L L O c d h N N s Y g 4 7 U S T 2 7 n f e a L a M C U f 7 D S h o c A j y W J G s H X S Y 7 V v 2 E j g w b g 6 K F f 8 m r 8 A W i d B T i q Q o z k o f / W H i q S C S k s 4 N q Y X + I k N M 6 w t I 5 z O S v 3 U 0 A S T C R 7 R n q M S C 2 r C b H H x D F 0 4 Z Y h i p V 1 J i x b q 7 4 k M C 2 O m I n K d A t u x W f X m 4 n 9 e L 7 X x d Z g x m a S W S r J c F K c c W Y X m 7 6 M h 0 5 R Y P n U E E 8 3 c r Y i M s c b E u p B K L o R g 9 e V 1 0 q 7 X A r 8 W 3 N c r j Z s 8 j i K c w T l c Q g B X 0 I A 7 a E I L C E h 4 h l d 4 8 4 z 3 4 r 1 7 H 8 v W g p f P n M I f e J 8 / 0 t q Q V A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 P 0 M Q b 1 + i p K 1 v h U u U q N x b R C j p A = " > A A A B 8 X i c b V D L S g N B E O y N r x h f U Y 9 e B h P B U 9 j N R U 8 S 8 O I x g n l g s o T Z y W w y Z B 7 L z K w Q l v y F F w + K e P V v v P k 3 T p I 9 a G J B Q 1 H V T X d X l H B m r O 9 / e 4 W N z a 3 t n e J u a W / / 4 P C o f H z S N i r V h L a I 4 k p 3 I 2 w o Z 5 K 2 L L O c d h N N s Y g 4 7 U S T 2 7 n f e a L a M C U f 7 D S h o c A j y W J G s H X S Y 7 V v 2 E j g w b g 6 K F f 8 m r 8 A W i d B T i q Q o z k o f / W H i q S C S k s 4 N q Y X + I k N M 6 w t I 5 z O S v 3 U 0 A S T C R 7 R n q M S C 2 r C b H H x D F 0 4 Z Y h i p V 1 J i x b q 7 4 k M C 2 O m I n K d A t u x W f X m 4 n 9 e L 7 X x d Z g x m a S W S r J c F K c c W Y X m 7 6 M h 0 5 R Y P n U E E 8 3 c r Y i M s c b E u p B K L o R g 9 e V 1 0 q 7 X A r 8 W 3 N c r j Z s 8 j i K c w T l c Q g B X 0 I A 7 a E I L C E h 4 h l d 4 8 4 z 3 4 r 1 7 H 8 v W g p f P n M I f e J 8 / 0 t q Q V A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 P 0 M Q b 1 + i p K 1 v h U u U q N x b R C j p A = " > A A A B 8 X i c b V D L S g N B E O y N r x h f U Y 9 e B h P B U 9 j N R U 8 S 8 O I x g n l g s o T Z y W w y Z B 7 L z K w Q l v y F F w + K e P V v v P k 3 T p I 9 a G J B Q 1 H V T X d X l H B m r O 9 / e 4 W N z a 3 t n e J u a W / / 4 P C o f H z S N i r V h L a I 4 k p 3 I 2 w o Z 5 K 2 L L O c d h N N s Y g 4 7 U S T 2 7 n f e a L a M C U f 7 D S h o c A j y W J G s H X S Y 7 V v 2 E j g w b g 6 K F f 8 m r 8 A W i d B T i q Q o z k o f / W H i q S C S k s 4 N q Y X + I k N M 6 w t I 5 z O S v 3 U 0 A S T C R 7 R n q M S C 2 r C b H H x D F 0 4 Z Y h i p V 1 J i x b q 7 4 k M C 2 O m I n K d A t u x W f X m 4 n 9 e L 7 X x d Z g x m a S W S r J c F K c c W Y X m 7 6 M h 0 5 R Y P n U E E 8 3 c r Y i M s c b E u p B K L o R g 9 e V 1 0 q 7 X A r 8 W 3 N c r j Z s 8 j i K c w T l c Q g B X 0 I A 7 a E I L C E h 4 h l d 4 8 4 z 3 4 r 1 7 H 8 v W g p f P n M I f e J 8 / 0 t q Q V A = = < / l a t e x i t > + < l a t e x i t s h a 1 _ b a s e 6 4 = " R g Q F r g P P 0 9 G 9 w i N k C B n z A 2 E s p t M = " > A A A B 6 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x V Y Q h J L 0 o s e i F 4 8 V 7 Q e 0 o W y 2 m 3 b p Z h N 2 J 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k E h h 0 H W / n b X 1 j c 2 t 7 c J O c X d v / + C w d H T c M n G q G W + y W M a 6 E 1 D D p V C 8 i Q I l 7 y S a 0 y i Q v B 2 M b 2 d + + 4 l r I 2 L 1 i J O E + x E d K h E K R t F K D 5 X L S r 9 U d q v u H G S V e D k p Q 4 5 G v / T V G 8 Q s j b h C J q k x X c 9 N 0 M + o R s E k n x Z 7 q e E J Z W M 6 5 F 1 L F Y 2 4 8 b P 5 q V N y b p U B C W N t S y G Z q 7 8 n M h o Z M 4 k C 2 x l R H J l l b y b + 5 3 V T D K / 9 T K g k R a 7 Y Y l G Y S o I x m f 1 N B k J z h n J i C W V a 2 F s J G 1 F N G d p 0 i j Y E b / n l V d K q V T 2 3 6 t 3 X y v W b P I 4 C n M I Z X I A H V 1 C H O 2 h A E x g M 4 R l e 4 c 2 R z o v z 7 n w s W t e c f O Y E / s D 5 / A E o U Y 0 J < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " R g Q F r g P P 0 9 G 9 w i N k C B n z A 2 E s p t M = " > A A A B 6 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x V Y Q h J L 0 o s e i F 4 8 V 7 Q e 0 o W y 2 m 3 b p Z h N 2 J 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k E h h 0 H W / n b X 1 j c 2 t 7 c J O c X d v / + C w d H T c M n G q G W + y W M a 6 E 1 D D p V C 8 i Q I l 7 y S a 0 y i Q v B 2 M b 2 d + + 4 l r I 2 L 1 i J O E + x E d K h E K R t F K D 5 X L S r 9 U d q v u H G S V e D k p Q 4 5 G v / T V G 8 Q s j b h C J q k x X c 9 N 0 M + o R s E k n x Z 7 q e E J Z W M 6 5 F 1 L F Y 2 4 8 b P 5 q V N y b p U B C W N t S y G Z q 7 8 n M h o Z M 4 k C 2 x l R H J l l b y b + 5 3 V T D K / 9 T K g k R a 7 Y Y l G Y S o I x m f 1 N B k J z h n J i C W V a 2 F s J G 1 F N G d p 0 i j Y E b / n l V d K q V T 2 3 6 t 3 X y v W b P I 4 C n M I Z X I A H V 1 C H O 2 h A E x g M 4 R l e 4 c 2 R z o v z 7 n w s W t e c f O Y E / s D 5 / A E o U Y 0 J < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " R g Q F r g P P 0 9 G 9 w i N k C B n z A 2 E s p t M = " > A A A B 6 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x V Y Q h J L 0 o s e i F 4 8 V 7 Q e 0 o W y 2 m 3 b p Z h N 2 J 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k E h h 0 H W / n b X 1 j c 2 t 7 c J O c X d v / + C w d H T c M n G q G W + y W M a 6 E 1 D D p V C 8 i Q I l 7 y S a 0 y i Q v B 2 M b 2 d + + 4 l r I 2 L 1 i J O E + x E d K h E K R t F K D 5 X L S r 9 U d q v u H G S V e D k p Q 4 5 G v / T V G 8 Q s j b h C J q k x X c 9 N 0 M + o R s E k n x Z 7 q e E J Z W M 6 5 F 1 L F Y 2 4 8 b P 5 q V N y b p U B C W N t S y G Z q 7 8 n M h o Z M 4 k C 2 x l R H J l l b y b + 5 3 V T D K / 9 T K g k R a 7 Y Y l G Y S o I x m f 1 N B k J z h n J i C W V a 2 F s J G 1 F N G d p 0 i j Y E b / n l V d K q V T 2 3 6 t 3 X y v W b P I 4 C n M I Z X I A H V 1 C H O 2 h A E x g M 4 R l e 4 c 2 R z o v z 7 n w s W t e c f O Y E / s D 5 / A E o U Y 0 J < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " R g Q F r g P P 0 9 G 9 w i N k C B n z A 2 E s p t M = " > A A A B 6 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x V Y Q h J L 0 o s e i F 4 8 V 7 Q e 0 o W y 2 m 3 b p Z h N 2 J 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k E h h 0 H W / n b X 1 j c 2 t 7 c J O c X d v / + C w d H T c M n G q G W + y W M a 6 E 1 D D p V C 8 i Q I l 7 y S a 0 y i Q v B 2 M b 2 d + + 4 l r I 2 L 1 i J O E + x E d K h E K R t F K D 5 X L S r 9 U d q v u H G S V e D k p Q 4 5 G v / T V G 8 Q s j b h C J q k x X c 9 N 0 M + o R s E k n x Z 7 q e E J Z W M 6 5 F 1 L F Y 2 4 8 b P 5 q V N y b p U B C W N t S y G Z q 7 8 n M h o Z M 4 k C 2 x l R H J l l b y b + 5 3 V T D K / 9 T K g k R a 7 Y Y l G Y S o I x m f 1 N B k J z h n J i C W V a 2 F s J G 1 F N G d p 0 i j Y E b / n l V d K q V T 2 3 6 t 3 X y v W b P I 4 C n M I Z X I A H V 1 C H O 2 h A E x g M 4 R l e 4 c 2 R z o v z 7 n w s W t e c f O Y E / s D 5 / A E o U Y 0 J < / l a t e x i t > + < l a t e x i t s h a 1 _ b a s e 6 4 = " R g Q F r g P P 0 9 G 9 w i N k C B n z A 2 E s p t M = " > A A A B 6 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x V Y Q h J L 0 o s e i F 4 8 V 7 Q e 0 o W y 2 m 3 b p Z h N 2 J 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k E h h 0 H W / n b X 1 j c 2 t 7 c J O c X d v / + C w d H T c M n G q G W + y W M a 6 E 1 D D p V C 8 i Q I l 7 y S a 0 y i Q v B 2 M b 2 d + + 4 l r I 2 L 1 i J O E + x E d K h E K R t F K D 5 X L S r 9 U d q v u H G S V e D k p Q 4 5 G v / T V G 8 Q s j b h C J q k x X c 9 N 0 M + o R s E k n x Z 7 q e E J Z W M 6 5 F 1 L F Y 2 4 8 b P 5 q V N y b p U B C W N t S y G Z q 7 8 n M h o Z M 4 k C 2 x l R H J l l b y b + 5 3 V T D K / 9 T K g k R a 7 Y Y l G Y S o I x m f 1 N B k J z h n J i C W V a 2 F s J G 1 F N G d p 0 i j Y E b / n l V d K q V T 2 3 6 t 3 X y v W b P I 4 C n M I Z X I A H V 1 C H O 2 h A E x g M 4 R l e 4 c 2 R z o v z 7 n w s W t e c f O Y E / s D 5 / A E o U Y 0 J < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " R g Q F r g P P 0 9 G 9 w i N k C B n z A 2 E s p t M = " > A A A B 6 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x V Y Q h J L 0 o s e i F 4 8 V 7 Q e 0 o W y 2 m 3 b p Z h N 2 J 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k E h h 0 H W / n b X 1 j c 2 t 7 c J O c X d v / + C w d H T c M n G q G W + y W M a 6 E 1 D D p V C 8 i Q I l 7 y S a 0 y i Q v B 2 M b 2 d + + 4 l r I 2 L 1 i J O E + x E d K h E K R t F K D 5 X L S r 9 U d q v u H G S V e D k p Q 4 5 G v / T V G 8 Q s j b h C J q k x X c 9 N 0 M + o R s E k n x Z 7 q e E J Z W M 6 5 F 1 L F Y 2 4 8 b P 5 q V N y b p U B C W N t S y G Z q 7 8 n M h o Z M 4 k C 2 x l R H J l l b y b + 5 3 V T D K / 9 T K g k R a 7 Y Y l G Y S o I x m f 1 N B k J z h n J i C W V a 2 F s J G 1 F N G d p 0 i j Y E b / n l V d K q V T 2 3 6 t 3 X y v W b P I 4 C n M I Z X I A H V 1 C H O 2 h A E x g M 4 R l e 4 c 2 R z o v z 7 n w s W t e c f O Y E / s D 5 / A E o U Y 0 J < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " R g Q F r g P P 0 9 G 9 w i N k C B n z A 2 E s p t M = " > A A A B 6 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x V Y Q h J L 0 o s e i F 4 8 V 7 Q e 0 o W y 2 m 3 b p Z h N 2 J 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k E h h 0 H W / n b X 1 j c 2 t 7 c J O c X d v / + C w d H T c M n G q G W + y W M a 6 E 1 D D p V C 8 i Q I l 7 y S a 0 y i Q v B 2 M b 2 d + + 4 l r I 2 L 1 i J O E + x E d K h E K R t F K D 5 X L S r 9 U d q v u H G S V e D k p Q 4 5 G v / T V G 8 Q s j b h C J q k x X c 9 N 0 M + o R s E k n x Z 7 q e E J Z W M 6 5 F 1 L F Y 2 4 8 b P 5 q V N y b p U B C W N t S y G Z q 7 8 n M h o Z M 4 k C 2 x l R H J l l b y b + 5 3 V T D K / 9 T K g k R a 7 Y Y l G Y S o I x m f 1 N B k J z h n J i C W V a 2 F s J G 1 F N G d p 0 i j Y E b / n l V d K q V T 2 3 6 t 3 X y v W b P I 4 C n M I Z X I A H V 1 C H O 2 h A E x g M 4 R l e 4 c 2 R z o v z 7 n w s W t e c f O Y E / s D 5 / A E o U Y 0 J < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " R g Q F r g P P 0 9 G 9 w i N k C B n z A 2 E s p t M = " > A A A B 6 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x V Y Q h J L 0 o s e i F 4 8 V 7 Q e 0 o W y 2 m 3 b p Z h N 2 J 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k E h h 0 H W / n b X 1 j c 2 t 7 c J O c X d v / + C w d H T c M n G q G W + y W M a 6 E 1 D D p V C 8 i Q I l 7 y S a 0 y i Q v B 2 M b 2 d + + 4 l r I 2 L 1 i J O E + x E d K h E K R t F K D 5 X L S r 9 U d q v u H G S V e D k p Q 4 5 G v / T V G 8 Q s j b h C J q k x X c 9 N 0 M + o R s E k n x Z 7 q e E J Z W M 6 5 F 1 L F Y 2 4 8 b P 5 q V N y b p U B C W N t S y G Z q 7 8 n M h o Z M 4 k C 2 x l R H J l l b y b + 5 3 V T D K / 9 T K g k R a 7 Y Y l G Y S o I x m f 1 N B k J z h n J i C W V a 2 F s J G 1 F N G d p 0 i j Y E b / n l V d K q V T 2 3 6 t 3 X y v W b P I 4 C n M I Z X I A H V 1 C H O 2 h A E x g M 4 R l e 4 c 2 R z o v z 7 n w s W t e c f O Y E / s D 5 / A E o U Y 0 J < / l a t e x i t > Figure 2: A basic building block of MGU RNNs where W r , W h ∈ R d h ×dx , U r , U h ∈ R d h ×d h , V ∈ R dy×d h , and r t ∈ R d h .

The notation denotes the Hadamard product (entrywise product) of vectors. Denote by F g,t the class of mappings from the first t inputs to the t-th output computed by gated (MGU or LSTM) RNNs. For simplicity, we consider σ being the sigmoid function, i.e., σ(x) = (1 + exp(-x)) -1 , σ h (•) = tanh(•), and σ y being ρ y -Lipschitz with σ y (0) = 0. Extensions to general Lipschitz activation operators as in Assumption 3 are straightforward. Suppose we have h 0 = 0 and the following assumption. Assumption 6. All the weight matrices have bounded spectral norms respectively, i.e.

W r 2 ≤ B Wr , W h 2 ≤ B W h , U r 2 ≤ B Ur , U h 2 ≤ B U h , and V 2 ≤ B V .

Using a similar argument for vanilla RNNs yields a generalization bound of MGU RNNs as follows.

Theorem 4. Let the activation operator σ y be given and Assumptions 1 and 6 hold. Then for (x t , z t ) T t=1 and S = (x i,t , z i,t ) T t=1 , i = 1, . . . , m drawn i.i.d. from any underlying distribution over R dx×T × {1, . . . , K}, with probability at least 1δ over S, for every margin value γ > 0 and every f t ∈ F g,t for integer t ≤ T , we have

P ( z t = z t ) ≤ R γ (f t ) + O dρ y B V min √ d,B W h B x β t -1 β-1 log θ t -1 θ-1 d √ m √ mγ + log 1 δ m , where d = 2d x d h + 2d 2 h + d h d y , β = max j≤t 1 -r j ∞ + B U h r j 2 ∞ , and θ = β + 2B Ur + B Ur B U h .

The detailed proof is provided in Appendix C.1. As can be seen, r t shrinks the magnitude of hidden state to reduce the dependence on d and t in generalization. Specifically, the hidden state h t is entrywise bounded by 1. Then for any integer t ≤ T , we have

r t ∞ ≤ 1 1 + exp (-B Wr B x -U r 1 ) and 1 -r t ∞ ≤ 1 1 + exp (-B Wr B x -U r 1 )

,

where U r denotes the maximal absolute row sum. By restricting B Wr and U r 1 sufficiently small, we can guarantee that β and θ are strictly smaller than 1 when

B U h = 1 or even B U h > 1.

As a result, with proper normalization of weight matrices, the generalization bound of MGU RNNs is less dependent on d and t than that of vanilla RNNs.

The LSTM RNNs are more complicated than MGU RNNs, which introduce more gates to control the information flow in RNNs. LSTM RNNs have two hidden states, and compute them as,

g t = σ(W g x t + U g h t-1 ), r t = σ(W r x t + U r h t-1 ), o t = σ(W o x t + U o h t-1 ), c t = σ c (W c x t + U c h t-1 ) , c t = g t c t-1 + r t c t , h t = o t tanh(c t ), Figure 3: A basic building block of LSTM RNNs where W g , W r , W o , W c ∈ R d h ×dx , U g , U r , U o , U c ∈ R d h ×d h , and g t , r t , o t ∈ R d h .

For simplicity, we also consider σ being the sigmoid function, and

σ c (•) = tanh(•). The t-th output is y t = σ y (V h t )

, where V ∈ R dy×d h , and σ y is ρ y -Lipschitz with σ y (0) = 0. Suppose we have h 0 = c 0 = 0 and the following assumption.

Assumption 7. The spectral norms of weight matrices are bounded respectively, i.e.

W g 2 ≤ B Wg , W r ≤ B Wr , W o 2 ≤ B Wo , W c 2 ≤ B Wc , U g 2 ≤ B Ug , U r 2 ≤ B Ur , U o 2 ≤ B Uo , U h ≤ B U h , and V 2 ≤ B V .

For properly normalized weight matrices W o and U o , the generalization bound of LSTM RNNs is given in the following theorem. Theorem 5. Let the activation operator σ y be given and Assumptions 1 and 7 hold. Then for (x t , z t ) T t=1 and S = (x i,t , z i,t ) T t=1 , i = 1, . . . , m drawn i.i.d. from any underlying distribution over R dx×T × {1, . . . , K}, with probability at least 1δ over S, for every margin value γ > 0 and every f t ∈ F g,t for integer t ≤ T , we have

P ( z t = z t ) ≤ R γ (f t ) + O dρ y B V min √ d,B W c B x β t -1 β-1 log θ t -1 θ-1 d √ m √ mγ + log 1 δ m , where d = 4d x d h + 4d 2 h + d h d y , β = max j≤t g j ∞ + B Uc r j ∞ o j ∞ and θ = β + B Ug + B Ur + B Uo .

The detailed proof is provided in Appendix C.2. Similar to MGU RNNs, LSTM RNNs also introduce extra decaying factors to reduce the dependence on d and t in generalization. However, LSTM RNNs are more complicated, but more flexible than MGU RNNs, since three factors, r t , o t and g t are used to jointly control the spectrum of U c . We further remark that LSTM RNNs also need the spectral norms of weight matrices, W g , W r , W o , U g , U r , and U o , to be properly controlled such that better generalization bounds can be obtained. Remark 1. We also extend our analysis to convolutional RNNs (Conv RNNs, Pinheiro and Collobert (2014) ; Liang and Hu (2015) ; Xingjian et al. (2015) ), and show that the convolutional filters in Conv RNNs can reduce the dependence on d through parameter sharing in generalization. Due to space limit, the detailed discussion is provided in Appendix D.

6. NUMERICAL EVALUATION:

We demonstrate a comparison among our obtained generalization bound with Bartlett et al. (2017) , Neyshabur et al. (2017), and Zhang et al. (2018) . Specifically, we train a vanilla RNN on the wikitext long term dependency language modeling dataset. We take σ h = tanh and set the hidden state h ∈ R 128 and the input x ∈ [0, 1] 14 . Accordingly, we have d = 128 and take the sequence length t = 56. We list the complexity bounds for vanilla RNNs in Theorem 2 (Ours), Zhang et al. (2018) (Bound 1), (3) of Theorem 3 (Bound 2), and (4) of Theorem 3 (Bound 3) in the following by neglecting common log factors in d and t, The corresponding complexity bounds are shown in Figure 4 in the logarithmic scale. As can be seen, our complexity bound in Theorem 2 is much smaller than Bounds 1-3. In more detail, the trained vanilla RNN takes B U = 2.6801 > 1. As discussed earlier, for B U > 1, only our bound in Theorem 2 is polynomial in the size of the network, while Bounds 1-3 are all exponential in t. The resulting complexity bounds corroborate such a conclusion.

• Ours: dB V min √ d, B W B t U -1 B U -1 log B t U -1 B U -1 ; • Bound 1: dt 2 B V B W max{1, B t U }; • Bound 2: B V B W (M U + M V + M W ) t B t U -1 B U -1 ; • Bound 3: min √ d, B W B t U -1 B U -1 B U + B W B t U -1 B U -1 d B 2 U,F + B 2 W,F + B 2 V,F .

Ours:

We also observe that Bound 3 is smaller than Bound 2. The reason behind is that the weight matrices in the trained vanilla RNN have relatively small Frobenius norms but large (2, 1) norms. For example, we have B U,F = 13.6823 and M U = 154.5439. Then, we can calculate the stable rank

B U,F B U = 5.1 < √ d

2 , and

M U B U,F = 11.3 ≈ √ d

, which implies the singular values of U are not evenly distributed, while the norms of row vectors in U are mostly approximately equal.

7. DISCUSSIONS AND OPEN PROBLEMS:

(I) Tighter bounds: Our obtained generalization bounds depend on the spectral norms of weight matrices and the network size (the total number of parameters). Can we exploit other modeling structures to further reduce the dependence on the network size? Or can we find better choices of norms of weight matrices that yield better bounds?

(II) Margin value: Our generalization bounds depend on the margin value of the predictors. As can be seen, a larger margin value yields a better generalization bound. However, establishing a sharp characterization of the margin value is technically very challenging, because of its complicated dependence on the underlying data distribution and the training algorithm.

(III) Implicit bias of SGD: Numerous empirical evidences have already shown that RNNs trained by stochastic gradient descent (SGD) algorithms have superior generalization performance. There have been a few theoretical results showing that SGD tends to yield low complexity models, which can generalize (Neyshabur et al., 2014; 2015; Zhang et al., 2016; Soudry et al., 2017) . Can we extend their argument to RNNs? For example, can SGD always yield weight matrices with well controlled spectra? As mentioned, this is crucial to the generalization of MGU and LSTM RNNs, since not well controlled spectra may even hurt generalization.

(IV) Adaptivity to the underlying distribution: The current PAC-Learning framework focuses on the worst case. Taking classification as an example, the theoretical analysis holds even when the input features and labels are completely independent. Therefore, this often yields very pessimistic results. For many real applications, however, data are not obtained adversarially. Some recent empirical evidences suggest that the generalization of neural networks seems very adaptive to the underlying distribution: Easier tasks lead to low complexity neural networks, while harder ones lead to highly complex neural networks. Unfortunately, none of the existing analysis can take the underlying distribution into consideration.

A PROOFS IN SECTION 2 A.1 LIPSCHITZ CONTINUITY OF M AND γ:

We show the Lipschitz continuity of the margin operator M and the loss function γ in the following lemma. Lemma 5. The margin operator M is 2-Lipschitz in its first argument with respect to vector Euclidean norm, and γ is 1 γ -Lipschitz.

Proof. Let y, y and z be given, then

M(y, z) -M (y , z) = y z -y z + max j =z y j -max j =z y j ≤ y z -y z + max j =z y j -y j ≤ 2 y -y ∞ ≤ y -y 2 .

For function γ , it is a piecewise linear function. Thus, it is straightforward to see that γ is 1 γ -Lipschitz.

A.2 PROOF OF LEMMA 2:

Proof. The Lemma is stated with matrix Frobenius norms. However, we can show a tighter bound only involving the spectral norms of weight matrices. Given weight matrices U, V, W and U , V , W , consider the t-th outputs y t and y t of vanilla RNNs,

y t -y t 2 = σ y (V h t ) -σ y (V h t ) 2 ≤ ρ y V h t -V h t + V h t -V h t 2 ≤ ρ y ( (V -V )h t 2 + V (h t -h t ) 2 ) ≤ ρ y ( h t 2 V -V 2 + B V h t -h t 2 ) .

(5)

We have to bound the norm of h t as in the following lemma. Lemma 6. Under Assumptions 1 to 3, for t ≥ 0, the norm of h t is bounded by

EQUATION

Proof. We prove by induction. Observe that for t ≥ 1, we have

EQUATION

Applying equation ( 7) recursively with h 0 = 0, we arrive at,

h t 2 ≤ ρ h B W B x t-1 j=0 (ρ h B U ) j = ρ h B W B x (ρ h B U ) t -1 ρ h B U -1 ,

We also have h t ∞ ≤ b. Thus, combining with the above upper bound, we get

h t 2 ≤ min b √ d, ρ h B W B x (ρ h B U ) t -1 ρ h B U -1

. Clearly, h 0 2 = 0 satisfies the upper bound.

When ρ h B U = 1, the ratio is defined, by L'Hospital's rule, to be the limit,

lim ρ h B U →1 (ρ h B U ) t -1 ρ h B U -1 = t.

With Lemma 6 in hand, we plug the bound (6) into equation ( 5) and end up with

y t -y t 2 ≤ ρ y ρ h B W B x (ρ h B U ) t -1 ρ h B U -1 V -V 2 + ρ y B V h t -h t 2 . (8)

The remaining task is to bound h th t 2 in terms of the spectral norms of the difference of weight matrices, W -W 2 and U -U 2 .

Lemma 7. Under Assumptions 1 to 3, for t ≥ 1, the difference of hidden states h t and h t satisfies

h t -h t 2 ≤ L W,t W -W 2 + L U,t U -U 2 ,

where

L W,t = ρ h B x (ρ h B U ) t -1 ρ h B U -1 and L U,t = ρ 2 h B W B x t (ρ h B U ) t 2 -1 (ρ h B U )-1 .

Proof. Similar to the proof of Lemma 6, we use induction.

h t -h t 2 = σ h (W x t + U h t-1 ) -σ h W x t + U h t-1 2 ≤ ρ h (W -W )x t + U h t-1 -U h t-1 2 ≤ ρ h (W -W )x t 2 + U h t-1 -U h t-1 2 ≤ ρ h B x W -W 2 + U h t-1 -U h t-1 + U h t-1 -U h t-1 2 ≤ ρ h B x W -W 2 + ρ h h t-1 2 U -U 2 + B U h t-1 -h t-1 2 .

Repeat this derivation recursively, we have

h t -h t 2 ≤ ρ h B x W -W 2 + ρ h h t-1 2 U -U 2 + ρ h B U h t-1 -h t-1 2 ≤ ρ h B x (1 + ρ h B U ) W -W 2 + ρ h ( h t-1 2 + ρ h B U h t-2 2 ) U -U 2 + (ρ h B U ) 2 h t-2 -h t-2 2 ≤ . . . . . . ≤ ρ h B x t-1 j=0 (ρ h B U ) j W -W 2 + ρ h t-1 j=0 (ρ h B U ) t-1-j h j 2 U -U 2 + (ρ h B U ) t h 0 -h 0 2 ≤ ρ h B x (ρ h B U ) t -1 ρ h B U -1 W -W 2 + ρ h t-1 j=0 (ρ h B U ) t-1-j h j 2 U -U 2 . (9)

We now plug in the upper bound (6) to calculate the summation involving the Euclidean norms of the hidden state h t .

t-1 j=0 (ρ h B U ) t-1-j h j 2 ≤ t-1 j=0 (j + 1)(ρ h B U ) j ρ h B W B x ≤ t t-1 j=0 (ρ h B U ) j ρ h B W B x ≤ ρ h B W B x t (ρ h B U ) t -1 ρ h B U -1 .

Plugging back into equation ( 9), we have as desired,

h t -h t 2 ≤ ρ h B x (ρ h B U ) t -1 ρ h B U -1 W -W 2 + ρ 2 h B W B x t (ρ h B U ) t -1 ρ h B U -1 U -U 2 .

Combining equation ( 8) and Lemma 7, and W F ≥ W 2 , we immediately get Lemma 2.

A.3 PROOF OF LEMMA 3:

Proof. Our goal is to construct a covering C(F t , , dist(•, •)), i.e., for any f t ∈ F t , there exists f t ∈ F t , for any input data (x t ) T t=1 , satisfying sup

Xt f t (X t ) -f t (X t ) 2 ≤ .

Note that f is determined by weight matrices U, V and W . By Lemma 2, we have

sup Xt f (X t ) -f t (X t ) 2 ≤ L V,t V -V F + L W,t W -W F + L U,t U -U F .

Then it is enough to construct three matrix coverings, 

C U, 3L U,t , • F , C V, 3L V,t , • F and C W, 3L W,t

N (G, , • F ) ≤ 1 + 2 min √ d 1 , √ d 2 λ d1d2 .

Proof. For any matrix A ∈ G, we define a mapping φ : R d1×d2 → R d1d2 , such that φ(A) = [A :,1 , A :,2 , . . . , A :,h ] , where A :,i denotes the i-th column of matrix A. Denote the vector space induced by the mapping φ by

V(G) = {φ(A) : A ∈ G}. Note that we have A 2 F = h i=1 A :,i A :,i = φ(A) 2

2 and the mapping φ is one-to-one and onto. By definition, the square of Frobenius norm equals the square of sum of singular values and the spectral norm is the largest singular value. Hence, the equivalence of Frobenius norm and spectral norm is given by the following inequalities,

A 2 ≤ A F ≤ min d 1 , d 2 A 2 . Now, we see that if we construct a covering C(V(G), , • 2 ), then φ -1 C(V(G), , • 2 ) = φ -1 (v) : v ∈ C(V(G), , • 2

) is a covering of G at scale with respect to the matrix Frobenius norm. Therefore, we get N (G, , • F ) ≤ N (V(G), , • 2 ). As a consequence, it is suffices to upper bound the covering number of V(G). In order to do so, we need another closely related concept, packing number. Definition 2 (Packing). Let G be an arbitrary set and > 0 be given. We say P(G, , • ) is a packing of G at scale with respect to the norm • , if for any two elements A, B ∈ P, we have A -B > . Denote by M(G, , • ) the maximal cardinality of P(G, , • ).

By the maximality, we can check that N (C, , • ) ≤ M(C, , • ). Indeed, let P * (G, , • ) be a maximal packing. Suppose there exists A ∈ G such that for any B ∈ P * (G, , • ), the inequality A -B > holds. Then we can add A to P * (G, , • ), while still keeping it being a packing, which contradicts the maximality of P * (G, , • ). Thus, we have

N (G, , • ) ≤ M(G, , • ). Observe that V(G) is contained in an Euclidean ball B(0; R) ∈ R d1d2 of radius at most R = max A∈G φ(A) 2 ≤ min d 1 , d 2 A 2 ≤ min d 1 , d 2 λ.

Additionally, the union of Euclidean balls B(v; /2) ⊂ R d1d2 with radius /2 and center v ∈

P(V(G), , • 2 ) is further contained in an Euclidean ball B(0; R ) of slightly enlarged radius R = min √ d 1 , √ d 2 λ + /2.

Those balls B(v; /2) are disjoint by the definition of packing, thus we have

N (V(C), , • 2 ) ≤ P(V(C), , • 2 ) ≤ vol(B(0, R )) vol(B(v; /2)) = R /2 d1d2 = 1 + 2 min{ √ d 1 , √ d 2 }λ d1d2 ,

where vol(•) denotes the volume.

By Lemma 8, we can directly write out the upper bounds on the covering numbers of weight matrices,

N U, 3L U,t , • F ≤ 1 + 6 √ d h B U L U,t d 2 h , N V, 3L V,t , • F ≤ 1 + 6 min{ d y , √ d h }B V L V,t dyd h

, and

N W, 3L W,t , • F ≤ 1 + 6 min{ √ d x , √ d h }B W L W,t dxd h .

Then we immediately have,

N (F t , , dist(•, •)) ≤ N U, 3L U,t , • F × N V, 3L V,t , • F × N W, 3L W,t , • F ≤ 1 + 6 √ d h B U L U,t d 2 h 1 + min{6 d y , √ d h }B V L V,t dyd h × 1 + 6 min{ √ d x , √ d h }B W L W,t dxd h .

Substituting the coefficients L U,t , L V,t and L W,t from Lemma 2, we get

N (F t , , dist(•, •)) ≤   1 + 6 √ dρ y ρ h B V B W B x (ρ h B U ) t -1 ρ h B U -1   2d 2   1 + 6 √ dρ y ρ 2 h B U B V B W B x t (ρ h B U ) t -1 ρ h B U -1   d 2 ≤   1 + 6c √ dt (ρ h B U ) t -1 ρ h B U -1   3d 2

,

where c = ρ y ρ h B V B W B x max {1, ρ h B U }.

For future usage, we also write down for small > 0,

such that 6c √ dt (ρ h B U ) t -1 ρ h B U -1 > 1, the logarithm of covering number satisfies, log N (F t , , dist(•, •)) ≤ 3d 2 log   12c √ dt (ρ h B U ) t -1 ρ h B U -1   . A.4 PROOF OF LEMMA 4 Proof. Define F M,t = {(X t , z t ) → M(f t (X t ), z t ) : f t ∈ F t }.

By Lemma 5, we see that M is 2-Lipschitz in its first argument. In order to cover F M,t at scale , it suffices to cover F t at scale 2 . This immediately gives us the covering number

N (F M,t , , • ∞ ) ≤ N (F t , /2, dist(•, •)).

We then give the statement of Dudley's entropy integral. Lemma 9. Let H be a real-valued function class taking values in [-r, r] for some constant r, and assume that 0 ∈ H. Let S = (s 1 , . . . , s m ) be given points, then

R S (H) ≤ inf α>0 4α √ m + 12 m 2r √ m α log N (H, , • )d .

The proof can be found in Bartlett et al. (2017) . Taking H = F M,t , we can easily verify that

F M,t takes values in [-r, r] with r = ρ y B V h t 2 ≤ ρ y B V min b √ d, ρ h B W B x (ρ h B U ) t -1 ρ h B U -1

and 0 ∈ F M . Thus, directly applying Lemma 9 yields the following bound,

R S (F M,t ) ≤ inf α>0 4α √ m + 12 m 2r √ m α log N (F M,t , , • ∞ )d .

We bound the integral as follows,

2r √ m α log N (F M,t , , • ∞ )d ≤ 2r √ m α 3d 2 log 24c √ dt (ρ h B U ) t -1 ρ h B U -1 d ≤ 2r √ m 3d 2 log 24c √ dt (ρ h B U ) t -1 ρ h B U -1 α . Picking α = 1 √ m is enough to give us an upper bound on R S (F M,t ), R S (F M ) ≤ 4 m + 24 √ m 3d 2 r 2 log 24c √ dmt (ρ h B U ) t -1 ρ h B U -1 .

Finally, by Talagrand's lemma (Mohri et al., 2012) and γ being 1 γ -Lipschitz, we have

R S (F γ,t ) ≤ 1 γ R S (F M,t ) ≤ 4 mγ + 24 √ mγ 3d 2 r 2 log 24c √ dmt (ρ h B U ) t -1 ρ h B U -1 . B PROOF IN SECTION 4 B.1 PROOF OF THEOREM 3

Proof. Under additional Assumption 4, we only need to show that, with the additional matrix induced norm bound, we have a refined upper bound on the matrix covering number. The proof relies on the following lemma adapted from Bartlett et al. (2017) Lemma 3.2. Lemma 10. Let G = A ∈ R d1×d2 : A 2,1 ≤ λ . We have the following matrix covering upper bound

log N (G, , • 2 ) ≤ λ 2 2 log(2d 1 d 2 ).

The above Lemma is a direct consequence of Lemma 3.2 in Bartlett et al. (2017) with X being identity, a = λ, b = 1, and m = d 1 , d = d 2 . We apply the same trick to split the overall covering accuracy into 3 parts, 3L U,t , 3L V,t , and 3L W,t , corresponding to U, V, W respectively. Then we derive a refined bound on the covering number of F t :

EQUATION

where d = max {d x , d y , d h }. Substituting (10) into the Dudley integral as in the proof of Lemma 4 yields

R S (F M,t ) ≤ inf α>0 4α √ m + 12 m 2r √ m α log N (F t , /2, • ∞ )d .

We bound the integral as follows,

2r √ m α log N (F t , /2, • ∞ )d ≤ 2r √ m α 36 M U L 2 U,t + M V L 2 V,t + M W L 2 W,t log(2d 2 )d = 36 M U L 2 U,t + M V L 2 V,t + M W L 2 W,t log(2d 2 ) log 2r √ m α . Choosing α = 1 √ m yields R S (F M ) ≤ 4 m + 432 √ m M U L 2 U,t + M V L 2 V,t + M W L 2 W,t log(2d 2 ) log 2m √ d .

Finally, substituting the Lipschitz constant L U,t , L V,t , L W,t into the expression, we have

R S (F γ,t ) ≤ 1 γ R S (F M,t ) ≤ 4 mγ + 432 γ √ m M U L 2 U,t + M V L 2 V,t + M W L 2 W,t log(2d 2 ) log 2m √ d ≤ O   α max{M U , M V , M W }t β t -1 β-1 γ √ m log d log m √ d   .

Combining with Lemma 1 completes the proof.

Under additional Assumption 5, our proof is based on the following result from Lemma 1 in Neyshabur et al. (2017) . Lemma 11. Let f α (x) : X → R d be any predictor with parameter α, and P be any distribution on the parameter that is independent of training data. Then, for any γ, δ > 0, with probability at least 1δ over the training set of size m, for any α and any random perturbation β s.t.

P β max x∈X |f α+β (x) -f α (x)| ∞ < γ 4 ≥ 1 2 , we have R 0 (f α ) -R γ (f α ) ≤ 4 KL (α + β P) + log 6m δ m -1 ,

where KL (α + β P) is KL divergence of distributions α + β and P.

For convenience, we omit the superscript for sample index. Denote h t (α) and h t (α + β) as the hidden variables with parameters α and α + β respectively. Then we provide an upper bound of the gap of hidden layers before and after the perturbation. Denote the parameters α = vec ({W, U, V }) and the perturbation β = vec ({δW, δU, δV }).

For any t ∈ {1, 2, . . . , T }, we have

EQUATION

By Lemma 6, we have that for any t ≤ T ,

EQUATION

Combining ( 11), ( 12), and h (0) = 0, we have

EQUATION

)

Denote y t (α) and y t (α + β) as the out with parameters α and α + β respectively. Then we have

EQUATION

where (i) is from Lipschitz continuity of σ y and (ii) is from ( 12) and ( 13).

Then choosing the prior distribution and the perturbation distribution as N 0, σ 2 I , and from the concentration result for the spectral norm bounds, we have

P A∼N (0,σ 2 I d×d ) [ A 2 > ξ] ≤ 2p exp -ξ 2 2dσ 2 .

This implies with probability at least 1/2, we have max {δB U , δB W , δB V } ≤ σ 2d ln (12d).

Taking σ = γ/4ρ y (λ ht ρ h B U + ρ h B x B W ) (ρ h B U ) t -1 ρ h B U -1 + λ ht 2d ln (12d

) and combining with ( 14), with probability at least 1/2, we have

max x∈Xm y t (α + β) -y t (α) 2 ≤ (λ ht ρ h B U + ρ h B x B W ) (ρ h B U ) t -1 ρ h B U -1 + λ ht • σ 2d ln (12d) ≤ γ 4 .

Finally, we calculate the KL divergence of P and α + β with respect to this choice of σ,

KL (α + β P) ≤ α 2 2 2σ 2 = O ρ 2 y γ 2 • (λ ht ρ h B U + ρ h B x B W ) (ρ h B U ) t -1 ρ h B U -1 + λ ht 2 d ln (d) B 2 U,F + B 2 W,F + B 2 V,F = O ρ 2 y (λ ht ρ h B U + ρ h B x B W ) 2 (β t -1) 2 p ln (p) B 2 U,F + B 2 W,F + B 2 V,F γ 2 (β -1) 2 .

We complete the proof by applying Lemma 11.

C PROOFS IN SECTION 5:

C.1 PROOF OF THEOREM 4

Proof. We use the same argument from the analysis of vanilla RNNs to investigate the Lipschitz continuity of MGU RNNs. Consider h t and h t computed by different sets of weight matrices.

h t -h t 2 = (1 -r t ) h t-1 + r t h t -(1 -r t ) h t-1 -r t h t 2 ≤ (r t -r t ) h t-1 2 + (1 -r t ) (h t-1 -h t-1 ) 2 + (r t -r t ) h t 2 + r t ( h t -h t ) 2 ≤ r t -r t 2 h t-1 ∞ + 1 -r t ∞ h t-1 -h t-1 2 + r t -r t 2 h t ∞ + r t ∞ h t -h t 2

Expand the expression of h t . Note that r t is nonnegative, and

r t ∞ ≤ 1. Then we have h t ∞ ≤ 1. Additionally tanh(•) is 1-Lipschitz. Thus we get h t -h t 2 ≤ U h (h t-1 r t ) + W h x t -U h (h t-1 r t ) -W h x t 2 ≤ U h (h t-1 r t ) -U h (h t-1 r t ) 2 + B x W h -W h 2 ≤ U h -U h 2 h t-1 r t 2 + B U h r t ∞ h t-1 -h t-1 2 + B U h h t-1 ∞ r t -r t 2 + B x W h -W h 2 ≤ h t 2 U h -U h 2 + B U h r t ∞ h t-1 -h t-1 2 + B U h r t -r t 2 + B x W h -W h 2 .

We have to expand r tr t as follows,

r t -r t 2 = W r x t + U r h t-1 -W r x t -U r h t-1 2 ≤ B x W r -W r 2 + B Ur h t-1 -h t-1 2 + h t-1 2 U r -U r 2 .

We also need to bound h t 2 ,

h t 2 ≤ 1 -r t ∞ h t-1 2 + r t ∞ h t 2 ≤ 1 -r t ∞ h t-1 2 + r t ∞ (B W h B x + B U h r t ∞ h t-1 2 ) = 1 -r t ∞ + B U h r t 2 ∞ h t-1 2 + B W h B x , ≤ max j≤t 1 -r j ∞ + B U h r j 2 ∞ h t-1 2 + B W h B x .

Applying the above inequality recursively and remember

EQUATION

∞ . Put all the above ingredients together, we have

h t -h t 2 ≤ (β + 2B Ur + B Ur B U h ) h t-1 -h t-1 2 + √ d U h -U h 2 + B x W h -W h 2 + (2 + B U h ) √ d U r -U r 2 + (2B x + B U h B x ) W r -W r 2 .

Apply the above inequality recursively, denote by θ = β + 2B Ur + B Ur B U h , we have

h t -h t 2 ≤ √ d t j=1 θ j U h -U h 2 + B x t j=1 θ j W h -W h 2 + 2 √ d + B U h √ d t j=1 θ j U r -U r 2 + (2B x + B U h B x ) t j=1 θ j W r -W r 2 .

We then derive the Lipschitz continuity of y t 2 ,

y t -y t 2 ≤ ρ y B V h t -h t 2 + ρ y √ d V -V 2 ≤ ρ y B V √ d θ t -1 θ -1 U h -U h 2 + ρ y B V B x θ t -1 θ -1 W h -W h 2 + ρ y √ d V -V 2 + ρ y B V 2 √ d + B U h √ d θ t -1 θ -1 U r -U r 2 + ρ y B V (2B x + B U h B x ) θ t -1 θ -1 W r -W r 2 .

Following the same argument for proving the generalization bound of vanilla RNNs, we can get the generalization bound for MGU RNNs as

P ( z t = z t ) ≤ R γ (f t ) + O     dρ y B V min √ d, B W h B x β t -1 β-1 log d √ m θ t -1 θ-1 √ mγ + log 1 δ m     .

C.2 PROOF OF THEOREM 5:

Proof. We first bound the norm of h t as follows,

h t 2 ≤ o t ∞ tanh(c t ) 2 ≤ o t ∞ c t 2 ≤ g t ∞ c t-1 2 + r t ∞ c t 2 ≤ g t ∞ c t-1 2 + r t ∞ (B Wc B x + B Uc h t-1 2 ) ≤ g t ∞ c t-1 2 + r t ∞ (B Wc B x + B Uc o t ∞ c t-1 2 ) ≤ ( g t ∞ + r t ∞ o t ∞ B Uc ) c t-1 2 + B Wc B x .

By applying the above inequality recursively, we have

h t 2 ≤ c t 2 ≤ B Wc B x β t -1 β t -1 , where β = max j≤t { g j ∞ + r j ∞ o j ∞ B Uc }. We also have h t 2 ≤ √ d. Thus, put together, we have h t 2 ≤ min √ d, B Wc B x β t -1 β t -1 . Next, we investigate the Lipschitz continuity of h t . h t -h t 2 ≤ o t tanh(c t ) -o t tanh(c t ) 2 ≤ o t -o t 2 tanh(c t ) ∞ + o t ∞ c t -c t 2 We have to expand o t -o t , o t -o t 2 ≤ B x W o -W o 2 + B Uo h t-1 -h t-1 2 + h t-1 2 U o -U o 2 .

Note that B Uo 2 is usually small, o t and o t are close, and we have

h t-1 -h t-1 2 ≤ o t ∞ c t-1 - c t-1 2 ≤ c t-1 -c t-1 2 . Thus, we can derive o t -o t 2 ≤ B x W o -W o 2 + B Uo c t-1 -c t-1 2 + √ d U o -U o 2 .

We also expand c tc t to get,

c t -c t 2 ≤ c t-1 ∞ g t -g t 2 + r t ∞ c t-1 -c t-1 2 + c t ∞ r t -r t 2 + r t ∞ c t -c t 2 .

We also have,

c t -c t 2 ≤ B Uc h t-1 -h t-1 2 + h t-1 2 U c -U c 2 + B x W c -W c 2 , g t -g t 2 ≤ B x W g -W g 2 + B Wg h t-1 -h t-1 2 + √ d U g -U g 2 , and r t -r t 2 ≤ B x W r -W r 2 + B Wr h t-1 -h t-1 2 + √ d U r -U r 2 .

Putting together, we get

c t -c t 2 ≤ B x W c -W c 2 + W g -W g 2 + W r -W r 2 + √ d U c -U c 2 + U g -U g 2 + U r -U r 2 + g t ∞ c t-1 -c t-1 2 + r t ∞ B Uc + B Ug + B Ur h t-1 -h t-1 2 ≤ B x W c -W c 2 + W g -W g 2 + W r -W r 2 + (B Uc + B Ug + B Ur ) W o -W o 2 + √ d U c -U c 2 + U g -U g 2 + U r -U r 2 + (B Uc + B Ug + B Ur ) U o -U o 2 + o t ∞ r t ∞ B Uc + B Ug + B Ur + B Uo c t-1 -c t-1 2 .

By induction, we have

c t -c t 2 ≤ B x θ t -1 θ -1 W c -W c 2 + W g -W g 2 + W r -W r 2 + (B Uc + B Ug + B Ur ) W o -W o 2 + √ d θ t -1 θ -1 U c -U c 2 + U g -U g 2 + U r -U r 2 + (B Uc + B Ug + B Ur ) U o -U o 2 ,

where θ = β + B Ug + B Ur + B Uo . Now we immediately have

h t -h t 2 ≤ B x θ t -1 θ -1 W c -W c 2 + W g -W g 2 + W r -W r 2 + (B Uc + B Ug + B Ur ) W o -W o 2 + √ d θ t -1 θ -1 U c -U c 2 + U g -U g 2 + U r -U r 2 + (B Uc + B Ug + B Ur ) U o -U o 2 .

Then the Lipschitz continuity of y t can be written as

y t -y t 2 ≤ ρ y B V h t -h t 2 + ρ y √ d V -V 2 .

Following the same argument for proving the generalization bound of vanilla RNNs, we can get the generalization bound for LSTM RNNs as

P ( z t = z t ) ≤ R γ (f t ) + O     dρ y B V min √ d, B Wc B x β t -1 β-1 log d √ m θ t -1 θ-1 √ mγ + log 1 δ m     .

D EXTENSION TO CONVOLUTIONAL RNNS:

We further extend our analysis to Convolutional RNNs (Conv RNNs). Conv RNNs integrate convolutional filters and recurrent neural networks. Specifically, we consider input x ∈ R d and k-channel k-dimensional convolutional filters I 1 , . . . , I k ∈ R k followed by an average pooling layer over the k channels for reducing dimensionality. Extensions to convolution with strides and other kinds of average pooling layers (e.g., blockwise pooling) are straightforward.

I > 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 3 Y D 6 4 S q U x T n b m U y c u z h n N E 9 6 A = " > A A A B / X i c b V D L S g M x F M 3 U V 6 2 v 8 b F z E 2 w F V 2 W m C L o s u N F d B f u A z j h k 0 k w b m k m G J C P U o f g r b l w o 4 t b / c O f f m G l n o a 0 H A o d z 7 u W e n D B h V G n H + b Z K K 6 t r 6 x v l z c r W 9 s 7 u n r 1 / 0 F E i l Z i 0 s W B C 9 k K k C K O c t D X V j P Q S S V A c M t I N x 1 e 5 3 3 0 g U l H B 7 / Q k I X 6 M h p x G F C N t p M A + q n k x 0 i O M W H Y z D d x 7 T 4 u k F t h V p + 7 M A J e J W 5 A q K N A K 7 C 9 v I H A a E 6 4 x Q 0 r 1 X S f R f o a k p p i R a c V L F U k Q H q M h 6 R v K U U y U n 8 3 S T + G p U Q Y w E t I 8 r u F M / b 2 R o V i p S R y a y T y q W v R y 8 T + v n + r o 0 s 8 o T 1 J N O J 4 f i l I G t Y B 5 F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j F l O A u f n m Z d B p 1 1 6 m 7 t 4 1 q 8 7 y o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 H G U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 3 Y D 6 4 S q U x T n b m U y c u z h n N E 9 6 A = " > A A A B / X i c b V D L S g M x F M 3 U V 6 2 v 8 b F z E 2 w F V 2 W m C L o s u N F d B f u A z j h k 0 k w b m k m G J C P U o f g r b l w o 4 t b / c O f f m G l n o a 0 H A o d z 7 u W e n D B h V G n H + b Z K K 6 t r 6 x v l z c r W 9 s 7 u n r 1 / 0 F E i l Z i 0 s W B C 9 k K k C K O c t D X V j P Q S S V A c M t I N x 1 e 5 3 3 0 g U l H B 7 / Q k I X 6 M h p x G F C N t p M A + q n k x 0 i O M W H Y z D d x 7 T 4 u k F t h V p + 7 M A J e J W 5 A q K N A K 7 C 9 v I H A a E 6 4 x Q 0 r 1 X S f R f o a k p p i R a c V L F U k Q H q M h 6 R v K U U y U n 8 3 S T + G p U Q Y w E t I 8 r u F M / b 2 R o V i p S R y a y T y q W v R y 8 T + v n + r o 0 s 8 o T 1 J N O J 4 f i l I G t Y B 5 F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j F l O A u f n m Z d B p 1 1 6 m 7 t 4 1 q 8 7 y o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 H G U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 3 Y D 6 4 S q U x T n b m U y c u z h n N E 9 6 A = " > A A A B / X i c b V D L S g M x F M 3 U V 6 2 v 8 b F z E 2 w F V 2 W m C L o s u N F d B f u A z j h k 0 k w b m k m G J C P U o f g r b l w o 4 t b / c O f f m G l n o a 0 H A o d z 7 u W e n D B h V G n H + b Z K K 6 t r 6 x v l z c r W 9 s 7 u n r 1 / 0 F E i l Z i 0 s W B C 9 k K k C K O c t D X V j P Q S S V A c M t I N x 1 e 5 3 3 0 g U l H B 7 / Q k I X 6 M h p x G F C N t p M A + q n k x 0 i O M W H Y z D d x 7 T 4 u k F t h V p + 7 M A J e J W 5 A q K N A K 7 C 9 v I H A a E 6 4 x Q 0 r 1 X S f R f o a k p p i R a c V L F U k Q H q M h 6 R v K U U y U n 8 3 S T + G p U Q Y w E t I 8 r u F M / b 2 R o V i p S R y a y T y q W v R y 8 T + v n + r o 0 s 8 o T 1 J N O J 4 f i l I G t Y B 5 F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j F l O A u f n m Z d B p 1 1 6 m 7 t 4 1 q 8 7 y o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 H G U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 3 Y D 6 4 S q U x T n b m U y c u z h n N E 9 6 A = " > A A A B / X i c b V D L S g M x F M 3 U V 6 2 v 8 b F z E 2 w F V 2 W m C L o s u N F d B f u A z j h k 0 k w b m k m G J C P U o f g r b l w o 4 t b / c O f f m G l n o a 0 H A o d z 7 u W e n D B h V G n H + b Z K K 6 t r 6 x v l z c r W 9 s 7 u n r 1 / 0 F E i l Z i 0 s W B C 9 k K k C K O c t D X V j P Q S S V A c M t I N x 1 e 5 3 3 0 g U l H B 7 / Q k I X 6 M h p x G F C N t p M A + q n k x 0 i O M W H Y z D d x 7 T 4 u k F t h V p + 7 M A J e J W 5 A q K N A K 7 C 9 v I H A a E 6 4 x Q 0 r 1 X S f R f o a k p p i R a c V L F U k Q H q M h 6 R v K U U y U n 8 3 S T + G p U Q Y w E t I 8 r u F M / b 2 R o V i p S R y a y T y q W v R y 8 T + v n + r o 0 s 8 o T 1 J N O J 4 f i l I G t Y B 5 F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j F l O A u f n m Z d B p 1 1 6 m 7 t 4 1 q 8 7 y o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 H G U v w = = < / l a t e x i t > I > 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 3 Y D 6 4 S q U x T n b m U y c u z h n N E 9 6 A = " > A A A B / X i c b V D L S g M x F M 3 U V 6 2 v 8 b F z E 2 w F V 2 W m C L o s u N F d B f u A z j h k 0 k w b m k m G J C P U o f g r b l w o 4 t b / c O f f m G l n o a 0 H A o d z 7 u W e n D B h V G n H + b Z K K 6 t r 6 x v l z c r W 9 s 7 u n r 1 / 0 F E i l Z i 0 s W B C 9 k K k C K O c t D X V j P Q S S V A c M t I N x 1 e 5 3 3 0 g U l H B 7 / Q k I X 6 M h p x G F C N t p M A + q n k x 0 i O M W H Y z D d x 7 T 4 u k F t h V p + 7 M A J e J W 5 A q K N A K 7 C 9 v I H A a E 6 4 x Q 0 r 1 X S f R f o a k p p i R a c V L F U k Q H q M h 6 R v K U U y U n 8 3 S T + G p U Q Y w E t I 8 r u F M / b 2 R o V i p S R y a y T y q W v R y 8 T + v n + r o 0 s 8 o T 1 J N O J 4 f i l I G t Y B 5 F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j F l O A u f n m Z d B p 1 1 6 m 7 t 4 1 q 8 7 y o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 H G U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 3 Y D 6 4 S q U x T n b m U y c u z h n N E 9 6 A = " > A A A B / X i c b V D L S g M x F M 3 U V 6 2 v 8 b F z E 2 w F V 2 W m C L o s u N F d B f u A z j h k 0 k w b m k m G J C P U o f g r b l w o 4 t b / c O f f m G l n o a 0 H A o d z 7 u W e n D B h V G n H + b Z K K 6 t r 6 x v l z c r W 9 s 7 u n r 1 / 0 F E i l Z i 0 s W B C 9 k K k C K O c t D X V j P Q S S V A c M t I N x 1 e 5 3 3 0 g U l H B 7 / Q k I X 6 M h p x G F C N t p M A + q n k x 0 i O M W H Y z D d x 7 T 4 u k F t h V p + 7 M A J e J W 5 A q K N A K 7 C 9 v I H A a E 6 4 x Q 0 r 1 X S f R f o a k p p i R a c V L F U k Q H q M h 6 R v K U U y U n 8 3 S T + G p U Q Y w E t I 8 r u F M / b 2 R o V i p S R y a y T y q W v R y 8 T + v n + r o 0 s 8 o T 1 J N O J 4 f i l I G t Y B 5 F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j F l O A u f n m Z d B p 1 1 6 m 7 t 4 1 q 8 7 y o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 H G U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 3 Y D 6 4 S q U x T n b m U y c u z h n N E 9 6 A = " > A A A B / X i c b V D L S g M x F M 3 U V 6 2 v 8 b F z E 2 w F V 2 W m C L o s u N F d B f u A z j h k 0 k w b m k m G J C P U o f g r b l w o 4 t b / c O f f m G l n o a 0 H A o d z 7 u W e n D B h V G n H + b Z K K 6 t r 6 x v l z c r W 9 s 7 u n r 1 / 0 F E i l Z i 0 s W B C 9 k K k C K O c t D X V j P Q S S V A c M t I N x 1 e 5 3 3 0 g U l H B 7 / Q k I X 6 M h p x G F C N t p M A + q n k x 0 i O M W H Y z D d x 7 T 4 u k F t h V p + 7 M A J e J W 5 A q K N A K 7 C 9 v I H A a E 6 4 x Q 0 r 1 X S f R f o a k p p i R a c V L F U k Q H q M h 6 R v K U U y U n 8 3 S T + G p U Q Y w E t I 8 r u F M / b 2 R o V i p S R y a y T y q W v R y 8 T + v n + r o 0 s 8 o T 1 J N O J 4 f i l I G t Y B 5 F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j F l O A u f n m Z d B p 1 1 6 m 7 t 4 1 q 8 7 y o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 H G U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 3 Y D 6 4 S q U x T n b m U y c u z h n N E 9 6 A = " > A A A B / X i c b V D L S g M x F M 3 U V 6 2 v 8 b F z E 2 w F V 2 W m C L o s u N F d B f u A z j h k 0 k w b m k m G J C P U o f g r b l w o 4 t b / c O f f m G l n o a 0 H A o d z 7 u W e n D B h V G n H + b Z K K 6 t r 6 x v l z c r W 9 s 7 u n r 1 / 0 F E i l Z i 0 s W B C 9 k K k C K O c t D X V j P Q S S V A c M t I N x 1 e 5 3 3 0 g U l H B 7 / Q k I X 6 M h p x G F C N t p M A + q n k x 0 i O M W H Y z D d x 7 T 4 u k F t h V p + 7 M A J e J W 5 A q K N A K 7 C 9 v I H A a E 6 4 x Q 0 r 1 X S f R f o a k p p i R a c V L F U k Q H q M h 6 R v K U U y U n 8 3 S T + G p U Q Y w E t I 8 r u F M / b 2 R o V i p S R y a y T y q W v R y 8 T + v n + r o 0 s 8 o T 1 J N O J 4 f i l I G t Y B 5 F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j F l O A u f n m Z d B p 1 1 6 m 7 t 4 1 q 8 7 y o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 H G U v w = = < / l a t e x i t > I > 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 3 Y D 6 4 S q U x T n b m U y c u z h n N E 9 6 A = " > A A A B / X i c b V D L S g M x F M 3 U V 6 2 v 8 b F z E 2 w F V 2 W m C L o s u N F d B f u A z j h k 0 k w b m k m G J C P U o f g r b l w o 4 t b / c O f f m G l n o a 0 H A o d z 7 u W e n D B h V G n H + b Z K K 6 t r 6 x v l z c r W 9 s 7 u n r 1 / 0 F E i l Z i 0 s W B C 9 k K k C K O c t D X V j P Q S S V A c M t I N x 1 e 5 3 3 0 g U l H B 7 / Q k I X 6 M h p x G F C N t p M A + q n k x 0 i O M W H Y z D d x 7 T 4 u k F t h V p + 7 M A J e J W 5 A q K N A K 7 C 9 v I H A a E 6 4 x Q 0 r 1 X S f R f o a k p p i R a c V L F U k Q H q M h 6 R v K U U y U n 8 3 S T + G p U Q Y w E t I 8 r u F M / b 2 R o V i p S R y a y T y q W v R y 8 T + v n + r o 0 s 8 o T 1 J N O J 4 f i l I G t Y B 5 F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j F l O A u f n m Z d B p 1 1 6 m 7 t 4 1 q 8 7 y o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 H G U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 3 Y D 6 4 S q U x T n b m U y c u z h n N E 9 6 A = " > A A A B / X i c b V D L S g M x F M 3 U V 6 2 v 8 b F z E 2 w F V 2 W m C L o s u N F d B f u A z j h k 0 k w b m k m G J C P U o f g r b l w o 4 t b / c O f f m G l n o a 0 H A o d z 7 u W e n D B h V G n H + b Z K K 6 t r 6 x v l z c r W 9 s 7 u n r 1 / 0 F E i l Z i 0 s W B C 9 k K k C K O c t D X V j P Q S S V A c M t I N x 1 e 5 3 3 0 g U l H B 7 / Q k I X 6 M h p x G F C N t p M A + q n k x 0 i O M W H Y z D d x 7 T 4 u k F t h V p + 7 M A J e J W 5 A q K N A K 7 C 9 v I H A a E 6 4 x Q 0 r 1 X S f R f o a k p p i R a c V L F U k Q H q M h 6 R v K U U y U n 8 3 S T + G p U Q Y w E t I 8 r u F M / b 2 R o V i p S R y a y T y q W v R y 8 T + v n + r o 0 s 8 o T 1 J N O J 4 f i l I G t Y B 5 F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j F l O A u f n m Z d B p 1 1 6 m 7 t 4 1 q 8 7 y o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 H G U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 3 Y D 6 4 S q U x T n b m U y c u z h n N E 9 6 A = " > A A A B / X i c b V D L S g M x F M 3 U V 6 2 v 8 b F z E 2 w F V 2 W m C L o s u N F d B f u A z j h k 0 k w b m k m G J C P U o f g r b l w o 4 t b / c O f f m G l n o a 0 H A o d z 7 u W e n D B h V G n H + b Z K K 6 t r 6 x v l z c r W 9 s 7 u n r 1 / 0 F E i l Z i 0 s W B C 9 k K k C K O c t D X V j P Q S S V A c M t I N x 1 e 5 3 3 0 g U l H B 7 / Q k I X 6 M h p x G F C N t p M A + q n k x 0 i O M W H Y z D d x 7 T 4 u k F t h V p + 7 M A J e J W 5 A q K N A K 7 C 9 v I H A a E 6 4 x Q 0 r 1 X S f R f o a k p p i R a c V L F U k Q H q M h 6 R v K U U y U n 8 3 S T + G p U Q Y w E t I 8 r u F M / b 2 R o V i p S R y a y T y q W v R y 8 T + v n + r o 0 s 8 o T 1 J N O J 4 f i l I G t Y B 5 F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j F l O A u f n m Z d B p 1 1 6 m 7 t 4 1 q 8 7 y o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 H G U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 3 Y D 6 4 S q U x T n b m U y c u z h n N E 9 6 A = " > A A A B / X i c b V D L S g M x F M 3 U V 6 2 v 8 b F z E 2 w F V 2 W m C L o s u N F d B f u A z j h k 0 k w b m k m G J C P U o f g r b l w o 4 t b / c O f f m G l n o a 0 H A o d z 7 u W e n D B h V G n H + b Z K K 6 t r 6 x v l z c r W 9 s 7 u n r 1 / 0 F E i l Z i 0 s W B C 9 k K k C K O c t D X V j P Q S S V A c M t I N x 1 e 5 3 3 0 g U l H B 7 / Q k I X 6 M h p x G F C N t p M A + q n k x 0 i O M W H Y z D d x 7 T 4 u k F t h V p + 7 M A J e J W 5 A q K N A K 7 C 9 v I H A a E 6 4 x Q 0 r 1 X S f R f o a k p p i R a c V L F U k Q H q M h 6 R v K U U y U n 8 3 S T + G p U Q Y w E t I 8 r u F M / b 2 R o V i p S R y a y T y q W v R y 8 T + v n + r o 0 s 8 o T 1 J N O J 4 f i l I G t Y B 5 F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j F l O A u f n m Z d B p 1 1 6 m 7 t 4 1 q 8 7 y o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 H G U v w = = < / l a t e x i t > I > 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 3 Y D 6 4 S q U x T n b m U y c u z h n N E 9 6 A = " > A A A B / X i c b V D L S g M x F M 3 U V 6 2 v 8 b F z E 2 w F V 2 W m C L o s u N F d B f u A z j h k 0 k w b m k m G J C P U o f g r b l w o 4 t b / c O f f m G l n o a 0 H A o d z 7 u W e n D B h V G n H + b Z K K 6 t r 6 x v l z c r W 9 s 7 u n r 1 / 0 F E i l Z i 0 s W B C 9 k K k C K O c t D X V j P Q S S V A c M t I N x 1 e 5 3 3 0 g U l H B 7 / Q k I X 6 M h p x G F C N t p M A + q n k x 0 i O M W H Y z D d x 7 T 4 u k F t h V p + 7 M A J e J W 5 A q K N A K 7 C 9 v I H A a E 6 4 x Q 0 r 1 X S f R f o a k p p i R a c V L F U k Q H q M h 6 R v K U U y U n 8 3 S T + G p U Q Y w E t I 8 r u F M / b 2 R o V i p S R y a y T y q W v R y 8 T + v n + r o 0 s 8 o T 1 J N O J 4 f i l I G t Y B 5 F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j F l O A u f n m Z d B p 1 1 6 m 7 t 4 1 q 8 7 y o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 H G U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 3 Y D 6 4 S q U x T n b m U y c u z h n N E 9 6 A = " > A A A B / X i c b V D L S g M x F M 3 U V 6 2 v 8 b F z E 2 w F V 2 W m C L o s u N F d B f u A z j h k 0 k w b m k m G J C P U o f g r b l w o 4 t b / c O f f m G l n o a 0 H A o d z 7 u W e n D B h V G n H + b Z K K 6 t r 6 x v l z c r W 9 s 7 u n r 1 / 0 F E i l Z i 0 s W B C 9 k K k C K O c t D X V j P Q S S V A c M t I N x 1 e 5 3 3 0 g U l H B 7 / Q k I X 6 M h p x G F C N t p M A + q n k x 0 i O M W H Y z D d x 7 T 4 u k F t h V p + 7 M A J e J W 5 A q K N A K 7 C 9 v I H A a E 6 4 x Q 0 r 1 X S f R f o a k p p i R a c V L F U k Q H q M h 6 R v K U U y U n 8 3 S T + G p U Q Y w E t I 8 r u F M / b 2 R o V i p S R y a y T y q W v R y 8 T + v n + r o 0 s 8 o T 1 J N O J 4 f i l I G t Y B 5 F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j F l O A u f n m Z d B p 1 1 6 m 7 t 4 1 q 8 7 y o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 H G U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 3 Y D 6 4 S q U x T n b m U y c u z h n N E 9 6 A = " > A A A B / X i c b V D L S g M x F M 3 U V 6 2 v 8 b F z E 2 w F V 2 W m C L o s u N F d B f u A z j h k 0 k w b m k m G J C P U o f g r b l w o 4 t b / c O f f m G l n o a 0 H A o d z 7 u W e n D B h V G n H + b Z K K 6 t r 6 x v l z c r W 9 s 7 u n r 1 / 0 F E i l Z i 0 s W B C 9 k K k C K O c t D X V j P Q S S V A c M t I N x 1 e 5 3 3 0 g U l H B 7 / Q k I X 6 M h p x G F C N t p M A + q n k x 0 i O M W H Y z D d x 7 T 4 u k F t h V p + 7 M A J e J W 5 A q K N A K 7 C 9 v I H A a E 6 4 x Q 0 r 1 X S f R f o a k p p i R a c V L F U k Q H q M h 6 R v K U U y U n 8 3 S T + G p U Q Y w E t I 8 r u F M / b 2 R o V i p S R y a y T y q W v R y 8 T + v n + r o 0 s 8 o T 1 J N O J 4 f i l I G t Y B 5 F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j F l O A u f n m Z d B p 1 1 6 m 7 t 4 1 q 8 7 y o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 H G U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 9 3 Y D 6 4 S q U x T n b m U y c u z h n N E 9 6 A = " > A A A B / X i c b V D L S g M x F M 3 U V 6 2 v 8 b F z E 2 w F V 2 W m C L o s u N F d B f u A z j h k 0 k w b m k m G J C P U o f g r b l w o 4 t b / c O f f m G l n o a 0 H A o d z 7 u W e n D B h V G n H + b Z K K 6 t r 6 x v l z c r W 9 s 7 u n r 1 / 0 F E i l Z i 0 s W B C 9 k K k C K O c t D X V j P Q S S V A c M t I N x 1 e 5 3 3 0 g U l H B 7 / Q k I X 6 M h p x G F C N t p M A + q n k x 0 i O M W H Y z D d x 7 T 4 u k F t h V p + 7 M A J e J W 5 A q K N A K 7 C 9 v I H A a E 6 4 x Q 0 r 1 X S f R f o a k p p i R a c V L F U k Q H q M h 6 R v K U U y U n 8 3 S T + G p U Q Y w E t I 8 r u F M / b 2 R o V i p S R y a y T y q W v R y 8 T + v n + r o 0 s 8 o T 1 J N O J 4 f i l I G t Y B 5 F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j F l O A u f n m Z d B p 1 1 6 m 7 t 4 1 q 8 7 y o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 H G U v w = = < / l a t e x i t > I >

3 < l a t e x i t s h a 1 _ b a s e 6 4 = " e f r s S X s P y t n

S P S O B B k G X X 4 g K h u I = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 W U A c Z K L L A V i T 6 k J k S O 6 7 R W H T u y H a Q S V f w K C w M I s f I f b P w N T p s B W o 5 k 6 e i c e 3 W P T 5 g w q r T j f F u l l d W 1 9 Y 3 y Z m V r e 2 d 3 z 9 4 / 6 C i R S k z a W D A h e y F S h F F O 2 p p q R n q J J C g O G e m G 4 6 v c 7 z 4 Q q a j g d 3 q S E D 9 G Q 0 4 j i p E 2 U m A f 1 b w Y 6 R F G L L u Z B u f 3 n h Z J L b C r T t 2 Z A S 4 T t y B V U K A V 2 F / e Q O A

0 J l x j h p T q u 0 6 i / Q x J T T E j 0 4 q X K p I g P E Z D 0 j e U o 5 g o P 5 u l n 8 J T o w x g J K R 5 X M O Z + n s j Q 7 F S k z g 0 k 3 l U t e j l 4 n 9 e P 9 X R p Z 9 R n q S a c D w

/ F K U M a g H z K u C A S o I 1 m x i C s K Q m K 8 Q j J B H W p r C K K c F d / P I y 6 T T q r l N 3 b x v V Z q O o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 u u U v w = = < /

l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e f r s S X s P y t n

S P S O B B k G X X 4 g K h u I = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 W U A c Z K L L A V i T 6 k J k S O 6 7 R W H T u y H a Q S V f w K C w M I s f I f b P w N T p s B W o 5 k 6 e i c e 3 W P T 5 g w q r T j f F u l l d W 1 9 Y 3 y Z m V r e 2 d 3 z 9 4 / 6 C i R S k z a W D A h e y F S h F F O 2 p p q R n q J J C g O G e m G 4 6 v c 7 z 4 Q q a j g d 3 q S E D 9 G Q 0 4 j i p E 2 U m A f 1 b w Y 6 R F G L L u Z B u f 3 n h Z J L b C r T t 2 Z A S 4 T t y B V U K A V 2 F / e Q O A

0 J l x j h p T q u 0 6 i / Q x J T T E j 0 4 q X K p I g P E Z D 0 j e U o 5 g o P 5 u l n 8 J T o w x g J K R 5 X M O Z + n s j Q 7 F S k z g 0 k 3 l U t e j l 4 n 9 e P 9 X R p Z 9 R n q S a c D w

/ F K U M a g H z K u C A S o I 1 m x i C s K Q m K 8 Q j J B H W p r C K K c F d / P I y 6 T T q r l N 3 b x v V Z q O o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 u u U v w = = < /

l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e f r s S X s P y t n

S P S O B B k G X X 4 g K h u I = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 W U A c Z K L L A V i T 6 k J k S O 6 7 R W H T u y H a Q S V f w K C w M I s f I f b P w N T p s B W o 5 k 6 e i c e 3 W P T 5 g w q r T j f F u l l d W 1 9 Y 3 y Z m V r e 2 d 3 z 9 4 / 6 C i R S k z a W D A h e y F S h F F O 2 p p q R n q J J C g O G e m G 4 6 v c 7 z 4 Q q a j g d 3 q S E D 9 G Q 0 4 j i p E 2 U m A f 1 b w Y 6 R F G L L u Z B u f 3 n h Z J L b C r T t 2 Z A S 4 T t y B V U K A V 2 F / e Q O A

0 J l x j h p T q u 0 6 i / Q x J T T E j 0 4 q X K p I g P E Z D 0 j e U o 5 g o P 5 u l n 8 J T o w x g J K R 5 X M O Z + n s j Q 7 F S k z g 0 k 3 l U t e j l 4 n 9 e P 9 X R p Z 9 R n q S a c D w

/ F K U M a g H z K u C A S o I 1 m x i C s K Q m K 8 Q j J B H W p r C K K c F d / P I y 6 T T q r l N 3 b x v V Z q O o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 u u U v w = = < /

l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e f r s S X s P y t n

S P S O B B k G X X 4 g K h u I = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 W U A c Z K L L A V i T 6 k J k S O 6 7 R W H T u y H a Q S V f w K C w M I s f I f b P w N T p s B W o 5 k 6 e i c e 3 W P T 5 g w q r T j f F u l l d W 1 9 Y 3 y Z m V r e 2 d 3 z 9 4 / 6 C i R S k z a W D A h e y F S h F F O 2 p p q R n q J J C g O G e m G 4 6 v c 7 z 4 Q q a j g d 3 q S E D 9 G Q 0 4 j i p E 2 U m A f 1 b w Y 6 R F G L L u Z B u f 3 n h Z J L b C r T t 2 Z A S 4 T t y B V U K A V 2 F / e Q O A

0 J l x j h p T q u 0 6 i / Q x J T T E j 0 4 q X K p I g P E Z D 0 j e U o 5 g o P 5 u l n 8 J T o w x g J K R 5 X M O Z + n s j Q 7 F S k z g 0 k 3 l U t e j l 4 n 9 e P 9 X R p Z 9 R n q S a c D w

/ F K U M a g H z K u C A S o I 1 m x i C s K Q m K 8 Q j J B H W p r C K K c F d / P I y 6 T T q r l N 3 b x v V Z q O o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 u u U v w = = < / l a t e x i t > I > 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 c G S s e x z o I g E 8 k z y x r h n z z W 6 l Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V k g b E S C 2 x F o i 1 S E y L H d V q r j h 3 Z D l K J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H J 0 o Z V d p x v q 3 K y u r a + k Z 1 s 7 a 1 v b O 7 Z + 8 f d J X I J C Y d L J i Q d x F S h F F O O p p q R u 5 S S V A S M d K L x p e F 3 3 s g U l H B b / U k J U G C h p z G F C N t p N A + a v g J 0 i O M W H 4 9 D b 1 7 X 4 u 0 E d p 1 p + n M A J e J W 5 I 6 K N E O 7 S 9 / I H C W E K 4 x Q 0 r 1 X S f V Q Y 6 k p p i R a c 3 P F E k R H q M h 6 R v K U U J U k M / S T + G p U Q Y w F t I 8 r u F M / b 2 R o 0 S p S R K Z y S K q W v Q K 8 T + v n + n 4 I s g p T z N N O J 4 f i j M G t Y B F F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j N l O A u f n m Z d L 2 m 6 z T d G 6 / e 8 s o 6 q u A Y n I A z 4 I J z 0 A J X o A 0 6 A I N H 8 A x e w Z v 1 Z L 1 Y 7 9 b H f L R i l T u H 4 A + s z x / R Y Z S + < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 c G S s e x z o I g E 8 k z y x r h n z z W 6 l Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V k g b E S C 2 x F o i 1 S E y L H d V q r j h 3 Z D l K J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H J 0 o Z V d p x v q 3 K y u r a + k Z 1 s 7 a 1 v b O 7 Z + 8 f d J X I J C Y d L J i Q d x F S h F F O O p p q R u 5 S S V A S M d K L x p e F 3 3 s g U l H B b / U k J U G C h p z G F C N t p N A + a v g J 0 i O M W H 4 9 D b 1 7 X 4 u 0 E d p 1 p + n M A J e J W 5 I 6 K N E O 7 S 9 / I H C W E K 4 x Q 0 r 1 X S f V Q Y 6 k p p i R a c 3 P F E k R H q M h 6 R v K U U J U k M / S T + G p U Q Y w F t I 8 r u F M / b 2 R o 0 S p S R K Z y S K q W v Q K 8 T + v n + n 4 I s g p T z N N O J 4 f i j M G t Y B F F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j N l O A u f n m Z d L 2 m 6 z T d G 6 / e 8 s o 6 q u A Y n I A z 4 I J z 0 A J X o A 0 6 A I N H 8 A x e w Z v 1 Z L 1 Y 7 9 b H f L R i l T u H 4 A + s z x / R Y Z S + < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 c G S s e x z o I g E 8 k z y x r h n z z W 6 l Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V k g b E S C 2 x F o i 1 S E y L H d V q r j h 3 Z D l K J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H J 0 o Z V d p x v q 3 K y u r a + k Z 1 s 7 a 1 v b O 7 Z + 8 f d J X I J C Y d L J i Q d x F S h F F O O p p q R u 5 S S V A S M d K L x p e F 3 3 s g U l H B b / U k J U G C h p z G F C N t p N A + a v g J 0 i O M W H 4 9 D b 1 7 X 4 u 0 E d p 1 p + n M A J e J W 5 I 6 K N E O 7 S 9 / I H C W E K 4 x Q 0 r 1 X S f V Q Y 6 k p p i R a c 3 P F E k R H q M h 6 R v K U U J U k M / S T + G p U Q Y w F t I 8 r u F M / b 2 R o 0 S p S R K Z y S K q W v Q K 8 T + v n + n 4 I s g p T z N N O J 4 f i j M G t Y B F F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j N l O A u f n m Z d L 2 m 6 z T d G 6 / e 8 s o 6 q u A Y n I A z 4 I J z 0 A J X o A 0 6 A I N H 8 A x e w Z v 1 Z L 1 Y 7 9 b H f L R i l T u H 4 A + s z x / R Y Z S + < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 c G S s e x z o I g E 8 k z y x r h n z z W 6 l Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V k g b E S C 2 x F o i 1 S E y L H d V q r j h 3 Z D l K J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H J 0 o Z V d p x v q 3 K y u r a + k Z 1 s 7 a 1 v b O 7 Z + 8 f d J X I J C Y d L J i Q d x F S h F F O O p p q R u 5 S S V A S M d K L x p e F 3 3 s g U l H B b / U k J U G C h p z G F C N t p N A + a v g J 0 i O M W H 4 9 D b 1 7 X 4 u 0 E d p 1 p + n M A J e J W 5 I 6 K N E O 7 S 9 / I H C W E K 4 x Q 0 r 1 X S f V Q Y 6 k p p i R a c 3 P F E k R H q M h 6 R v K U U J U k M / S T + G p U Q Y w F t I 8 r u F M / b 2 R o 0 S p S R K Z y S K q W v Q K 8 T + v n + n 4 I s g p T z N N O J 4 f i j M G t Y B F F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j N l O A u f n m Z d L 2 m 6 z T d G 6 / e 8 s o 6 q u A Y n I A z 4 I J z 0 A J X o A 0 6 A I N H 8 A x e w Z v 1 Z L 1 Y 7 9 b H f L R i l T u H 4 A + s z x / R Y Z S + < / l a t e x i t > I > 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 c G S s e x z o I g E 8 k z y x r h n z z W 6 l Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V k g b E S C 2 x F o i 1 S E y L H d V q r j h 3 Z D l K J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H J 0 o Z V d p x v q 3 K y u r a + k Z 1 s 7 a 1 v b O 7 Z + 8 f d J X I J C Y d L J i Q d x F S h F F O O p p q R u 5 S S V A S M d K L x p e F 3 3 s g U l H B b / U k J U G C h p z G F C N t p N A + a v g J 0 i O M W H 4 9 D b 1 7 X 4 u 0 E d p 1 p + n M A J e J W 5 I 6 K N E O 7 S 9 / I H C W E K 4 x Q 0 r 1 X S f V Q Y 6 k p p i R a c 3 P F E k R H q M h 6 R v K U U J U k M / S T + G p U Q Y w F t I 8 r u F M / b 2 R o 0 S p S R K Z y S K q W v Q K 8 T + v n + n 4 I s g p T z N N O J 4 f i j M G t Y B F F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j N l O A u f n m Z d L 2 m 6 z T d G 6 / e 8 s o 6 q u A Y n I A z 4 I J z 0 A J X o A 0 6 A I N H 8 A x e w Z v 1 Z L 1 Y 7 9 b H f L R i l T u H 4 A + s z x / R Y Z S + < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 c G S s e x z o I g E 8 k z y x r h n z z W 6 l Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V k g b E S C 2 x F o i 1 S E y L H d V q r j h 3 Z D l K J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H J 0 o Z V d p x v q 3 K y u r a + k Z 1 s 7 a 1 v b O 7 Z + 8 f d J X I J C Y d L J i Q d x F S h F F O O p p q R u 5 S S V A S M d K L x p e F 3 3 s g U l H B b / U k J U G C h p z G F C N t p N A + a v g J 0 i O M W H 4 9 D b 1 7 X 4 u 0 E d p 1 p + n M A J e J W 5 I 6 K N E O 7 S 9 / I H C W E K 4 x Q 0 r 1 X S f V Q Y 6 k p p i R a c 3 P F E k R H q M h 6 R v K U U J U k M / S T + G p U Q Y w F t I 8 r u F M / b 2 R o 0 S p S R K Z y S K q W v Q K 8 T + v n + n 4 I s g p T z N N O J 4 f i j M G t Y B F F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j N l O A u f n m Z d L 2 m 6 z T d G 6 / e 8 s o 6 q u A Y n I A z 4 I J z 0 A J X o A 0 6 A I N H 8 A x e w Z v 1 Z L 1 Y 7 9 b H f L R i l T u H 4 A + s z x / R Y Z S + < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 c G S s e x z o I g E 8 k z y x r h n z z W 6 l Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V k g b E S C 2 x F o i 1 S E y L H d V q r j h 3 Z D l K J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H J 0 o Z V d p x v q 3 K y u r a + k Z 1 s 7 a 1 v b O 7 Z + 8 f d J X I J C Y d L J i Q d x F S h F F O O p p q R u 5 S S V A S M d K L x p e F 3 3 s g U l H B b / U k J U G C h p z G F C N t p N A + a v g J 0 i O M W H 4 9 D b 1 7 X 4 u 0 E d p 1 p + n M A J e J W 5 I 6 K N E O 7 S 9 / I H C W E K 4 x Q 0 r 1 X S f V Q Y 6 k p p i R a c 3 P F E k R H q M h 6 R v K U U J U k M / S T + G p U Q Y w F t I 8 r u F M / b 2 R o 0 S p S R K Z y S K q W v Q K 8 T + v n + n 4 I s g p T z N N O J 4 f i j M G t Y B F F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j N l O A u f n m Z d L 2 m 6 z T d G 6 / e 8 s o 6 q u A Y n I A z 4 I J z 0 A J X o A 0 6 A I N H 8 A x e w Z v 1 Z L 1 Y 7 9 b H f L R i l T u H 4 A + s z x / R Y Z S + < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 c G S s e x z o I g E 8 k z y x r h n z z W 6 l Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V k g b E S C 2 x F o i 1 S E y L H d V q r j h 3 Z D l K J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H J 0 o Z V d p x v q 3 K y u r a + k Z 1 s 7 a 1 v b O 7 Z + 8 f d J X I J C Y d L J i Q d x F S h F F O O p p q R u 5 S S V A S M d K L x p e F 3 3 s g U l H B b / U k J U G C h p z G F C N t p N A + a v g J 0 i O M W H 4 9 D b 1 7 X 4 u 0 E d p 1 p + n M A J e J W 5 I 6 K N E O 7 S 9 / I H C W E K 4 x Q 0 r 1 X S f V Q Y 6 k p p i R a c 3 P F E k R H q M h 6 R v K U U J U k M / S T + G p U Q Y w F t I 8 r u F M / b 2 R o 0 S p S R K Z y S K q W v Q K 8 T + v n + n 4 I s g p T z N N O J 4 f i j M G t Y B F F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j N l O A u f n m Z d L 2 m 6 z T d G 6 / e 8 s o 6 q u A Y n I A z 4 I J z 0 A J X o A 0 6 A I N H 8 A x e w Z v 1 Z L 1 Y 7 9 b H f L R i l T u H 4 A + s z x / R Y Z S + < / l a t e x i t > I > 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 c G S s e x z o I g E 8 k z y x r h n z z W 6 l Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V k g b E S C 2 x F o i 1 S E y L H d V q r j h 3 Z D l K J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H J 0 o Z V d p x v q 3 K y u r a + k Z 1 s 7 a 1 v b O 7 Z + 8 f d J X I J C Y d L J i Q d x F S h F F O O p p q R u 5 S S V A S M d K L x p e F 3 3 s g U l H B b / U k J U G C h p z G F C N t p N A + a v g J 0 i O M W H 4 9 D b 1 7 X 4 u 0 E d p 1 p + n M A J e J W 5 I 6 K N E O 7 S 9 / I H C W E K 4 x Q 0 r 1 X S f V Q Y 6 k p p i R a c 3 P F E k R H q M h 6 R v K U U J U k M / S T + G p U Q Y w F t I 8 r u F M / b 2 R o 0 S p S R K Z y S K q W v Q K 8 T + v n + n 4 I s g p T z N N O J 4 f i j M G t Y B F F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j N l O A u f n m Z d L 2 m 6 z T d G 6 / e 8 s o 6 q u A Y n I A z 4 I J z 0 A J X o A 0 6 A I N H 8 A x e w Z v 1 Z L 1 Y 7 9 b H f L R i l T u H 4 A + s z x / R Y Z S + < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 c G S s e x z o I g E 8 k z y x r h n z z W 6 l Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V k g b E S C 2 x F o i 1 S E y L H d V q r j h 3 Z D l K J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H J 0 o Z V d p x v q 3 K y u r a + k Z 1 s 7 a 1 v b O 7 Z + 8 f d J X I J C Y d L J i Q d x F S h F F O O p p q R u 5 S S V A S M d K L x p e F 3 3 s g U l H B b / U k J U G C h p z G F C N t p N A + a v g J 0 i O M W H 4 9 D b 1 7 X 4 u 0 E d p 1 p + n M A J e J W 5 I 6 K N E O 7 S 9 / I H C W E K 4 x Q 0 r 1 X S f V Q Y 6 k p p i R a c 3 P F E k R H q M h 6 R v K U U J U k M / S T + G p U Q Y w F t I 8 r u F M / b 2 R o 0 S p S R K Z y S K q W v Q K 8 T + v n + n 4 I s g p T z N N O J 4 f i j M G t Y B F F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j N l O A u f n m Z d L 2 m 6 z T d G 6 / e 8 s o 6 q u A Y n I A z 4 I J z 0 A J X o A 0 6 A I N H 8 A x e w Z v 1 Z L 1 Y 7 9 b H f L R i l T u H 4 A + s z x / R Y Z S + < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 c G S s e x z o I g E 8 k z y x r h n z z W 6 l Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V k g b E S C 2 x F o i 1 S E y L H d V q r j h 3 Z D l K J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H J 0 o Z V d p x v q 3 K y u r a + k Z 1 s 7 a 1 v b O 7 Z + 8 f d J X I J C Y d L J i Q d x F S h F F O O p p q R u 5 S S V A S M d K L x p e F 3 3 s g U l H B b / U k J U G C h p z G F C N t p N A + a v g J 0 i O M W H 4 9 D b 1 7 X 4 u 0 E d p 1 p + n M A J e J W 5 I 6 K N E O 7 S 9 / I H C W E K 4 x Q 0 r 1 X S f V Q Y 6 k p p i R a c 3 P F E k R H q M h 6 R v K U U J U k M / S T + G p U Q Y w F t I 8 r u F M / b 2 R o 0 S p S R K Z y S K q W v Q K 8 T + v n + n 4 I s g p T z N N O J 4 f i j M G t Y B F F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j N l O A u f n m Z d L 2 m 6 z T d G 6 / e 8 s o 6 q u A Y n I A z 4 I J z 0 A J X o A 0 6 A I N H 8 A x e w Z v 1 Z L 1 Y 7 9 b H f L R i l T u H 4 A + s z x / R Y Z S + < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 c G S s e x z o I g E 8 k z y x r h n z z W 6 l Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V k g b E S C 2 x F o i 1 S E y L H d V q r j h 3 Z D l K J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H J 0 o Z V d p x v q 3 K y u r a + k Z 1 s 7 a 1 v b O 7 Z + 8 f d J X I J C Y d L J i Q d x F S h F F O O p p q R u 5 S S V A S M d K L x p e F 3 3 s g U l H B b / U k J U G C h p z G F C N t p N A + a v g J 0 i O M W H 4 9 D b 1 7 X 4 u 0 E d p 1 p + n M A J e J W 5 I 6 K N E O 7 S 9 / I H C W E K 4 x Q 0 r 1 X S f V Q Y 6 k p p i R a c 3 P F E k R H q M h 6 R v K U U J U k M / S T + G p U Q Y w F t I 8 r u F M / b 2 R o 0 S p S R K Z y S K q W v Q K 8 T + v n + n 4 I s g p T z N N O J 4 f i j M G t Y B F F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j N l O A u f n m Z d L 2 m 6 z T d G 6 / e 8 s o 6 q u A Y n I A z 4 I J z 0 A J X o A 0 6 A I N H 8 A x e w Z v 1 Z L 1 Y 7 9 b H f L R i l T u H 4 A + s z x / R Y Z S + < / l a t e x i t > I > 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 c G S s e x z o I g E 8 k z y x r h n z z W 6 l Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V k g b E S C 2 x F o i 1 S E y L H d V q r j h 3 Z D l K J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H J 0 o Z V d p x v q 3 K y u r a + k Z 1 s 7 a 1 v b O 7 Z + 8 f d J X I J C Y d L J i Q d x F S h F F O O p p q R u 5 S S V A S M d K L x p e F 3 3 s g U l H B b / U k J U G C h p z G F C N t p N A + a v g J 0 i O M W H 4 9 D b 1 7 X 4 u 0 E d p 1 p + n M A J e J W 5 I 6 K N E O 7 S 9 / I H C W E K 4 x Q 0 r 1 X S f V Q Y 6 k p p i R a c 3 P F E k R H q M h 6 R v K U U J U k M / S T + G p U Q Y w F t I 8 r u F M / b 2 R o 0 S p S R K Z y S K q W v Q K 8 T + v n + n 4 I s g p T z N N O J 4 f i j M G t Y B F F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j N l O A u f n m Z d L 2 m 6 z T d G 6 / e 8 s o 6 q u A Y n I A z 4 I J z 0 A J X o A 0 6 A I N H 8 A x e w Z v 1 Z L 1 Y 7 9 b H f L R i l T u H 4 A + s z x / R Y Z S + < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 c G S s e x z o I g E 8 k z y x r h n z z W 6 l Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V k g b E S C 2 x F o i 1 S E y L H d V q r j h 3 Z D l K J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H J 0 o Z V d p x v q 3 K y u r a + k Z 1 s 7 a 1 v b O 7 Z + 8 f d J X I J C Y d L J i Q d x F S h F F O O p p q R u 5 S S V A S M d K L x p e F 3 3 s g U l H B b / U k J U G C h p z G F C N t p N A + a v g J 0 i O M W H 4 9 D b 1 7 X 4 u 0 E d p 1 p + n M A J e J W 5 I 6 K N E O 7 S 9 / I H C W E K 4 x Q 0 r 1 X S f V Q Y 6 k p p i R a c 3 P F E k R H q M h 6 R v K U U J U k M / S T + G p U Q Y w F t I 8 r u F M / b 2 R o 0 S p S R K Z y S K q W v Q K 8 T + v n + n 4 I s g p T z N N O J 4 f i j M G t Y B F F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j N l O A u f n m Z d L 2 m 6 z T d G 6 / e 8 s o 6 q u A Y n I A z 4 I J z 0 A J X o A 0 6 A I N H 8 A x e w Z v 1 Z L 1 Y 7 9 b H f L R i l T u H 4 A + s z x / R Y Z S + < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 c G S s e x z o I g E 8 k z y x r h n z z W 6 l Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V k g b E S C 2 x F o i 1 S E y L H d V q r j h 3 Z D l K J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H J 0 o Z V d p x v q 3 K y u r a + k Z 1 s 7 a 1 v b O 7 Z + 8 f d J X I J C Y d L J i Q d x F S h F F O O p p q R u 5 S S V A S M d K L x p e F 3 3 s g U l H B b / U k J U G C h p z G F C N t p N A + a v g J 0 i O M W H 4 9 D b 1 7 X 4 u 0 E d p 1 p + n M A J e J W 5 I 6 K N E O 7 S 9 / I H C W E K 4 x Q 0 r 1 X S f V Q Y 6 k p p i R a c 3 P F E k R H q M h 6 R v K U U J U k M / S T + G p U Q Y w F t I 8 r u F M / b 2 R o 0 S p S R K Z y S K q W v Q K 8 T + v n + n 4 I s g p T z N N O J 4 f i j M G t Y B F F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j N l O A u f n m Z d L 2 m 6 z T d G 6 / e 8 s o 6 q u A Y n I A z 4 I J z 0 A J X o A 0 6 A I N H 8 A x e w Z v 1 Z L 1 Y 7 9 b H f L R i l T u H 4 A + s z x / R Y Z S + < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 c G S s e x z o I g E 8 k z y x r h n z z W 6 l Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V k g b E S C 2 x F o i 1 S E y L H d V q r j h 3 Z D l K J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H J 0 o Z V d p x v q 3 K y u r a + k Z 1 s 7 a 1 v b O 7 Z + 8 f d J X I J C Y d L J i Q d x F S h F F O O p p q R u 5 S S V A S M d K L x p e F 3 3 s g U l H B b / U k J U G C h p z G F C N t p N A + a v g J 0 i O M W H 4 9 D b 1 7 X 4 u 0 E d p 1 p + n M A J e J W 5 I 6 K N E O 7 S 9 / I H C W E K 4 x Q 0 r 1 X S f V Q Y 6 k p p i R a c 3 P F E k R H q M h 6 R v K U U J U k M / S T + G p U Q Y w F t I 8 r u F M / b 2 R o 0 S p S R K Z y S K q W v Q K 8 T + v n + n 4 I s g p T z N N O J 4 f i j M G t Y B F F X B A J c G a T Q x B W F K T F e I R k g h r U 1 j N l O A u f n m Z d L 2 m 6 z T d G 6 / e 8 s o 6 q u A Y n I A z 4 I J z 0 A J X o A 0 6 A I N H 8 A x e w Z v 1 Z L 1 Y 7 9 b H f L R i l T u H 4 A + s z x / R Y Z S + < / l a t e x i t > I > 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " e f r s S X s P y t n S P S O B B k G X X 4 g K h u I = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 W U A c Z K L L A V i T 6 k J k S O 6 7 R W H T u y H a Q S V f w K C w M I s f I f b P w N T p s B W o 5 k 6 e i c e 3 W P T 5 g w q r T j f F u l l d W 1 9 Y 3 y Z m V r e 2 d 3 z 9 4 / 6 C i R S k z a W D A h e y F S h F F O 2 p p q R n q J J C g O G e m G 4 6 v c 7 z 4 Q q a j g d 3 q S E D 9 G Q 0 4 j i p E 2 U m A f 1 b w Y 6 R F G L L u Z B u f 3 n h Z J L b C r T t 2 Z A S 4 T t y B V U K A V 2 F / e Q O A 0 J l x j h p T q u 0 6 i / Q x J T T E j 0 4 q X K p I g P E Z D 0 j e U o 5 g o P 5 u l n 8 J T o w x g J K R 5 X M O Z + n s j Q 7 F S k z g 0 k 3 l U t e j l 4 n 9 e P 9 X R p Z 9 R n q S a c D w / F K U M a g H z K u C A S o I 1 m x i C s K Q m K 8 Q j J B H W p r C K K c F d / P I y 6 T T q r l N 3 b x v V Z q O o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 u u U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e f r s S X s P y t n S P S O B B k G X X 4 g K h u I = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 W U A c Z K L L A V i T 6 k J k S O 6 7 R W H T u y H a Q S V f w K C w M I s f I f b P w N T p s B W o 5 k 6 e i c e 3 W P T 5 g w q r T j f F u l l d W 1 9 Y 3 y Z m V r e 2 d 3 z 9 4 / 6 C i R S k z a W D A h e y F S h F F O 2 p p q R n q J J C g O G e m G 4 6 v c 7 z 4 Q q a j g d 3 q S E D 9 G Q 0 4 j i p E 2 U m A f 1 b w Y 6 R F G L L u Z B u f 3 n h Z J L b C r T t 2 Z A S 4 T t y B V U K A V 2 F / e Q O A 0 J l x j h p T q u 0 6 i / Q x J T T E j 0 4 q X K p I g P E Z D 0 j e U o 5 g o P 5 u l n 8 J T o w x g J K R 5 X M O Z + n s j Q 7 F S k z g 0 k 3 l U t e j l 4 n 9 e P 9 X R p Z 9 R n q S a c D w / F K U M a g H z K u C A S o I 1 m x i C s K Q m K 8 Q j J B H W p r C K K c F d / P I y 6 T T q r l N 3 b x v V Z q O o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 u u U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e f r s S X s P y t n S P S O B B k G X X 4 g K h u I = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 W U A c Z K L L A V i T 6 k J k S O 6 7 R W H T u y H a Q S V f w K C w M I s f I f b P w N T p s B W o 5 k 6 e i c e 3 W P T 5 g w q r T j f F u l l d W 1 9 Y 3 y Z m V r e 2 d 3 z 9 4 / 6 C i R S k z a W D A h e y F S h F F O 2 p p q R n q J J C g O G e m G 4 6 v c 7 z 4 Q q a j g d 3 q S E D 9 G Q 0 4 j i p E 2 U m A f 1 b w Y 6 R F G L L u Z B u f 3 n h Z J L b C r T t 2 Z A S 4 T t y B V U K A V 2 F / e Q O A 0 J l x j h p T q u 0 6 i / Q x J T T E j 0 4 q X K p I g P E Z D 0 j e U o 5 g o P 5 u l n 8 J T o w x g J K R 5 X M O Z + n s j Q 7 F S k z g 0 k 3 l U t e j l 4 n 9 e P 9 X R p Z 9 R n q S a c D w / F K U M a g H z K u C A S o I 1 m x i C s K Q m K 8 Q j J B H W p r C K K c F d / P I y 6 T T q r l N 3 b x v V Z q O o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 u u U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e f r s S X s P y t n S P S O B B k G X X 4 g K h u I = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 W U A c Z K L L A V i T 6 k J k S O 6 7 R W H T u y H a Q S V f w K C w M I s f I f b P w N T p s B W o 5 k 6 e i c e 3 W P T 5 g w q r T j f F u l l d W 1 9 Y 3 y Z m V r e 2 d 3 z 9 4 / 6 C i R S k z a W D A h e y F S h F F O 2 p p q R n q J J C g O G e m G 4 6 v c 7 z 4 Q q a j g d 3 q S E D 9 G Q 0 4 j i p E 2 U m A f 1 b w Y 6 R F G L L u Z B u f 3 n h Z J L b C r T t 2 Z A S 4 T t y B V U K A V 2 F / e Q O A 0 J l x j h p T q u 0 6 i / Q x J T T E j 0 4 q X K p I g P E Z D 0 j e U o 5 g o P 5 u l n 8 J T o w x g J K R 5 X M O Z + n s j Q 7 F S k z g 0 k 3 l U t e j l 4 n 9 e P 9 X R p Z 9 R n q S a c D w / F K U M a g H z K u C A S o I 1 m x i C s K Q m K 8 Q j J B H W p r C K K c F d / P I y 6 T T q r l N 3 b x v V Z q O o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 u u U v w = = < / l a t e x i t > I > 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " e f r s S X s P y t n S P S O B B k G X X 4 g K h u I = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 W U A c Z K L L A V i T 6 k J k S O 6 7 R W H T u y H a Q S V f w K C w M I s f I f b P w N T p s B W o 5 k 6 e i c e 3 W P T 5 g w q r T j f F u l l d W 1 9 Y 3 y Z m V r e 2 d 3 z 9 4 / 6 C i R S k z a W D A h e y F S h F F O 2 p p q R n q J J C g O G e m G 4 6 v c 7 z 4 Q q a j g d 3 q S E D 9 G Q 0 4 j i p E 2 U m A f 1 b w Y 6 R F G L L u Z B u f 3 n h Z J L b C r T t 2 Z A S 4 T t y B V U K A V 2 F / e Q O A 0 J l x j h p T q u 0 6 i / Q x J T T E j 0 4 q X K p I g P E Z D 0 j e U o 5 g o P 5 u l n 8 J T o w x g J K R 5 X M O Z + n s j Q 7 F S k z g 0 k 3 l U t e j l 4 n 9 e P 9 X R p Z 9 R n q S a c D w / F K U M a g H z K u C A S o I 1 m x i C s K Q m K 8 Q j J B H W p r C K K c F d / P I y 6 T T q r l N 3 b x v V Z q O o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 u u U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e f r s S X s P y t n S P S O B B k G X X 4 g K h u I = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 W U A c Z K L L A V i T 6 k J k S O 6 7 R W H T u y H a Q S V f w K C w M I s f I f b P w N T p s B W o 5 k 6 e i c e 3 W P T 5 g w q r T j f F u l l d W 1 9 Y 3 y Z m V r e 2 d 3 z 9 4 / 6 C i R S k z a W D A h e y F S h F F O 2 p p q R n q J J C g O G e m G 4 6 v c 7 z 4 Q q a j g d 3 q S E D 9 G Q 0 4 j i p E 2 U m A f 1 b w Y 6 R F G L L u Z B u f 3 n h Z J L b C r T t 2 Z A S 4 T t y B V U K A V 2 F / e Q O A 0 J l x j h p T q u 0 6 i / Q x J T T E j 0 4 q X K p I g P E Z D 0 j e U o 5 g o P 5 u l n 8 J T o w x g J K R 5 X M O Z + n s j Q 7 F S k z g 0 k 3 l U t e j l 4 n 9 e P 9 X R p Z 9 R n q S a c D w / F K U M a g H z K u C A S o I 1 m x i C s K Q m K 8 Q j J B H W p r C K K c F d / P I y 6 T T q r l N 3 b x v V Z q O o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 u u U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e f r s S X s P y t n S P S O B B k G X X 4 g K h u I = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 W U A c Z K L L A V i T 6 k J k S O 6 7 R W H T u y H a Q S V f w K C w M I s f I f b P w N T p s B W o 5 k 6 e i c e 3 W P T 5 g w q r T j f F u l l d W 1 9 Y 3 y Z m V r e 2 d 3 z 9 4 / 6 C i R S k z a W D A h e y F S h F F O 2 p p q R n q J J C g O G e m G 4 6 v c 7 z 4 Q q a j g d 3 q S E D 9 G Q 0 4 j i p E 2 U m A f 1 b w Y 6 R F G L L u Z B u f 3 n h Z J L b C r T t 2 Z A S 4 T t y B V U K A V 2 F / e Q O A 0 J l x j h p T q u 0 6 i / Q x J T T E j 0 4 q X K p I g P E Z D 0 j e U o 5 g o P 5 u l n 8 J T o w x g J K R 5 X M O Z + n s j Q 7 F S k z g 0 k 3 l U t e j l 4 n 9 e P 9 X R p Z 9 R n q S a c D w / F K U M a g H z K u C A S o I 1 m x i C s K Q m K 8 Q j J B H W p r C K K c F d / P I y 6 T T q r l N 3 b x v V Z q O o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 u u U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e f r s S X s P y t n S P S O B B k G X X 4 g K h u I = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 W U A c Z K L L A V i T 6 k J k S O 6 7 R W H T u y H a Q S V f w K C w M I s f I f b P w N T p s B W o 5 k 6 e i c e 3 W P T 5 g w q r T j f F u l l d W 1 9 Y 3 y Z m V r e 2 d 3 z 9 4 / 6 C i R S k z a W D A h e y F S h F F O 2 p p q R n q J J C g O G e m G 4 6 v c 7 z 4 Q q a j g d 3 q S E D 9 G Q 0 4 j i p E 2 U m A f 1 b w Y 6 R F G L L u Z B u f 3 n h Z J L b C r T t 2 Z A S 4 T t y B V U K A V 2 F / e Q O A 0 J l x j h p T q u 0 6 i / Q x J T T E j 0 4 q X K p I g P E Z D 0 j e U o 5 g o P 5 u l n 8 J T o w x g J K R 5 X M O Z + n s j Q 7 F S k z g 0 k 3 l U t e j l 4 n 9 e P 9 X R p Z 9 R n q S a c D w / F K U M a g H z K u C A S o I 1 m x i C s K Q m K 8 Q j J B H W p r C K K c F d / P I y 6 T T q r l N 3 b x v V Z q O o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 u u U v w = = < / l a t e x i t > I > 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " e f r s S X s P y t n S P S O B B k G X X 4 g K h u I = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 W U A c Z K L L A V i T 6 k J k S O 6 7 R W H T u y H a Q S V f w K C w M I s f I f b P w N T p s B W o 5 k 6 e i c e 3 W P T 5 g w q r T j f F u l l d W 1 9 Y 3 y Z m V r e 2 d 3 z 9 4 / 6 C i R S k z a W D A h e y F S h F F O 2 p p q R n q J J C g O G e m G 4 6 v c 7 z 4 Q q a j g d 3 q S E D 9 G Q 0 4 j i p E 2 U m A f 1 b w Y 6 R F G L L u Z B u f 3 n h Z J L b C r T t 2 Z A S 4 T t y B V U K A V 2 F / e Q O A 0 J l x j h p T q u 0 6 i / Q x J T T E j 0 4 q X K p I g P E Z D 0 j e U o 5 g o P 5 u l n 8 J T o w x g J K R 5 X M O Z + n s j Q 7 F S k z g 0 k 3 l U t e j l 4 n 9 e P 9 X R p Z 9 R n q S a c D w / F K U M a g H z K u C A S o I 1 m x i C s K Q m K 8 Q j J B H W p r C K K c F d / P I y 6 T T q r l N 3 b x v V Z q O o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 u u U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e f r s S X s P y t n S P S O B B k G X X 4 g K h u I = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 W U A c Z K L L A V i T 6 k J k S O 6 7 R W H T u y H a Q S V f w K C w M I s f I f b P w N T p s B W o 5 k 6 e i c e 3 W P T 5 g w q r T j f F u l l d W 1 9 Y 3 y Z m V r e 2 d 3 z 9 4 / 6 C i R S k z a W D A h e y F S h F F O 2 p p q R n q J J C g O G e m G 4 6 v c 7 z 4 Q q a j g d 3 q S E D 9 G Q 0 4 j i p E 2 U m A f 1 b w Y 6 R F G L L u Z B u f 3 n h Z J L b C r T t 2 Z A S 4 T t y B V U K A V 2 F / e Q O A 0 J l x j h p T q u 0 6 i / Q x J T T E j 0 4 q X K p I g P E Z D 0 j e U o 5 g o P 5 u l n 8 J T o w x g J K R 5 X M O Z + n s j Q 7 F S k z g 0 k 3 l U t e j l 4 n 9 e P 9 X R p Z 9 R n q S a c D w / F K U M a g H z K u C A S o I 1 m x i C s K Q m K 8 Q j J B H W p r C K K c F d / P I y 6 T T q r l N 3 b x v V Z q O o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 u u U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e f r s S X s P y t n S P S O B B k G X X 4 g K h u I = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 W U A c Z K L L A V i T 6 k J k S O 6 7 R W H T u y H a Q S V f w K C w M I s f I f b P w N T p s B W o 5 k 6 e i c e 3 W P T 5 g w q r T j f F u l l d W 1 9 Y 3 y Z m V r e 2 d 3 z 9 4 / 6 C i R S k z a W D A h e y F S h F F O 2 p p q R n q J J C g O G e m G 4 6 v c 7 z 4 Q q a j g d 3 q S E D 9 G Q 0 4 j i p E 2 U m A f 1 b w Y 6 R F G L L u Z B u f 3 n h Z J L b C r T t 2 Z A S 4 T t y B V U K A V 2 F / e Q O A 0 J l x j h p T q u 0 6 i / Q x J T T E j 0 4 q X K p I g P E Z D 0 j e U o 5 g o P 5 u l n 8 J T o w x g J K R 5 X M O Z + n s j Q 7 F S k z g 0 k 3 l U t e j l 4 n 9 e P 9 X R p Z 9 R n q S a c D w / F K U M a g H z K u C A S o I 1 m x i C s K Q m K 8 Q j J B H W p r C K K c F d / P I y 6 T T q r l N 3 b x v V Z q O o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 u u U v w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e f r s S X s P y t n S P S O B B k G X X 4 g K h u I = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 W U A c Z K L L A V i T 6 k J k S O 6 7 R W H T u y H a Q S V f w K C w M I s f I f b P w N T p s B W o 5 k 6 e i c e 3 W P T 5 g w q r T j f F u l l d W 1 9 Y 3 y Z m V r e 2 d 3 z 9 4 / 6 C i R S k z a W D A h e y F S h F F O 2 p p q R n q J J C g O G e m G 4 6 v c 7 z 4 Q q a j g d 3 q S E D 9 G Q 0 4 j i p E 2 U m A f 1 b w Y 6 R F G L L u Z B u f 3 n h Z J L b C r T t 2 Z A S 4 T t y B V U K A V 2 F / e Q O A 0 J l x j h p T q u 0 6 i / Q x J T T E j 0 4 q X K p I g P E Z D 0 j e U o 5 g o P 5 u l n 8 J T o w x g J K R 5 X M O Z + n s j Q 7 F S k z g 0 k 3 l U t e j l 4 n 9 e P 9 X R p Z 9 R n q S a c D w / F K U M a g H z K u C A S o I 1 m x i C s K Q m K 8 Q j J B H W p r C K K c F d / P I y 6 T T q r l N 3 b x v V Z q O o o w y O w Q k 4 A y 6 4 A E 1 w D V q g D T B 4 B M / g F b x Z T 9 a L 9 W 5 9 z E d L V r F z C P 7 A + v w B 0 u u U v w = = < / l a t e x i t > ⇥ < l a t e x i t s h a 1 _ b a s e 6 4 = " o / T z t k g j x W N 4 e K N C g A h p t E P q j k A = " > A A A B 7 3 i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 2 l p j I R w I X s r f s w Y a 9 v X N 3 z o R c + B M 2 F h p j 6 9 + x 8 9 + 4 w B U K v m S S l / d m M j M v S K Q w 6 L r f T m F j c 2 t 7 p 7 h b 2 t s / O D w q H 5 + 0 T Z x q x l s s l r H u B t R w K R R v o U D J u 4 n m N A o k 7 w S T 2 7 n f e e L a i F g 9 4 D T h f k R H S o S C U b R S t 9 p H E X F T H Z Q r b s 1 d g K w T L y c V y N E c l L / 6 w 5 i l E V f I J D W m 5 7 k J + h n V K J j k s 1 I / N T y h b E J H v G e p o n a L n y 3 u n Z E L q w x J G G t b C s l C / T 2 R 0 c i Y a R T Y z o j i 2 K x 6 c / E / r 5 d i e O 1 n Q i U p c s W W i 8 J U E o z J / H k y F J o z l F N L K N P C 3 k r Y m G r K 0 E Z U s i F 4 q y + v k 3 a 9 5 r k 1 7 7 5 e a d z k c R T h D M 7 h E j y 4 g g b c Q R N a w E D C M 7 z C m / P o v D j v z s e y t e D k M 6 f w B 8 7 n D 3 A S j 4 4 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o / T z t k g j x W N 4 e K N C g A h p t E P q j k A = " > A A A B 7 3 i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 2 l p j I R w I X s r f s w Y a 9 v X N 3 z o R c + B M 2 F h p j 6 9 + x 8 9 + 4 w B U K v m S S l / d m M j M v S K Q w 6 L r f T m F j c 2 t 7 p 7 h b 2 t s / O D w q H 5 + 0 T Z x q x l s s l r H u B t R w K R R v o U D J u 4 n m N A o k 7 w S T 2 7 n f e e L a i F g 9 4 D T h f k R H S o S C U b R S t 9 p H E X F T H Z Q r b s 1 d g K w T L y c V y N E c l L / 6 w 5 i l E V f I J D W m 5 7 k J + h n V K J j k s 1 I / N T y h b E J H v G e p o n a L n y 3 u n Z E L q w x J G G t b C s l C / T 2 R 0 c i Y a R T Y z o j i 2 K x 6 c / E / r 5 d i e O 1 n Q i U p c s W W i 8 J U E o z J / H k y F J o z l F N L K N P C 3 k r Y m G r K 0 E Z U s i F 4 q y + v k 3 a 9 5 r k 1 7 7 5 e a d z k c R T h D M 7 h E j y 4 g g b c Q R N a w E D C M 7 z C m / P o v D j v z s e y t e D k M 6 f w B 8 7 n D 3 A S j 4 4 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o / T z t k g j x W N 4 e K N C g A h p t E P q j k A = " > A A A B 7 3 i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 2 l p j I R w I X s r f s w Y a 9 v X N 3 z o R c + B M 2 F h p j 6 9 + x 8 9 + 4 w B U K v m S S l / d m M j M v S K Q w 6 L r f T m F j c 2 t 7 p 7 h b 2 t s / O D w q H 5 + 0 T Z x q x l s s l r H u B t R w K R R v o U D J u 4 n m N A o k 7 w S T 2 7 n f e e L a i F g 9 4 D T h f k R H S o S C U b R S t 9 p H E X F T H Z Q r b s 1 d g K w T L y c V y N E c l L / 6 w 5 i l E V f I J D W m 5 7 k J + h n V K J j k s 1 I / N T y h b E J H v G e p o n a L n y 3 u n Z E L q w x J G G t b C s l C / T 2 R 0 c i Y a R T Y z o j i 2 K x 6 c / E / r 5 d i e O 1 n Q i U p c s W W i 8 J U E o z J / H k y F J o z l F N L K N P C 3 k r Y m G r K 0 E Z U s i F 4 q y + v k 3 a 9 5 r k 1 7 7 5 e a d z k c R T h D M 7 h E j y 4 g g b c Q R N a w E D C M 7 z C m / P o v D j v z s e y t e D k M 6 f w B 8 7 n D 3 A S j 4 4 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o / T z t k g j x W N 4 e K N C g A h p t E P q j k A = " > A A A B 7 3 i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 2 l p j I R w I X s r f s w Y a 9 v X N 3 z o R c + B M 2 F h p j 6 9 + x 8 9 + 4 w B U K v m S S l / d m M j M v S K Q w 6 L r f T m F j c 2 t 7 p 7 h b 2 t s / O D w q H 5 + 0 T Z x q x l s s l r H u B t R w K R R v o U D J u 4 n m N A o k 7 w S T 2 7 n f e e L a i F g 9 4 D T h f k R H S o S C U b R S t 9 p H E X F T H Z Q r b s 1 d g K w T L y c V y N E c l L / 6 w 5 i l E V f I J D W m 5 7 k J + h n V K J j k s 1 I / N T y h b E J H v G e p o n a L n y 3 u n Z E L q w x J G G t b C s l C / T 2 R 0 c i Y a R T Y z o j i 2 K x 6 c / E / r 5 d i e O 1 n Q i U p c s W W i 8 J U E o z J / H k y F J o z l F N L K N P C 3 k r Y m G r K 0 E Z U s i F 4 q y + v k 3 a 9 5 r k 1 7 7 5 e a d z k c R T h D M 7 h E j y 4 g g b c Q R N a w E D C M 7 z C m / P o v D j v z s e y t e D k M 6 f w B 8 7 n D 3 A S j 4 4 = < / l a t e x i t > =) < l a t e x i t s h a 1 _ b a s e 6 4 = " v C K c / 6 h B H k Q X g f n 2 B g 3 R p l K l H z A = " > A A A B + n i c b V B N T 8 J A E J 3 i F + J X 0 a O X R j D x R F o u e i R 6 8 e A B E / l I o C H b Z V s 2 b H e b 3 a 2 E V H 6 K F w 8 a 4 9 V f 4 s 1 / 4 w I 9 K P i S S V 7 e m 8 n M v C B h V G n X / b Y K G 5 t b 2 z v F 3 d L e / s H h k V 0 + b i u R S k x a W D A h u w F S h F F O W p p q R r q J J C g O G O k E 4 5 u 5 3 3 k k U l H B H / Q 0 I X 6 M I k 5 D i p E 2 0 s A u V / t 3 g k e S R i O N p B S T 6 s C u u D V 3 A W e d e D m p Q I 7 m w P 7 q D w V O Y 8 I 1 Z k i p n u c m 2 s + Q 1 B Q z M i v 1 U 0 U S h M c o I j 1 D O Y q J 8 r P F 6 T P n 3 C h D J x T S F N f O Q v 0 9 k a F Y q W k c m M 4 Y 6 Z F a 9 e b i f 1 4 v 1 e G V n 1 G e p J p w v F w U p s z R w p n n 4 A y p J F i z q S E I S 2 p u d f A I S Y S 1 S a t k Q v B W X 1 4 n 7 X r N c 2 v e f b 3 S u M 7 j K M I p n M E F e H A J D b i F J r Q A w w S e 4

R X e r C f r x X q 3 P p a t B S u f O Y E / s D 5 / A P 9 N k 9 A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = "

v C K c / 6 h B H k Q X g f n 2 B g 3 R p l K l H z A = " > A A A B + n i c b V B N T 8 J A E J 3 i F + J X 0 a O X R j D x R F o u e i R 6 8 e A B E / l I o C H b Z V s 2 b H e b 3 a 2 E V H 6 K F w 8 a 4 9 V f 4 s 1 / 4 w I 9 K P i S S V 7 e m 8 n M v C B h V G n X / b Y K G 5 t b 2 z v F 3 d L e / s H h k V 0 + b i u R S k x a W D A h u w F S h F F O W p p q R r q J J C g O G O k E 4 5 u 5 3 3 k k U l H B H / Q 0 I X 6 M I k 5 D i p E 2 0 s A u V / t 3 g k e S R i O N p B S T 6 s C u u D V 3 A W e d e D m p Q I 7 m w P 7 q D w V O Y 8 I 1 Z k i p n u c m 2 s + Q 1 B Q z M i v 1 U 0 U S h M c o I j 1 D O Y q J 8 r P F 6 T P n 3 C h D J x T S F N f O Q v 0 9 k a F Y q W k c m M 4 Y 6 Z F a 9 e b i f 1 4 v 1 e G V n 1 G e p J p w v F w U p s z R w p n n 4 A y p J F i z q S E I S 2 p u d f A I S Y S 1 S a t k Q v B W X 1 4 n 7 X r N c 2 v e f b 3 S u M 7 j K M I p n M E F e H A J D b i F J r Q A w w S e 4

R X e r C f r x X q 3 P p a t B S u f O Y E / s D 5 / A P 9 N k 9 A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = "

v C K c / 6 h B H k Q X g f n 2 B g 3 R p l K l H z A = " > A A A B + n i c b V B N T 8 J A E J 3 i F + J X 0 a O X R j D x R F o u e i R 6 8 e A B E / l I o C H b Z V s 2 b H e b 3 a 2 E V H 6 K F w 8 a 4 9 V f 4 s 1 / 4 w I 9 K P i S S V 7 e m 8 n M v C B h V G n X / b Y K G 5 t b 2 z v F 3 d L e / s H h k V 0 + b i u R S k x a W D A h u w F S h F F O W p p q R r q J J C g O G O k E 4 5 u 5 3 3 k k U l H B H / Q 0 I X 6 M I k 5 D i p E 2 0 s A u V / t 3 g k e S R i O N p B S T 6 s C u u D V 3 A W e d e D m p Q I 7 m w P 7 q D w V O Y 8 I 1 Z k i p n u c m 2 s + Q 1 B Q z M i v 1 U 0 U S h M c o I j 1 D O Y q J 8 r P F 6 T P n 3 C h D J x T S F N f O Q v 0 9 k a F Y q W k c m M 4 Y 6 Z F a 9 e b i f 1 4 v 1 e G V n 1 G e p J p w v F w U p s z R w p n n 4 A y p J F i z q S E I S 2 p u d f A I S Y S 1 S a t k Q v B W X 1 4 n 7 X r N c 2 v e f b 3 S u M 7 j K M I p n M E F e H A J D b i F J r Q A w w S e 4

R X e r C f r x X q 3 P p a t B S u f O Y E / s D 5 / A P 9 N k 9 A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 

v C K c / 6 h B H k Q X g f n 2 B g 3 R p l K l H z A = " > A A A B + n i c b V B N T 8 J A E J 3 i F + J X 0 a O X R j D x R F o u e i R 6 8 e A B E / l I o C H b Z V s 2 b H e b 3 a 2 E V H 6 K F w 8 a 4 9 V f 4 s 1 / 4 w I 9 K P i S S V 7 e m 8 n M v C B h V G n X / b Y K G 5 t b 2 z v F 3 d L e / s H h k V 0 + b i u R S k x a W D A h u w F S h F F O W p p q R r q J J C g O G O k E 4 5 u 5 3 3 k k U l H B H / Q 0 I X 6 M I k 5 D i p E 2 0 s A u V / t 3 g k e S R i O N p B S T 6 s C u u D V 3 A W e d e D m p Q I 7 m w P 7 q D w V O Y 8 I 1 Z k i p n u c m 2 s + Q 1 B Q z M i v 1 U 0 U S h M c o I j 1 D O Y q J 8 r P F 6 T P n 3 C h D J x T S F N f O Q v 0 9 k a F Y q W k c m M 4 Y 6 Z F

= " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 0 l p h 4 Q A I X s r f M w Y a 9 v c v u n g k h / A Y b C 4 2 x 9 Q f Z + W 9 c 4 A o F X z L J y 3 s z m Z k X p o J r 4 7 r f T m F r e 2 d 3 r 7 h f O j g 8 O j 4 p n 5 6 1 d Z I p h j 5 L R K K 6 I d U o u E T f c C O w m y q k c S i w E 0 6 a C 7 / z h E r z R D 6 a a Y p B T E e S R 5 x R Y y W / 2 h z U q 4 N y x a 2 5 S 5 B N 4 u W k A j l a g / J X f 5 i w L E Z p m K B a 9 z w 3 N c G M K s O Z w H m p n 2 l M K Z v Q E f Y s l T R G H c y W x 8 7 J l V W G J E q U L W n I U v 0 9 M a O x 1 t M 4 t J 0 x N W O 9 7 i 3 E / 7 x e Z q L b Y M Z l m h m U b L U o y g Q x C V l 8 T o Z c I T N i a g l l i t t b C R t T R Z m x + Z R s C N 7 6 y 5 u k X a 9 5 b s 1 7 q F c a d 3 k c R b i A S 7 g G D 2 6 g A f f Q A h 8 Y c H i G V

= " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 0 l p h 4 Q A I X s r f M w Y a 9 v c v u n g k h / A Y b C 4 2 x 9 Q f Z + W 9 c 4 A o F X z L J y 3 s z m Z k X p o J r 4 7 r f T m F r e 2 d 3 r 7 h f O j g 8 O j 4 p n 5 6 1 d Z I p h j 5 L R K K 6 I d U o u E T f c C O w m y q k c S i w E 0 6 a C 7 / z h E r z R D 6 a a Y p B T E e S R 5 x R Y y W / 2 h z U q 4 N y x a 2 5 S 5 B N 4 u W k A j l a g / J X f 5 i w L E Z p m K B a 9 z w 3 N c G M K s O Z w H m p n 2 l M K Z v Q E f Y s l T R G H c y W x 8 7 J l V W G J E q U L W n I U v 0 9 M a O x 1 t M 4 t J 0 x N W O 9 7 i 3 E / 7 x e Z q L b Y M Z l m h m U b L U o y g Q x C V l 8 T o Z c I T N i a g l l i t t b C R t T R Z m x + Z R s C N 7 6 y 5 u k X a 9 5 b s 1 7 q F c a d 3 k c R b i A S 7 g G D 2 6 g A f f Q A h 8 Y c H i G V

= " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 0 l p h 4 Q A I X s r f M w Y a 9 v c v u n g k h / A Y b C 4 2 x 9 Q f Z + W 9 c 4 A o F X z L J y 3 s z m Z k X p o J r 4 7 r f T m F r e 2 d 3 r 7 h f O j g 8 O j 4 p n 5 6 1 d Z I p h j 5 L R K K 6 I d U o u E T f c C O w m y q k c S i w E 0 6 a C 7 / z h E r z R D 6 a a Y p B T E e S R 5 x R Y y W / 2 h z U q 4 N y x a 2 5 S 5 B N 4 u W k A j l a g / J X f 5 i w L E Z p m K B a 9 z w 3 N c G M K s O Z w H m p n 2 l M K Z v Q E f Y s l T R G H c y W x 8 7 J l V W G J E q U L W n I U v 0 9 M a O x 1 t M 4 t J 0 x N W O 9 7 i 3 E / 7 x e Z q L b Y M Z l m h m U b L U o y g Q x C V l 8 T o Z c I T N i a g l l i t t b C R t T R Z m x + Z R s C N 7 6 y 5 u k X a 9 5 b s 1 7 q F c a d 3 k c R b i A S 7 g G D 2 6 g A f f Q A h 8 Y c H i G V

= " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 0 l p h 4 Q A I X s r f M w Y a 9 v c v u n g k h / A Y b C 4 2 x 9 Q f Z + W 9 c 4 A o F X z L J y 3 s z m Z k X p o J r 4 7 r f T m F r e 2 d 3 r 7 h f O j g 8 O j 4 p n 5 6 1 d Z I p h j 5 L R K K 6 I d U o u E T f c C O w m y q k c S i w E 0 6 a C 7 / z h E r z R D 6 a a Y p B T E e S R 5 x R Y y W / 2 h z U q 4 N y x a 2 5 S 5 B N 4 u W k A j l a g / J X f 5 i w L E Z p m K B a 9 z w 3 N c G M K s O Z w H m p n 2 l M K Z v Q E f Y s l T R G H c y W x 8 7 J l V W G J E q U L W n I U v 0 9 M a O x 1 t M 4 t J 0 x N W O 9 7 i 3 E / 7 x e Z q L b Y M Z l m h m U b L U o y g Q x C V l 8 T o Z c I T N i a g l l i t t b C R t T R Z m x + Z R s C N 7 6 y 5 u k X a 9 5 b s 1 7 q F c a d 3 k c R b i A S 7 g G D 2 6 g A f f Q A h 8 Y c H i G V 3 h z p P P i v D s f q 9 a C k 8 + c w x 8 4 n z 9 0 L I 3 G < / l a t e x i t > C 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " 8 p n I 6 2 g / O W E 2 R 3 y z l b m X z g 5 Q R n k = " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s s t C T S W G L i I Q l c y N 4 y B

x v 2 9 i 6 7 e y a E 8 B t s L D T G 1 h 9 k 5 7 9 x g S s U f M k k L + / N Z G Z e m A q u j e t + O 4 W N z a 3 t n e J u a W / / 4 P C o f H z S 1 k m m G P o s E Y n q h F S j 4 B J 9 w 4 3 A T q q Q x q H A x 3 D c n P u P T 6 g 0 T q V g k 7 v Q k p l 4 I Q 8 E C R k A b q W 8 f V X o h 6 B E B n t 5 M c Q + U x g + V v l 1 2 q s 4 M e J m 4 O S m j H I 2 + / d U b R C Q J q d C E g 1 J d 1 4 m 1 l 4 L U j H A 6 L f U S R W M g Y x j S r q E C Q q q 8 d J Z + i k + N M s B B J M 0 T G s / U 3 x s p h E p N Q t 9 M Z l H V o p e J / 3 n d R A c X X s p E n G g q y P x Q k H C s I 5 x V g Q d M U q L 5 x B A g k p m s m I x A A t G m s J I p w V 3 8 8 j J p 1 a q u U 3 V v a + X 6 Z V 5 H E R 2 j E 3 S G X H S O 6 u g a N V A T E f S I n t E r e r O e r B f r 3 f q Y j x a s f O c Q / Y H 1 + Q N x Q Z S M < / l a t e x i t >

T + S D m a Q Y x H Q o e c Q Z N V b y q 8 3 + V b V f r r g 1 d w G y T r y c V C B H q 1 / + 6 g 0 S l s U o D R N U 6 6 7 n p i a Y U m U 4 E z g r 9 T K N K W V j O s S u p Z L G q I P p 4 t g Z u b D K g E S J s i U N W a i / J 6 Y 0 1 n o S h 7 Y z p m a k V 7 2 5 + J / X z U x 0 E 0 y 5 T D O D k i 0 X R Z k g J i H z z 8 m A K 2 R G T C y h T H F 7 K 2 E j q i g z N p + S D c F b f X m d t O s 1 z 6 1 5 9 / V K 4 z a P o w h n c A 6 X 4 M E 1 N O A O W u A D A

C o f H z S 1 k m m G P o s E Y n q h F S j 4 B J 9 w 4 3 A T q q Q x q H A x 3 D c n P u P T 6 g 0 T + S D m a Q Y x H Q o e c Q Z N V b y q 8 3 + V b V f r r g 1 d w G y T r y c V C B H q 1 / + 6 g 0 S l s U o D R N U 6 6 7 n p i a Y U m U 4 E z g r 9 T K N K W V j O s S u p Z L G q I P p 4 t g Z u b D K g E S J s i U N W a i / J 6 Y 0 1 n o S h 7 Y z p m a k V 7 2 5 + J / X z U x 0 E 0 y 5 T D O D k i 0 X R Z k g J i H z z 8 m A K 2 R G T C y h T H F 7 K 2 E j q i g z N p + S D c F b f X m d t O s 1 z 6 1 5 9 / V K 4 z a P o w h n c A 6 X 4 M E 1 N O A O W u A D A

C o f H z S 1 k m m G P o s E Y n q h F S j 4 B J 9 w 4 3 A T q q Q x q H A x 3 D c n P u P T 6 g 0 T + S D m a Q Y x H Q o e c Q Z N V b y q 8 3 + V b V f r r g 1 d w G y T r y c V C B H q 1 / + 6 g 0 S l s U o D R N U 6 6 7 n p i a Y U m U 4 E z g r 9 T K N K W V j O s S u p Z L G q I P p 4 t g Z u b D K g E S J s i U N W a i / J 6 Y 0 1 n o S h 7 Y z p m a k V 7 2 5 + J / X z U x 0 E 0 y 5 T D O D k i 0 X R Z k g J i H z z 8 m A K 2 R G T C y h T H F 7 K 2 E j q i g z N p + S D c F b f X m d t O s 1 z 6 1 5 9 / V K 4 z a P o w h n c A 6 X 4 M E 1 N O A O W u A D A

C o f H z S 1 k m m G P o s E Y n q h F S j 4 B J 9 w 4 3 A T q q Q x q H A x 3 D c n P u P T 6 g 0 T + S D m a Q Y x H Q o e c Q Z N V b y q 8 3 + V b V f r r g 1 d w G y T r y c V C B H q 1 / + 6 g 0 S l s U o D R N U 6 6 7 n p i a Y U m U 4 E z g r 9 T K N K W V j O s S u p Z L G q I P p 4 t g Z u b D K g E S J s i U N W a i / J 6 Y 0 1 n o S h 7 Y z p m a k V 7 2 5 + J / X z U x 0 E 0 y 5 T D O D k i 0 X R Z k g J i H z z 8 m A K 2 R G T C y h T H F 7 K 2 E j q i g z N p + S D c F b f X m d t O s 1 z 6 1 5 9 / V K 4 z a P o w h n c A 6 X 4 M E 1 N O A O W u A D A w 7 P 8 A p v j n R e n H f n Y 9 l a c P K Z U / g D 5 / M H d b G N x w = = < / l a t e x i t > x < l a t e x i t s h a 1 _ b a s e 6 4 = " G 4 9 X 2 k J V e h B S / V i c d Z I W w 1 N 1 g u U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 2 l h g F T O B C 9 p Y 9 2 L C 3 d 9 m d M 5 I L P 8 H G Q m N s / U V 2 / h s X u E L B l 0 z y 8 t 5 M Z u Y F i R Q G X f f b K a y t b 2 x u F b d L O 7 t 7 + w f l w 6 O 2 i V P N e I v F M t Y P A T V c C s V b K F D y h 0 R z G g W S d 4 L x 9 c z v P H J t R K z u c Z J w P 6 J D J U L B K F r p r v p U 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E V f I J D W m 6 7 k J + h n V K J j k 0 1 I v N T y h b E y H v G u p o h E 3 f j Y / d U r O r D I g Y a x t K S R z 9 f d E R i N j J l F g O y O K I 7 P s z c T / v G 6 K 4 a W f C Z W k y B V b L A p T S T A m s 7 / J Q G j O U E 4 s o U w L e y t h I 6 o p Q 5 t O y Y b g L b + 8 S t r 1 m u f W v N t 6 p X G V x 1 G E E z i F c / D g A h p w A 0 1 o A Y M h P M M r v D n S e X H e n Y 9 F a 8 H J Z 4 7 h D 5 z P H 5 1 S j V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " G 4 9 X 2 k J V e h B S / V i c d Z I W w 1 N 1 g u U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 2 l h g F T O B C 9 p Y 9 2 L C 3 d 9 m d M 5 I L P 8 H G Q m N s / U V 2 / h s X u E L B l 0 z y 8 t 5 M Z u Y F i R Q G X f f b K a y t b 2 x u F b d L O 7 t 7 + w f l w 6 O 2 i V P N e I v F M t Y P A T V c C s V b K F D y h 0 R z G g W S d 4 L x 9 c z v P H J t R K z u c Z J w P 6 J D J U L B K F r p r v p U 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E V f I J D W m 6 7 k J + h n V K J j k 0 1 I v N T y h b E y H v G u p o h E 3 f j Y / d U r O r D I g Y a x t K S R z 9 f d E R i N j J l F g O y O K I 7 P s z c T / v G 6 K 4 a W f C Z W k y B V b L A p T S T A m s 7 / J Q G j O U E 4 s o U w L e y t h I 6 o p Q 5 t O y Y b g L b + 8 S t r 1 m u f W v N t 6 p X G V x 1 G E E z i F c / D g A h p w A 0 1 o A Y M h P M M r v D n S e X H e n Y 9 F a 8 H J Z 4 7 h D 5 z P H 5 1 S j V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " G 4 9 X 2 k J V e h B S / V i c d Z I W w 1 N 1 g u U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 2 l h g F T O B C 9 p Y 9 2 L C 3 d 9 m d M 5 I L P 8 H G Q m N s / U V 2 / h s X u E L B l 0 z y 8 t 5 M Z u Y F i R Q G X f f b K a y t b 2 x u F b d L O 7 t 7 + w f l w 6 O 2 i V P N e I v F M t Y P A T V c C s V b K F D y h 0 R z G g W S d 4 L x 9 c z v P H J t R K z u c Z J w P 6 J D J U L B K F r p r v p U 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E V f I J D W m 6 7 k J + h n V K J j k 0 1 I v N T y h b E y H v G u p o h E 3 f j Y / d U r O r D I g Y a x t K S R z 9 f d E R i N j J l F g O y O K I 7 P s z c T / v G 6 K 4 a W f C Z W k y B V b L A p T S T A m s 7 / J Q G j O U E 4 s o U w L e y t h I 6 o p Q 5 t O y Y b g L b + 8 S t r 1 m u f W v N t 6 p X G V x 1 G E E z i F c / D g A h p w A 0 1 o A Y M h P M M r v D n S e X H e n Y 9 F a 8 H J Z 4 7 h D 5 z P H 5 1 S j V Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " G 4 9 X 2 k J V e h B S / V i c d Z I W w 1 N 1 g u U = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 2 l h g F T O B C 9 p Y 9 2 L C 3 d 9 m d M 5 I L P 8 H G Q m N s / U V 2 / h s X u E L B l 0 z y 8 t 5 M Z u Y F i R Q G X f f b K a y t b 2 x u F b d L O 7 t 7 + w f l w 6 O 2 i V P N e I v F M t Y P A T V c C s V b K F D y h 0 R z G g W S d 4 L x 9 c z v P H J t R K z u c Z J w P 6 J D J U L B K F r p r v p U 7 Z c r b s 2 d g 6 w S L y c V y N H s l 7 9 6 g 5 i l E V f I J D W m 6 7 k J + h n V K J j k 0 1 I v N T y h b E y H v G u p o h E 3 f j Y / d U r O r D I g Y a x t K S R z 9 f d E R i N j J l F g O y O K I 7 P s z c T / v G 6 K 4 a W f C Z W k y B V b L A p T S T A m s 7 / J Q G j O U E 4 s o U w L e y t h I 6 o p Q 5 t O y Y b g L b + 8 S t r 1 m u f W v N t 6 p X G V x 1 G E E z i F c / D g A h p w A 0 1 o A Y M h P M M r v D n S e X H e n Y 9 F a 8 H J Z 4 7 h D 5 z P H 5 1 S j V Y = < / l a t e x i t > W I < l a t e x i t s h a 1 _ b a s e 6 4 = " J F / q h y L 3 o + t F X 1 E V F Q z t a Z F M u b M = " > A A A B + n i c b V A 9 T 8 M w F H w p X 6 V 8 p T C y W L R I T F X S B c Y K F t i K R F u k N o o c 1 2 m t O k 5 k O 6 A q 9 K e w M I A Q K 7 + E j X + D 0 2 a A l p M s n e 7 e 0 z t f k H C m t O N 8 W 6 W 1 9 Y 3 N r f J 2 Z W d 3 b / / A r h 5 2 V Z x K Q j s k 5 r G 8 D 7 C i n A n a 0 U x z e p 9 I i q O A 0 1 4 w u c r 9 3 g O V i s X i T k 8 T 6 k V 4 J F j I C N Z G 8 u 1 q v e d n g w j r M c E 8 u 5 n N 6 r 5 d c x r O H G i V u A W p Q Y G 2 b 3 8 N h j F J I y o 0 4 V i p v u s k 2 s u w 1 I x w O q s M U k U T T C Z 4 R P u G C h x R 5 W X z 6 D N 0 a p Q h C m N p n t B o r v 7 e y H C k 1 D Q K z G Q e U i 1 7 u f i f 1 0 9 1 e O F l T C S p p o I s D o U p R z p G e Q 9 o y C Q l m k 8 N w U Q y k x W R M Z a Y a N N W x Z T g L n 9 5 l X S b D d d p u L f N W u u y q K M M x 3 A C Z + D C O b T g G t r Q A Q K P 8 A y v 8 G Y 9 W S / W u / W x G C 1 Z x c 4 R / I H 1 + Q P n A J P A < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " J F / q h y L 3 o + t F X 1 E V F Q z t a Z F M u b M = " > A A A B + n i c b V A 9 T 8 M w F H w p X 6 V 8 p T C y W L R I T F X S B c Y K F t i K R F u k N o o c 1 2 m t O k 5 k O 6 A q 9 K e w M I A Q K 7 + E j X + D 0 2 a A l p M s n e 7 e 0 z t f k H C m t O N 8 W 6 W 1 9 Y 3 N r f J 2 Z W d 3 b / / A r h 5 2 V Z x K Q j s k 5 r G 8 D 7 C i n A n a 0 U x z e p 9 I i q O A 0 1 4 w u c r 9 3 g O V i s X i T k 8 T 6 k V 4 J F j I C N Z G 8 u 1 q v e d n g w j r M c E 8 u 5 n N 6 r 5 d c x r O H G i V u A W p Q Y G 2 b 3 8 N h j F J I y o 0 4 V i p v u s k 2 s u w 1 I x w O q s M U k U T T C Z 4 R P u G C h x R 5 W X z 6 D N 0 a p Q h C m N p n t B o r v 7 e y H C k 1 D Q K z G Q e U i 1 7 u f i f 1 0 9 1 e O F l T C S p p o I s D o U p R z p G e Q 9 o y C Q l m k 8 N w U Q y k x W R M Z a Y a N N W x Z T g L n 9 5 l X S b D d d p u L f N W u u y q K M M x 3 A C Z + D C O b T g G t r Q A Q K P 8 A y v 8 G Y 9 W S / W u / W x G C 1 Z x c 4 R / I H 1 + Q P n A J P A < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " J F / q h y L 3 o + t F X 1 E V F Q z t a Z F M u b M = " > A A A B + n i c b V A 9 T 8 M w F H w p X 6 V 8 p T C y W L R I T F X S B c Y K F t i K R F u k N o o c 1 2 m t O k 5 k O 6 A q 9 K e w M I A Q K 7 + E j X + D 0 2 a A l p M s n e 7 e 0 z t f k H C m t O N 8 W 6 W 1 9 Y 3 N r f J 2 Z W d 3 b / / A r h 5 2 V Z x K Q j s k 5 r G 8 D 7 C i n A n a 0 U x z e p 9 I i q O A 0 1 4 w u c r 9 3 g O V i s X i T k 8 T 6 k V 4 J F j I C N Z G 8 u 1 q v e d n g w j r M c E 8 u 5 n N 6 r 5 d c x r O H G i V u A W p Q Y G 2 b 3 8 N h j F J I y o 0 4 V i p v u s k 2 s u w 1 I x w O q s M U k U T T C Z 4 R P u G C h x R 5 W X z 6 D N 0 a p Q h C m N p n t B o r v 7 e y H C k 1 D Q K z G Q e U i 1 7 u f i f 1 0 9 1 e O F l T C S p p o I s D o U p R z p G e Q 9 o y C Q l m k 8 N w U Q y k x W R M Z a Y a N N W x Z T g L n 9 5 l X S b D d d p u L f N W u u y q K M M x 3 A C Z + D C O b T g G t r Q A Q K P 8 A y v 8 G Y 9 W S / W u / W x G C 1 Z x c 4 R / I H 1 + Q P n A J P A < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " J F / q h y L 3 o + t F X 1 E V F Q z t a Z F M u b M = " > A A A B + n i c b V A 9 T 8 M w F H w p X 6 V 8 p T C y W L R I T F X S B c Y K F t i K R F u k N o o c 1 2 m t O k 5 k O 6 A q 9 K e w M I A Q K 7 + E j X + D 0 2 a A l p M s n e 7 e 0 z t f k H C m t O N 8 W 6 W 1 9 Y 3 N r f J 2 Z W d 3 b / / A r h 5 2 V Z x K Q j s k 5 r G 8 D 7 C i n A n a 0 U x z e p 9 I i q O A 0 1 4 w u c r 9 3 g O V i s X i T k 8 T 6 k V 4 J F j I C N Z G 8 u 1 q v e d n g w j r M c E 8 u 5 n N 6 r 5 d c x r O H G i V u A W p Q Y G 2 b 3 8 N h j F J I y o 0 4 V i p v u s k 2 s u w 1 I x w O q s M U k U T T C Z 4 R P u G C h x R 5 W X z 6 D N 0 a p Q h C m N p n t B o r v 7 e y H C k 1 D Q K z G Q e U i 1 7 u f i f 1 0 9 1 e O F l T C S p p o I s D o U p R z p G e Q 9 o y C Q l m k 8 N w U Q y k x W R M Z a Y a N N W x Z T g L n 9 5 l X S b D d d p u L f N W u u y q K M M x 3 A C Z + D C O b T g G t r Q A Q K P 8 A y v 8 G Y 9 W S / W u / W x G C 1 Z x c 4 R / I H 1 + Q P n A J P A < / l a t e x i t > = ) < l a t e x i t s h a 1 _ b a s e 6 4 = " v C K c / 6 h B H k Q X g f n 2 B g 3 R p l K l H z A = " > A A A B + n i c b V B N T 8 J A E J 3 i F + J X 0 a O X R j D x R F o u e i R 6 8 e A B E / l I o C H b Z V s 2 b H e b 3 a 2 E V H 6 K F w 8 a 4 9 V f 4 s 1 / 4 w I 9 K P i S S V 7 e m 8 n M v C B h V G n X / b Y K G 5 t b 2 z v F 3 d L e / s H h k V 0 + b i u R S k x a W D A h u w F S h F F O W p p q R r q J J C g O G O k E 4 5 u 5 3 3 k k U l H B H / Q 0 I X 6 M I k 5 D i p E 2 0 s A u V / t 3 g k e S R i O N p B S T 6 s C u u D V 3 A W e d e D m p Q I 7 m w P 7 q D w V O Y 8 I 1 Z k i p n u c m 2 s + Q 1 B Q z M i v 1 U 0 U S h M c o I j 1 D O Y q J 8 r P F 6 T P n 3 C h D J x T S F N f O Q v 0 9 k a F Y q W k c m M 4 Y 6 Z F a 9 e b i f 1 4 v 1 e G V n 1 G e p J p w v F w U p s z R w p n n 4 A y p J F i z q S E I S 2 p u d f A I S Y S 1 S a t k Q v B W X 1 4 n 7 X r N c 2 v e f b 3 S u M 7 j K M I p n M E F e H A J D b i F J r Q A w w S e 4 R X e r C f r x X q 3 P p a t B S u f O Y E / s D 5 / A P 9 N k 9 A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " v C K c / 6 h B H k Q X g f n 2 B g 3 R p l K l H z A = " > A A A B + n i c b V B N T 8 J A E J 3 i F + J X 0 a O X R j D x R F o u e i R 6 8 e A B E / l I o C H b Z V s 2 b H e b 3 a 2 E V H 6 K F w 8 a 4 9 V f 4 s 1 / 4 w I 9 K P i S S V 7 e m 8 n M v C B h V G n X / b Y K G 5 t b 2 z v F 3 d L e / s H h k V 0 + b i u R S k x a W D A h u w F S h F F O W p p q R r q J J C g O G O k E 4 5 u 5 3 3 k k U l H B H / Q 0 I X 6 M I k 5 D i p E 2 0 s A u V / t 3 g k e S R i O N p B S T 6 s C u u D V 3 A W e d e D m p Q I 7 m w P 7 q D w V O Y 8 I 1 Z k i p n u c m 2 s + Q 1 B Q z M i v 1 U 0 U S h M c o I j 1 D O Y q J 8 r P F 6 T P n 3 C h D J x T S F N f O Q v 0 9 k a F Y q W k c m M 4 Y 6 Z F a 9 e b i f 1 4 v 1 e G V n 1 G e p J p w v F w U p s z R w p n n 4 A y p J F i z q S E I S 2 p u d f A I S Y S 1 S a t k Q v B W X 1 4 n 7 X r N c 2 v e f b 3 S u M 7 j K M I p n M E F e H A J D b i F J r Q A w w S e 4 R X e r C f r x X q 3 P p a t B S u f O Y E / s D 5 / A P 9 N k 9 A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " v C K c / 6 h B H k Q X g f n 2 B g 3 R p l K l H z A = " > A A A B + n i c b V B N T 8 J A E J 3 i F + J X 0 a O X R j D x R F o u e i R 6 8 e A B E / l I o C H b Z V s 2 b H e b 3 a 2 E V H 6 K F w 8 a 4 9 V f 4 s 1 / 4 w I 9 K P i S S V 7 e m 8 n M v C B h V G n X / b Y K G 5 t b 2 z v F 3 d L e / s H h k V 0 + b i u R S k x a W D A h u w F S h F F O W p p q R r q J J C g O G O k E 4 5 u 5 3 3 k k U l H B H / Q 0 I X 6 M I k 5 D i p E 2 0 s A u V / t 3 g k e S R i O N p B S T 6 s C u u D V 3 A W e d e D m p Q I 7 m w P 7 q D w V O Y 8 I 1 Z k i p n u c m 2 s + Q 1 B Q z M i v 1 U 0 U S h M c o I j 1 D O Y q J 8 r P F 6 T P n 3 C h D J x T S F N f O Q v 0 9 k a F Y q W k c m M 4 Y 6 Z F a 9 e b i f 1 4 v 1 e G V n 1 G e p J p w v F w U p s z R w p n n 4 A y p J F i z q S E I S 2 p u d f A I S Y S 1 S a t k Q v B W X 1 4 n 7 X r N c 2 v e f b 3 S u M 7 j K M I p n M E F e H A J D b i F J r Q A w w S e 4 R X e r C f r x X q 3 P p a t B S u f O Y E / s D 5 / A P 9 N k 9 A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " v C K c / 6 h B H k Q X g f n 2 B g 3 R p l K l H z A = " > A A A B + n i c b V B N T 8 J A E J 3 i F + J X 0 a O X R j D x R F o u e i R 6 8 e A B E / l I o C H b Z V s 2 b H e b 3 a 2 E V H 6 K F w 8 a 4 9 V f 4 s 1 / 4 w I 9 K P i S S V 7 e m 8 n M v C B h V G n X / b Y K G 5 t b 2 z v F 3 d L e / s H h k V 0 + b i u R S k x a W D A h u w F S h F F O W p p q R r q J J C g O G O k E 4 5 u 5 3 3 k k U l H B H / Q 0 I X 6 M I k 5 D i p E 2 0 s A u V / t 3 g k e S R i O N p B S T 6 s C u u D V 3 A W e d e D m p Q I 7 m w P 7 q D w V O Y 8 I 1 Z k i p n u c m 2 s + Q 1 B Q z M i v 1 U 0 U S h M c o I j 1 D O Y q J 8 r P F 6 T P n 3 C h D J x T S F N f O Q v 0 9 k a F Y q W k c m M 4 Y 6 Z F a 9 e b i f 1 4 v 1 e G V n 1 G e p J p w v F w U p s z R w p n n 4 A y p J F i z q S E I S 2 p u d f A I S Y S 1 S a t k Q v B W X 1 4 n 7 X r N c 2 v e f b 3 S u M 7 j K M I p n M E F e H A J D b i F J r Q A w w S e 4 R X e r C f r x X q 3 P p a t B S u f O Y E / s D 5 / A P 9 N k 9 A = < / l a t e x i t > Average Pooling < l a t e x i t s h a _ b a s e = " j k k U w N y z z z O v l B H O E J H T c = " > A A A B + H i c b V D L S g M x F L T X U + O u r S T b A I r s p M E X R Z c e O y g n A O R M m m l D M m Q Z I Q E v c u F D E r Z / i z r x n c C W w E D u f c V Y c K a N p Y N r e d m l b / / g s O o e H X e T B W h b S K V L Q a q Z o G D D K e R F E c h x w + n t w u + U q W Z F A m l t A g x m P B I k a w s d L Q r d Y F p a k l p Z y H b s r e z n Q O v E L U o M C r a H N R h J k s Z U G M K x n f S y Q Y W U Y X R e G a S a J p h M Y + p Q L H V A d Z f v g c n V t l h C K p B M G e r v j g z H W s / i F b G E z q r c Q / / P q Y m u g y J J D V U k O W i K O X I S L R I A Y Y o s T w m S W Y K G Z v R W S C F S b G Z l W x I f i r X n n U b d + r + f a P W v C z i K M M p n M E F + H A F T b i D F r S B Q A r P A p v z p P z r w H v S k l P n M A f O J / c G S A = = < / l a t e x i t > < l a t e x i t s h a _ b a s e = " j k k U w N y z z z O v l B H O E J H T c = " > A A A B + H i c b V D L S g M x F L T X U + O u r S T b A I r s p M E X R Z c e O y g n A O R M m m l D M m Q Z I Q E v c u F D E r Z / i z r x n c C W w E D u f c V Y c K a N p Y N r e d m l b / / g s O o e H X e T B W h b S K V L Q a q Z o G D D K e R F E c h x w + n t w u + U q W Z F A m l t A g x m P B I k a w s d L Q r d Y F p a k l p Z y H b s r e z n Q O v E L U o M C r a H N R h J k s Z U G M K x n f S y Q Y W U Y X R e G a S a J p h M Y + p Q L H V A d Z f v g c n V t l h C K p B M G e r v j g z H W s / i F b G E z q r c Q / / P q Y m u g y J J D V U k O W i K O X I S L R I A Y Y o s T w m S W Y K G Z v R W S C F S b G Z l W x I f i r X n n U b d + r + f a P W v C z i K M M p n M E F + H A F T b i D F r S B Q A r P A p v z p P z r w H v S k l P n M A f O J / c G S A = = < / l a t e x i t > < l a t e x i t s h a _ b a s e = " j k k U w N y z z z O v l B H O E J H T c = " > A A A B + H i c b V D L S g M x F L T X U + O u r S T b A I r s p M E X R Z c e O y g n A O R M m m l D M m Q Z I Q E v c u F D E r Z / i z r x n c C W w E D u f c V Y c K a N p Y N r e d m l b / / g s O o e H X e T B W h b S K V L Q a q Z o G D D K e R F E c h x w + n t w u + U q W Z F A m l t A

Figure 5 : Illustration of input x ∈ R 6 convolving with 3-channel 3-dimensional convolutional filters I 1 , I 2 , and I 3 , followed by an average pooling.

Here we denote the circulant-like matrix generated by I i as Given a sample (x t , z t ) T t=1 , the Conv RNNs compute h t and y t as follows, h t = σ h (U * h t-1 + W * x t ) , and y t = σ y (V * h t ) , where h t , x t ∈ R d , and U, V, W ∈ R k×k are matrices with column vectors being k-dimensional convolutional filters. We use zero-padding to ensure the output dimension of convolutional filters matches the input (Krizhevsky et al., 2012) . To get y t , we convolve h t with V followed by an average pooling to reduce the dimension to K. Since we aim to show that Conv RNNs reduce the dependence on d in generalization through parameter sharing, we simplify the notations to assume h 0 = 0, and impose the following assumption. Extensions to general settings are straightforward. Assumption 8. The activation operators σ h and σ y are 1-Lipschitz with σ h (0) = σ y (0) = 0. σ h is entrywise bounded by 1. The convolutional filters U, V, and W are orthogonal with normalized columns, i.e., U U = UU = 1 k I k , V V = VV = 1 k I k , and W W = WW = 1 k I k .

EQUATION

We remark that the orthogonality constraints enhance the diversity among convolutional filters (Xie et al., 2017; Huang et al., 2017) . Additionally, the normalization factor 1 k is to control the spectral norms of W U , W V , and W W , which prevents the blowup of hidden state. Denote by F c,t the class of mappings from the first t inputs to the t-th output computed by Conv RNNs. Then the generalization bound is given in the following theorem. Theorem 6. Let activation operators σ h and σ y be given, and Assumptions 1 and hold. Then for (x t , z t ) T t=1 and S = (x i,t , z i,t ) T t=1 , i = 1, . . . , m drawn i.i.d. from any underlying distribution over R d×T × {1, . . . , K}, with probability at least 1δ over S, for every margin value γ > 0 and every f t ∈ F c,t for integer t ≤ T , we have

P ( z t = z t ) ≤ R γ (f t ) + O   B x kt log (dt √ m) √ mγ + log 1 δ m   .

The detailed proof is provided in D.1. Similar to the analysis of vanilla RNNs, our proof is based on the Lipschitz continuity of Conv RNNs with respect to its model parameters in the convolutional filters. Specifically, by Assumption 8, the spectral norms of W U , W V , and W W are all bounded by 1. Combining with the inequality, W U F ≤ √ d U F , we have y ty t 2 ≤ L V,t V -V F + L U ,t U -U F + L W,t W -W F , where L U ,t , L V,t , and L W,t are polynomials in d and t. Additionally, observe that the total number of parameters in a Conv RNN is at most 3k 2 , which is independent of input dimension d. As a consequence, the generalization bound of Conv RNNs only has a lieanr dependence on k and t.

D.1 PROOF OF THEOREM 6:

Proof. We first characterize the Lipschitz continuity of y t 2 with respect to model parameters U, W and V. We have

y t -y t 2 ≤ ρ y h t 2 W V -W V 2 + ρ y W V 2 h t -h t 2 .

Since h t ∞ ≤ 1, we have h t 2 ≤ √ d. Then we expand h th t , h th t 2 ≤ ρ h U * h t-1 + W * x t -U * h t-1 -W * x t 2 = ρ h P W U h t-1 + P W W x t -P W U h t-1 -P W W x t 2

≤ ρ h P 2 W U h t-1 + W W x t -W U h t-1 -W W x t 2 ≤ ρ h P 2 B x W W -W W 2 + √ d W U -W U 2 + W U 2 h t-1 -h t-1 2

Observe that we have by the definition of circulant matrix,

W U -W U 2 2 ≤ W U -W U 2 F = (d -k) U -U 2 F ≤ d U -U 2 F .

The same holds for W W -W W and W V -W V . We also have P 2 = 1. The remaining task is to bound the spectral norm of W U and W V . Consider the matrix product W U W U . We claim that the diagonal elements of W U W U is bounded by Thus, the spectral norm W U 2 ≤ k i=1 U i 2 2 ≤ 1, and W V 2 , W W 2 ≤ 1 also hold. Then we can derive

h t -h t 2 ≤ ρ h B x √ d W -W F + ρ h d U -U F + ρ h h t-1 -h t-1 2 .

Apply the above inequality recursively, we get

h t -h t 2 ≤ ρ h B x √ d ρ t h -1 ρ h -1 W -W F + ρ h d ρ t h -1 ρ h -1 U -U F ≤ B x √ dt W -W F + dt U -U F .

Thus, we have the following Lipschitz continuity of y t 2 ,

y t -y t 2 ≤ d V -V F + B x √ dt W -W F + dt U -U F .

We also bound the norm of h t by induction. Specifically, we have

h t 2 ≤ ρ h P W U h t-1 + P W W x t 2 ≤ ρ h W U h t-1 2 + ρ h W W x t 2 ≤ h t-1 2 + B x .

Applying the above expression recursively, we have h t 2 ≤ min{ √ d, B x t} ≤ B x t. Then following the same argument for proving the generalization bound of vanilla RNNs, we can get the generalization bound for Conv RNNs as

P ( z t = z t ) ≤ R γ (f t ) + O   B x kt log (dt √ m) √ mγ + log 1 δ m   .

:

To ease the discussion, we assume U

does not scale with t. Therefore, U 2 < 1 is equivalent to U 2 ≤ 1 -∆ for a constant ∆ > 0. A precise statement can be found following Theorem 2.

For simplicity, we assume again that β does not scale with t. By β < 1, we mean β ≤ 1 -∆ for a constant ∆ > 1. A similar characterization applies to β > 1.

Figures and Tables:

Figure 1: A basic building block of vanilla RNNs

Figure 4: A comparison between generalization bounds for the same vanilla RNN trained on the wikitext dataset. The vertical axis is the logarithmic scale of the corresponding bounds.

a 9 e b i f 1 4 v 1 e G V n 1 G e p J p w v F w U p s z R w p n n 4 A y p J F i z q S E I S 2 p u d f A I S Y S 1 S a t k Q v B W X 1 4 n 7 X r N c 2 v e f b 3 S u M 7 j K M I p n M E F e H A J D b i F J r Q A w w S e 4 R X e r C f r x X q 3 P p a t B S u f O Y E / s D 5 / A P 9 N k 9 A = < / l a t e x i t > C 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " q 9 T S b Q Ox n C z o o 9 k G x p o d j 5 + G 2 V c = " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 0 l p h 4 Q A I X s r f M w Y a 9 v c v u n g k h / A Y b C 4 2 x 9 Q f Z + W 9 c 4 A o F X z L J y 3 s z m Z k X p o J r 4 7 r f T m F r e 2 d 3 r 7 h f O j g 8 O j 4 p n 5 6 1 d Z I p h j 5 L R K K 6 I d U o u E T f c C O w m y q k c S i w E 0 6 a C 7 / z h E r z R D 6 a a Y p B T E e S R 5 x R Y y W / 2 h x 4 1 U G 5 4 t b c J c g m 8 X J S g R y t Q f m r P 0 x Y F q M 0 T F C t e 5 6 b m m B G l e F M 4 L z U z z S m l E 3 o C H u W S h q j D m b L Y + f k y i p D E i X K l j R k q f 6 e m N F Y 6 2 k c 2 s 6 Y m r F e 9 x b i f 1 4 v M 9 F t M O M y z Q x K t l o U Z Y K Y h C w + J 0 O u k B k x t Y Q y x e 2 t h I 2 p o s z Y f E o 2 B G / 9 5 U 3 S r t c 8 t + Y 9 1 C u N u z y O I l z A J V y D B z f Q g H t o g Q 8 M O D z D K 7 w 5 0 n l x 3 p 2 P V W v B y W f O 4 Q + c z x 9 y p 4 3 F < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " q 9 T S b Q O x n C z o o 9 k G x p o d j 5 + G 2 V c = " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 0 l p h 4 Q A I X s r f M w Y a 9 v c v u n g k h / A Y b C 4 2 x 9 Q f Z + W 9 c 4 A o F X z L J y 3 s z m Z k X p o J r 4 7 r f T m F r e 2 d 3 r 7 h f O j g 8 O j 4 p n 5 6 1 d Z I p h j 5 L R K K 6 I d U o u E T f c C O w m y q k c S i w E 0 6 a C 7 / z h E r z R D 6 a a Y p B T E e S R 5 x R Y y W / 2 h x 4 1 U G 5 4 t b c J c g m 8 X J S g R y t Q f m r P 0 x Y F q M 0 T F C t e 5 6 b m m B G l e F M 4 L z U z z S m l E 3 o C H u W S h q j D m b L Y + f k y i p D E i X K l j R k q f 6 e m N F Y 6 2 k c 2 s 6 Y m r F e 9 x b i f 1 4 v M 9 F t M O M y z Q x K t l o U Z Y K Y h C w + J 0 O u k B k x t Y Q y x e 2 t h I 2 p o s z Y f E o 2 B G / 9 5 U 3 S r t c 8 t + Y 9 1 C u N u z y O I l z A J V y D B z f Q g H t o g Q 8 M O D z D K 7 w 5 0 n l x 3 p 2 P V W v B y W f O 4 Q + cz x 9 y p 4 3 F < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " q 9 T S b Q Ox n C z o o 9 k G x p o d j 5 + G 2 V c = " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 0 l p h 4 Q A I X s r f M w Y a 9 v c v u n g k h / A Y b C 4 2 x 9 Q f Z + W 9 c 4 A o F X z L J y 3 s z m Z k X p o J r 4 7 r f T m F r e 2 d 3 r 7 h f O j g 8 O j 4 p n 5 6 1 d Z I p h j 5 L R K K 6 I d U o u E T f c C O w m y q k c S i w E 0 6 a C 7 / z h E r z R D 6 a a Y p B T E e S R 5 x R Y y W / 2 h x 4 1 U G 5 4 t b c J c g m 8 X J S g R y t Q f m r P 0 x Y F q M 0 T F C t e 5 6 b m m B G l e F M 4 L z U z z S m l E 3 o C H u W S h q j D m b L Y + f k y i p D E i X K l j R k q f 6 e m N F Y 6 2 k c 2 s 6 Y m r F e 9 x b i f 1 4 v M 9 F t M O M y z Q x K t l o U Z Y K Y h C w + J 0 O u k B k x t Y Q y x e 2 t h I 2 p o s z Y f E o 2 B G / 9 5 U 3 S r t c 8 t + Y 9 1 C u N u z y O I l z A J V y D B z f Q g H t o g Q 8 M O D z D K 7 w 5 0 n l x 3 p 2 P V W v B y W f O 4 Q + cz x 9 y p 4 3 F < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " q 9 T S b Q Ox n C z o o 9 k G x p o d j 5 + G 2 V c = " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 0 l p h 4 Q A I X s r f M w Y a 9 v c v u n g k h / A Y b C 4 2 x 9 Q f Z + W 9 c 4 A o F X z L J y 3 s z m Z k X p o J r 4 7 r f T m F r e 2 d 3 r 7 h f O j g 8 O j 4 p n 5 6 1 d Z I p h j 5 L R K K 6 I d U o u E T f c C O w m y q k c S i w E 0 6 a C 7 / z h E r z R D 6 a a Y p B T E e S R 5 x R Y y W / 2 h x 4 1 U G 5 4 t b c J c g m 8 X J S g R y t Q f m r P 0 x Y F q M 0 T F C t e 5 6 b m m B G l e F M 4 L z U z z S m l E 3 o C H u W S h q j D m b L Y + f k y i p D E i X K l j R k q f 6 e m N F Y 6 2 k c 2 s 6 Y m r F e 9 x b i f 1 4 v M 9 F t M O M y z Q x K t l o U Z Y K Y h C w + J 0 O u k B k x t Y Q y xe 2 t h I 2 p o s z Y f E o 2 B G / 9 5 U 3 S r t c 8 t + Y 9 1 C u N u z y O I l z A J V y D B z f Q g H t o g Q 8 M O D z D K 7 w 5 0 n l x 3 p 2 P V W v B y W f O 4 Q + c z x 9 y p 4 3 F < / l a t e x i t > C 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " 0 u z e f C 9 E + M d s M H f x o y 0 6 h u L M d 9 s

3 h z p P P i v D s f q 9 a C k 8 + c w x 8 4 n z 9 0 L I 3 G < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 0 u z e f C 9 E + M d s M H f x o y 0 6 h u L M d 9 s

3 h z p P P i v D s f q 9 a C k 8 + c w x 8 4 n z 9 0 L I 3 G < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 0 u z e f C 9 E + M d s M H f x o y 0 6 h u L M d 9 s

3 h z p P P i v D s f q 9 a C k 8 + c w x 8 4 n z 9 0 L I 3 G < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 0 u z e f C 9 E + M d s M H f x o y 0 6 h u L M d 9 s

w 7 P 8 A p v j n R e n H f n Y 9 l a c P K Z U / g D 5 / M H d b G N x w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 8 p n I 6 2 g / O W E 2 R 3 y z l b m X z g 5 Q R n k = " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s s t C T S W G L i I Q l c y N 4 y B x v 2 9 i 6 7 e y a E 8 B t s L D T G 1 h 9 k 5 7 9 x g S s U f M k k L + / N Z G Z e m Aq u j e t + O 4 W N z a 3 t n e J u a W / / 4 P

w 7 P 8 A p v j n R e n H f n Y 9 l a c P K Z U / g D 5 / M H d b G N x w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 8 p n I 6 2 g / O W E 2 R 3 y z l b m X z g 5 Q R n k = " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s s t C T S W G L i I Q l c y N 4 y B x v 2 9 i 6 7 e y a E 8 B t s L D T G 1 h 9 k 5 7 9 x g S s U f M k k L + / N Z G Z e m Aq u j e t + O 4 W N z a 3 t n e J u a W / / 4 P

w 7 P 8 A p v j n R e n H f n Y 9 l a c P K Z U / g D 5 / M H d b G N x w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 8 p n I 6 2 g / O W E 2 R 3 y z l b m X z g 5 Q R n k = " > A A A B 7 H i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s s t C T S W G L i I Q l c y N 4 y B x v 2 9 i 6 7 e y a E 8 B t s L D T G 1 h 9 k 5 7 9 x g S s U f M k k L + / N Z G Z e m Aq u j e t + O 4 W N z a 3 t n e J u a W / / 4 P

g x m P B I k a w s d L Q r d Y F p a k l p Z y H b s r e z n Q Ov E L U o M C r a H N R h J k s Z U G M K x n f S y Q Y W U Y X R e G a S a J p h M Y + p Q L H V A d Z f v g c n V t l h C K p B M G e r v j g z H W s / i F b G E z q r c Q / / P q Y m u g y J J D V U k O W i K O X I S L R I A Y Y o s T w m S W Y K G Z v R W S C F S b G Z l W x I f i r X n n U b d + r + f a P W v C z i K M M p n M E F + H A F T b i D Fr S B Q A r P A p v z p P z r w H v S k l P n M A f O J / c G S A = = < / l a t e x i t > < l a t e x i t s h a _ b a s e = " j k k U w N y z zz O v l B H O E J H T c = " > A A A B + H i c b V D L S g M x F L T X U + O u r S T b A I r s p M E X R Z c e O y g n A O R M m m l D M m Q Z I Q E v c u F D E r Z / i z r x n c C W w E D u f c V Y c K a N p Y N r e d m l b / / g s O o e H X e T B W h b S K V L Q a q Z o G D D K e R F E c h x w + n t w u + U q W Z F A m l t A g x m P B I k a w s d L Q r d Y F p a k l p Z y H b s r e z n Q O v E L U o M C r a H N R h J k s Z U G M K x n f S y Q Y W U Y X R e G a S a J p h M Y + p Q L H V A d Z f v g c n V t l h C K p B M G e r v j g z H W s / i F b G E z q r c Q / / P q Y m u g y J J D V U k O W i K O X I S L R I A Y Y o s T w m S W Y K G Z v R W S C F S b G Z l W x I f i r X n n U b d + r + f a P W v C z i K M M p n M E F + H A F T b i D F r S B Q A r P A p v z p P z r w H v S k l P n M A f O J / c G S A = = < / l a t e x i t > P < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 o M 9 L Z W w 0 N 5 B O D N D 2 5 N m H o P c w j 8 = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 2 l h j l I 4 E L 2 V v 2 Y M P e 3 m V 3 z o R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S K Q w 6 L r f T m F j c 2 t 7 p 7 h b 2 t s / O D w q H 5 + 0 T Z x q x l s s l r H u B t R w K R R v o U D J u 4 n m N A o k 7 w S T 2 7 n f e e L a i F g 9 4 j T h f k R H S o S C U b T S Q 7 V Z H Z Q r b s 1 d g K w T L y c V y N E c l L / 6 w 5 i l E V f I J D W m 5 7 k J + h n V K J j k s 1 I / N T y h b E J H v G e p o h E 3 f r Y 4 d U Y u r D I k Y a x t K S Q L 9 f d E R i N j p l F g O y O K Y 7 P q z c X / v F 6 K 4 b W f C Z W k y B V b L g p T S T A m 8 7 / J U G j O U E 4 t o U w L e y t h Y 6 o p Q 5 t O y Y b g r b 6 8 T t r 1 m u f W v P t 6 p X G T x 1 G E M z i H S / D g C h p w B 0 1 o A Y M R P M M r v D n S e X H e n Y 9 l a 8 H J Z 0 7 h D 5 z P H 2 C K j S 4 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 o M 9 L Z W w 0 N 5 B O D N D 2 5 N m H o P c w j 8 = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 2 l h j l I 4 E L 2 V v 2 Y M P e 3 m V 3 z o R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S K Q w 6 L r f T m F j c 2 t 7 p 7 h b 2 t s / O D w q H 5 + 0 T Z x q x l s s l r H u B t R w K R R v o U D J u 4 n m N A o k 7 w S T 2 7 n f e e L a i F g 9 4 j T h f k R H S o S C U b T S Q 7 V Z H Z Q r b s 1 d g K w T L y c V y N E c l L / 6 w 5 i l E V f I J D W m 5 7 k J + h n V K J j k s 1 I / N T y h b E J H v G e p o h E 3 f r Y 4 d U Y u r D I k Y a x t K S Q L 9 f d E R i N j p l F g O y O K Y 7 P q z c X / v F 6 K 4 b W f C Z W k y B V b L g p T S T A m 8 7 / J U G j O U E 4 t o U w L e y t h Y 6 o p Q 5 t O y Y b g r b 6 8 T t r 1 m u f W v P t 6 p X G T x 1 G E M z i H S / D g C h p w B 0 1 o A Y M R P M M r v D n S e X H e n Y 9 l a 8 H J Z 0 7 h D 5 z P H 2 C K j S 4 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 o M 9 L Z W w 0 N 5 B O D N D 2 5 N m H o P c w j 8 = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 2 l h j l I 4 E L 2 V v 2 Y M P e 3 m V 3 z o R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S K Q w 6 L r f T m F j c 2 t 7 p 7 h b 2 t s / O D w q H 5 + 0 T Z x q x l s s l r H u B t R w K R R v o U D J u 4 n m N A o k 7 w S T 2 7 n f e e L a i F g 9 4 j T h f k R H S o S C U b T S Q 7 V Z H Z Q r b s 1 d g K w T L y c V y N E c l L / 6 w 5 i l E V f I J D W m 5 7 k J + h n V K J j k s 1 I / N T y h b E J H v G e p o h E 3 f r Y 4 d U Y u r D I k Y a x t K S Q L 9 f d E R i N j p l F g O y O K Y 7 P q z c X / v F 6 K 4 b W f C Z W k y B V b L g p T S T A m 8 7 / J U G j O U E 4 t o U w L e y t h Y 6 o p Q 5 t O y Y b g r b 6 8 T t r 1 m u f W v P t 6 p X G T x 1 G E M z i H S / D g C h p w B 0 1 o A Y M R P M M r v D n S e X H e n Y 9 l a 8 H J Z 0 7 h D 5 z P H 2 C K j S 4 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 o M 9 L Z W w 0 N 5 B O D N D 2 5 N m H o P c w j 8 = " > A A A B 6 n i c b V A 9 T w J B E J 3 D L 8 Q v 1 N J m I 5 h Y k T s a L Y k 2 l h j l I 4 E L 2 V v 2 Y M P e 3 m V 3 z o R c + A k 2 F h p j 6 y + y 8 9 + 4 w B U K v m S S l / d m M j M v S K Q w 6 L r f T m F j c 2 t 7 p 7 h b 2 t s / O D w q H 5 + 0 T Z x q x l s s l r H u B t R w K R R v o U D J u 4 n m N A o k 7 w S T 2 7 n f e e L a i F g 9 4 j T h f k R H S o S C U b T S Q 7 V Z H Z Q r b s 1 d g K w T L y c V y N E c l L / 6 w 5 i l E V f I J D W m 5 7 k J + h n V K J j k s 1 I / N T y h b E J H v G e p o h E 3 f r Y 4 d U Y u r D I k Y a x t K S Q L 9 f d E R i N j p l F g O y O K Y 7 P q z c X / v F 6 K 4 b W f C Z W k y B V b L g p T S T A m 8 7 / J U G j O U E 4 t o U w L e y t h Y 6 o p Q 5 t O y Y b g r b 6 8 T t r 1 m u f W v P t 6 p X G T x 1 G E M z i H S / D g C h p w B 0 1 o A Y M R P M M r v D n S e X H e n Y 9 l a 8 H J Z 0 7 h D 5 z P H 2 C K j S 4 = < / l a t e x i t > I ⇤ x < l a t e x i t s h a 1 _ b a s e 6 4 = " z 6 1 8 f G D m a x g E 3 M b 7 z V M a L k R P K E Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V 0 g b G C B b Y i 0 Y f U R t W N 6 7 R W H S e y H U S J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H x 4 8 5 U 9 p x v q 3 C y u r a + k Z x s 7 S 1 v b O 7 Z + 8 f t F S U S E K b J O K R 7 P i g K G e C N j X T n H Z i S S H 0 O W 3 7 4 6 v M b 9 9 T q V g k 7 v Q k p l 4 I Q 8 E C R k A b q W 8 f V X o h 6 B E B n t 5 M c Q + U x g + V v l 1 2 q s 4 M e J m 4 O S m j H I 2 + / d U b R C Q J q d C E g 1 J d 1 4 m 1 l 4 L U j H A 6 L f U S R W M g Y x j S r q E C Q q q 8 d J Z + i k + N M s B B J M 0 T G s / U 3 x s p h E p N Q t 9 M Z l H V o p e J / 3 n d R A c X X s p E n G g q y P x Q k H C s I 5 x V g Q d M U q L 5 x B A g k p m s m I x A A t G m s J I p w V 3 8 8 j J p 1 a q u U 3 V v a + X 6 Z V 5 H E R 2 j E 3 S G X H S O 6 u g a N V A T E f S I n t E r e r O e r B f r 3 f q Y j x a s f O c Q / Y H 1 + Q N x Q Z S M < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " z 6 1 8 f G D m a x g E 3 M b 7 z V M a L k R P K E Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V 0 g b G C B b Y i 0 Y f U R t W N 6 7 R W H S e y H U S J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H x 4 8 5 U 9 p x v q 3 C y u r a + k Z x s 7 S 1 v b O 7 Z + 8 f t F S U S E K b J O K R 7 P i g K G e C N j X T n H Z i S S H 0 O W 3 7 4 6 v M b 9 9 T q V g k 7 v Q k p l 4 I Q 8 E C R k A b q W 8 f V X o h 6 B E B n t 5 M c Q + U x g + V v l 1 2 q s 4 M e J m 4 O S m j H I 2 + / d U b R C Q J q d C E g 1 J d 1 4 m 1 l 4 L U j H A 6 L f U S R W M g Y x j S r q E C Q q q 8 d J Z + i k + N M s B B J M 0 T G s / U 3 x s p h E p N Q t 9 M Z l H V o p e J / 3 n d R A c X X s p E n G g q y P x Q k H C s I 5 x V g Q d M U q L 5 x B A g k p m s m I x A A t G m s J I p w V 3 8 8 j J p 1 a q u U 3 V v a + X 6 Z V 5 H E R 2 j E 3 S G X H S O 6 u g a N V A T E f S I n t E r e r O e r B f r 3 f q Y j x a s f O c Q / Y H 1 + Q N x Q Z S M < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " z 6 1 8 f G D m a x g E 3 M b 7 z V M a L k R P K E Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V 0 g b G C B b Y i 0 Y f U R t W N 6 7 R W H S e y H U S J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H x 4 8 5 U 9 p x v q 3 C y u r a + k Z x s 7 S 1 v b O 7 Z + 8 f t F S U S E K b J O K R 7 P i g K G e C N j X T n H Z i S S H 0 O W 3 7 4 6 v M b 9 9 T q V g k 7 v Q k p l 4 I Q 8 E C R k A b q W 8 f V X o h 6 B E B n t 5 M c Q + U x g + V v l 1 2 q s 4 M e J m 4 O S m j H I 2 + / d U b R C Q J q d C E g 1 J d 1 4 m 1 l 4 L U j H A 6 L f U S R W M g Y x j S r q E C Q q q 8 d J Z + i k + N M s B B J M 0 T G s / U 3 x s p h E p N Q t 9 M Z l H Vo p e J / 3 n d R A c X X s p E n G g q y P x Q k H C s I 5 x V g Q d M U q L 5 x B A g k p m s m I x A A t G m s J I p w V 3 8 8 j J p 1 a q u U 3 V v a + X 6 Z V 5 H E R 2 j E 3 S G X H S O 6 u g a N V A T E f S I n t E r e r O e r B f r 3 f q Y j x a s f O c Q / Y H 1 + Q N x Q Z S M < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " z 6 1 8 f G D m a x g E 3 M b 7 z V M a L k R P K E Q = " > A A A B / X i c b V C 7 T s M w F H X K q 5 R X e G w s F i 0 S U 5 V 0 g b G C B b Y i 0 Y f U R t W N 6 7 R W H S e y H U S J K n 6 F h Q G E W P k P N v 4 G p 8 0 A L U e y d H T O v b r H x 4 8 5 U 9 p x v q 3 C y u r a + k Z x s 7 S 1 v b O 7 Z + 8 f t F S U S E K b J O K R 7 P i g K G e C N j X T n H Z i S S H 0 O W 3 7 4 6 v M b 9 9

d-k+1)×d , and write W I = [C 1 , . . . , C k ] . We further denote P = 1 k [I d-k+1 I d-k+1 • • • I d-k+1 ] totally k identity matrices , where I d denotes the d-dimensional identity matrix. Define I = [I 1 , . . . , I k ], and I * x = P W I x.

the off-diagonal elements are zero. To see this, denote by C Ui the circulant like matrix generated by U i . Then we haveW U = [C U1 , . . . , C U k ] . The diagonal elements of W U W U are W U W U ii =By the orthogonality of U, the off-diagonal elements are W U W U pq = Uj :p C Uj :q = 0.

which only has a logarithmic dependence on t; (II) When U 2 = 1, the generalization bound is O dt √ mγ , which has a linear dependence on d and t; (III) When U 2 > 1, the generalization bound is O √ mγ , which has a polynomial dependence on d and t.

Table content: <table><tr><td>√</td><td>d 3 t</td></tr></table>

, • F . Their Cartesian product gives us the covering C(F t , , dist(•, •)). The following lemma gives an upper bound on the covering number of matrices with a bounded Frobenius norm. Lemma 8. Let G = A ∈ R d1×d2 : A 2 ≤ λ be the set of matrices with bounded spectral norm and > 0 be given. The covering number N (G, , • F ) is upper bounded by

Table content: <table/>

Peer review (iclr2019):
 The paper focuses on the generalization performance of RNNs and its variant in a theoretical perspective. Compared to the previous result (Zhang et al., 2018) for RNNs, this paper refines the generalization bounds for vanilla RNNs in all cases and fills the blank for RNN variants like MGU and LSTM. Specifically, in the work of Zhang et al. (2018), the complexity term quadratically depends on the layer (or say, current sequence length, denoted by t in original paper), making it less instructive. This paper improves it to at most linear dependence and can achieve at logarithmic dependence in some cases, which should be accredited. The key step in the proof is Lemma 2. In Lemma 2, the spectral norms of weight matrices and the number of weight parameters are decoupled. With Lemma 2, it is natural to construct a epsilon-net for RNNs and then upper bound the empirical Rademacher complexity by Dudley’s entropy integral, since such methodology is not so novel. Bartlett, et al. (2017) developed this technique to analyze the generalization bound for neural networks in a margin-based multiclass classification. However, it seems a little unexplainable to apply a technique developed from classification to analyze RNNs, since the main task of RNNs never should be classification. I wonder the motivation of analyzing generalization of RNNs by the techniques established by Bartlett.