A Theory of Unsupervised Speech Recognition
Year: 2023

Abstract:
Unsupervised speech recognition (ASR-U) is the problem of learning automatic speech recognition (ASR) systems from unpaired speech-only and text-only corpora. While various algorithms exist to solve this problem, a theoretical framework is missing to study their properties and address such issues as sensitivity to hyperparameters and training instability. In this paper, we proposed a general theoretical framework to study the properties of ASR-U systems based on random matrix theory and the theory of neural tangent kernels. Such a framework allows us to prove various learnability conditions and sample complexity bounds of ASR-U. Extensive ASR-U experiments on synthetic languages with three classes of transition graphs provide strong empirical evidence for our theory (code available at cactuswiththoughts/UnsupASRTheory.git).

heading: 1 Introduction
text: Unsupervised speech recognition (ASR-U) is the problem of learning automatic speech recognition (ASR) systems from unpaired speech-only and textonly corpora. Such a system can not only significantly reduce the amount of annotation resources required for training state-of-the-art ASR system, but serve as a bridge between spoken and written language understanding tasks in the low-resource setting. Since its first proposal (Liu et al., 2018), it has seen remarkable progress and the current best system (Baevski et al., 2021) has achieved comparable performance to systems trained with paired data on various languages. However, there are several mysteries surrounding ASR-U, which potentially hinder the future development of such systems. In particular, prior experiments have found that training the current state-of-the-art ASR-U model, wav2vec-U (Baevski et al., 2021), requires careful tuning over the weights of various regularization losses to avoid converging to bad local optima and that even despite extensive regularization
weight tuning, wav2vec-U may still fail to converge (Ni et al., 2022). Therefore, it remains a mystery whether or when unpaired speech and text data indeed provide sufficient information for learning an ASR system. Another mystery is whether the success of existing ASR-U models based on generative adversarial net (GAN) (Goodfellow et al., 2014) is sufficiently explained by the GAN objective function per se, or also requires other factors, such as randomness in training, quirks in the data used and careful domain-specific hyper-parameter settings, etc.
In this paper, we provide a theoretical analysis of ASR-U to investigate the mysteries surrounding ASR-U. First, we prove learnability conditions and sample complexity bounds that crucially depend on the eigenvalue spacings of the transition probability matrix of the spoken language. Random matrix theory shows that such learnability conditions are achievable with high probability. Next, we study the gradient flow of GAN-based ASR-U and provide conditions under which the generator minimizing the GAN objective converges to the true generator. Finally, to verify our theory empirically, we perform GAN-based ASR-U experiments on three classes of synthetic languages. Not only do we observe phase transition phenomena predicted by our theory, but we achieve stable training with lower test word error rate by several modifications of the existing state-of-the-art ASR-U system inspired by our theory.

heading: 2 Problem formulation
text: General formulation The training data comprise a set of sequences of quantized speech vectors, and a set of sequences of phoneme labels. The data are unpaired: there is no label sequence that matches any one of the speech sequences. The data are, however, matched in distribution. Let PXi(x) and PYj (y) be the probability mass functions (pmfs) of the ith speech vector in a sequence, x ∈ X, and the
1192
jth phoneme in a sequence, y ∈ Y, respectively: the requirement that they are matched in distribution is the requirement that there exists some generator function O : (X,Y) → {0, 1} such that
∑ x∈X PXi(x)O(x, y) = PYi(y) (1)
The problem of ASR-U is to find the generator function O.
GAN-based ASR-U Eq. (1) leverages sequence information to remove ambiguity: O must be an optimal generator not only for the positionindependent distributions of X and Y , but also for their position-dependent distributions PXi , PYi∀i ∈ N0. In reality we cannot observe every possible sequence of speech vectors, or every possible sequence of phonemes, but instead must estimate O from samples. To address this issue, a GAN can be used to reproduce the empirical distribution of the training dataset with minimum error, subject to the generator’s inductive bias, e.g., subject to the constraint that the function O is a matrix of the form O ∈ {0, 1}|X|×|Y|, where |X| and |Y| are the sizes of the alphabets X and Y, respectively. As shown in Figure 1, a GAN achieves this goal by computing O as the output of a neural network, O = G(x, y; θ), and by requiring G to play a zerosum game with another neural network called the discriminator D with the following general utility function:
min G max D
J(G,D) := EY∼PY [a(D(Y ))]−
EX∼PX [b(D(G(X)))]. (2)
For the original GAN (Goodfellow et al., 2014), a(D) = log(σ(D)) and b(D) = − log(1−σ(D)), where σ is the sigmoid function. For the Wasserstein GAN (Arjovsky et al., 2017), D(Y ) is a
Lipschitz-continuous scalar function, and a(D) = b(D) = D. A maximum mean discrepancy (MMD) GAN (Li et al., 2017) minimizes the squred norm of Eq. (2), where D(Y ) is an embedding into a reproducing kernel Hilbert space (RKHS). In this paper we take the RKHS embedding to be the probability mass function of a scalar random variable D(Y ), and assume that the discriminator is trained well enough to maintain Eq. (2). In this situation, the MMD GAN minimizes Eq. (2) with a(D) = b(D) = Y . In practice, Eq. (2) is optimized by alternatively updating the parameters of the discriminator and the generator using gradient descent/ascent:
ϕi+1 = ϕi + η∇ϕJ(Gθi , Dϕi) (3) θi+1 = θi − ν∇θJ(Gθi , Dϕi+1). (4)
Theoretical questions of ASR-U The aforementioned formulation of ASR-U is ill-posed. Intuitively, the function O has finite degrees of freedom (O ∈ {0, 1}|X|×|Y|), while Eq. (1) must be valid for an infinite number of distributions (PXi and PYi for i ∈ N), so there is no guarantee that a solution exists. On the other hand, if the sequence is unimportant (PXi = PXj∀i, j ∈ N0), then the solution may not be unique. One important question is then: what are the necessary and sufficient conditions for Eq. (1) to have a unique solution?
Further, it is well-known that gradient-based training of GAN can be unstable and prior works on ASR-U (Yeh et al., 2019; Baevski et al., 2021) have used various regularization losses to stabilize training. Therefore, another question of practical significance is: what are the necessary and sufficient conditions for the alternate gradient method as described by Eq. (3)-(4) to converge to the true generator for ASR-U? In the subsequent sections, we set out to answer these questions.

heading: 3 Theoretical analysis of ASR-U

heading: 3.1 Learnability of ASR-U: a sufficient condition
text: A key assumption of our theory is that the distribution of the speech and the text units can be modeled by a single hidden Markov model whose hidden states are N -grams of speech units and whose outputs are N -grams of text units, as shown in Figure 1.
The parameters of this HMM are its initial probability vector, π, which specifies the distribution
of the first N speech vectors X0:(N−1) ∈ XN , its transition probability matrix A, which specifies the probability of any given sequence of N speech vectors given the preceding N speech vectors, and its observation probability matrix, which specifies the distribution of one phone symbol given one speech vector:
π := PX0:N−1 ∈ ∆|X| N A := PXN :2N−1|X0:N−1 ∈ ∆|X| N×|X|N O := PY |X ∈ ∆|X|×|Y|,
where ∆k is the k-dimensional probability simplex. The first-order Markov assumption is made plausible by the use of N-gram states, X0:N−1, rather than unigram states; with sufficiently long N , natural language may be considered to be approximately first-order Markov. The connection between the N -gram states and the unigram observations requires the use of a selector matrix, E = 1|X|N−1 ⊗ I|X|, where ⊗ denotes the Kronecker product, thus PXkN = π
⊤AkE, and for multiples of N , Eq. (1) can be written PYkN = π
⊤AkEO. It turns out that a crucial feature for a spoken language to be learnable in an unsupervised fashion is that it needs to be “complex” enough such that a simple, symmetric and repetitive graph is not sufficient to generate the language. This is captured by the following assumptions on the parameters A and π.
Assumption 1. There exists an invertible matrix U ∈ R|X|N−1×|X|N−1 = [U1|U2| · · · |UK ], where the columns of each matrix Uj = [uj1| · · · |ujNj ] are eigenvectors with the same eigenvalue and a diagonal matrix Λ = blkdiag(Λ1, · · · ,ΛK), where each matrix Λk is a diagonal matrix with all diagonal elements equal to the same scalar λk, such that A = UΛU−1 with |X|N ≥ K ≥ |X| nonzero eigenvalues λ1 > λ2 > · · · > λK . Assumption 2. For at least |X| values of j, there is at least one k s.t. π⊤ujk ̸= 0.
With Assumptions 1 and 2, we can consider the following algorithm: First, we construct the following matrices
PX :=   P⊤X0 P⊤XN
... P⊤X(L−1)N
 , P Y :=   P⊤Y0 P⊤YN
... P⊤Y(L−1)N
  , (5)
Then, O satisfies the following matrix equation
PXO = P Y . (6)
The binary matrix O in Eq. (6) is unique if and only if PX has full column rank. The following theorem proves that this is indeed the case under our assumptions.
Theorem 1. Under Assumptions 1 and 2, PX has full column rank and perfect ASR-U is possible. Further, the true phoneme assignment function is O = PX+P Y , where PX+ = (PX⊤PX)−1PX⊤ is the left-inverse of PX .
Further, if we measure how far the matrix PX is from being singular by its smallest singular value defined as
σmin(P X) := min v∈R|X| ∥PXv∥2 ∥v∥2 ,
we can see that PX becomes further and further away from being singular as the sequence length L gets larger. An equivalent result for a different purpose has appeared in the Theorem 1 of (Bazán, 2000).
Lemma 1. Under Assumptions 1 and 2 and for simplicity assuming the number of distinct eigenvalues K = |X| for T , then we have
σmin(P X) ≥
δ (|X|−1)/2|X| min ∑L−|X|−1 l=0 λ 2l min(A)
κ(V|X|(λ1:|X|)) min j
∥r̂j∥ (7)
where δmin := mini ̸=j |λi(A) − λj(A)|, λmin(A) is the smallest eigenvalue of square matrix A, κ(V|X|(λ1:|X|)) is the condition number of the square Vandermonde matrix created from eigenvalues λ1(A), . . . , λ|X|(A), rj = πTUjΩ⊤j E, and Ω ⊤ j is the set of rows of U−1 corresponding to eigenvalue λj(A), after orthogonalizing them from every other block of rows, i.e., U−1 = L[Ω1| · · · |ΩK ]T such that L is lower-triangular, and the blocks Ωi and Ωj are orthogonal.
Next, we will show that Assumption 1 can be easily met using random matrix arguments.

heading: 3.2 Finite-sample learnability of ASR-U
text: Matched setup Now we show that the requirement for distinct eigenvalues is a mild one as it can easily be satisfied with random transition matrices. According to such a result, ASR-U is feasible
with high probability in the (empirically) matched setting commonly used in the ASR-U literature, where the empirical generated and true distributions can be matched exactly by some generator in the function class (Liu et al., 2018). Our proof relies crucially on the seminal work of (Nguyen et al., 2017) on eigenvalue gaps of symmetric random matrices with independent entries.
In the context of ASR-U, it is of particular interest to study the eigenvalue gaps of a Markov random matrix, which unlike the symmetric case, is asymmetric with correlated entries. Fortunately, by modifying the proof for Theorem 2.6 of (Nguyen et al., 2017), we can show that if the language model belongs to a special but rather broad class of Markov random matrices defined below and the states are non-overlapping N -gram instead of the more common overlapping ones, it should have at least |X| distinct eigenvalues with minimum spacing depending on |X| and the N for the N -gram. Definition 1. (symmetric Markov random matrix) A symmetric Markov random matrix is a matrix of the form A := D−1W , where the adjacency matrix W is a real, symmetric random matrix with positive entries and bounded variance and D is a diagonal matrix with dii = ∑ j Wij > 0.
Intuitively, a symmetric Markov random matrix is the transition matrix for a reversible Markov chain formed by normalizing edge weights of a weighted, undirected graph.
Theorem 2. (simple spectrum of symmetric Markov random matrix) Let An = D−1n Wn ∈ Rn×n be a real symmetric Markov random matrix with adjacency matrix Wn. Further, suppose Wn = Fn + Xn, where Fn is a deterministic symmetric matrix with eigenvalues of order nγ and Xn is a symmetric random matrix of zeromean, unit variance sub-Gaussian random variables. Then we have for any C > 0, there exists B > 4γ′C + 7γ′ + 1 such that
max 1≤i≤n−1
Pr[|λi − λi+1| ≤ n−B] ≤ n−C ,
with probability at least 1 − O(exp(−α0n)) for some α0 > 0 dependent on B and γ′ = max{γ, 1/2}. Corollary 1. Suppose the speech feature transition probability is a symmetric Markov random matrix A := D−1W with entries Wij ∼ Uniform(0, 2 √ 3) and D is a diagonal matrix with
dii = ∑
j Wij . Then for any ϵ > 0, there exists α0 > 0 such that with probability at least 1−O ( |X|−CN + exp ( −α0|X|N )) , the transition probability matrix A has |X|N distinct eigenvalues with minimum gap |X|−BN > 0.
The proof of Theorem 2 and Corollary 1 are presented in detail in the Appendix A.2.
Unmatched setup In the finite-sample, unmatched setup, the empirical distribution of the fake text data generated by the GAN does not necessarily match the empirical distribution of the true text data. Assuming the discriminator is perfect in the sense that it maintains Eq. (2) non-negative, and assuming D(Y ) is a scalar random variable, then minimizing Eq. (2) is equivalent to minimizing a divergence measure d(·, ·), between the empirical text distribution, P Y , and the text distribution generated by Ox(y) = P̂Y |X(y|x):
min O∈∆|X|×|Y|
dγ(P Y , PXO), (8)
where γ > 0. For example, for the original GAN, d(·, ·) is the Jensen-Shannon distance and for the MMD GAN, d(·, ·) is the Lγ distance between the expectations E[D(Y )] under the distributions P Y and PXO. In both cases, however, Eq. (8) can be minimized using a decomposable discriminator defined to be:
EPY [a(D(Y ))] = L−1∑
l=0
EPYl [a(Dl(Yl))] (9)
EPX [b(D(G(X)))] = L−1∑
l=0
EPXl [b(Dl(Gl(X))],
(10)
with components Dl : |Y| 7→ R, l = 1, · · · , L. Under the assumption that D is decomposable and that the MMD GAN is used, we have the following sample complexity bound on perfect ASR-U.
Theorem 3. The empirical risk minimizer (ERM) of Eq. (8) recovers the true assignment O perfectly from nX speech frames and nY text characters with probability 1− 2δ if
σmin(P X) ≥
√ 4L|Y|(nX + nY ) + L|X|nX
nXnY +
10 √ L log 1δ nX ∧ nY ,
where nX ∧ nY := min{nX , nY }.

heading: 3.3 Training dynamic of GAN-based ASR-U
text: So far, we have assumed the GAN training is able to find the optimal parameters for the discriminator and the generator. However, there is no guarantee that this is indeed the case with gradient updates such as Eq. (3). To analyze the behaviour of the GAN training dynamic for ASR-U, we follow prior works on neural tangent kernel (NTK) (Jacot et al., 2018) to focus on the infinite-width, continuoustime regime, or NTK regime, where the generator and the discriminator are assumed to be neural networks with an infinite number of hidden neurons trained with gradient descent at an infinitely small learning rate. Though highly idealized, studying such a regime is practically useful as results from this regime can often be converted to finite-width, discrete-time settings (See, e.g., (Du et al., 2019)).
For simplicity, denote fτ := Dϕτ and gt := Gθt and define Lt(f) := J(gt, f), then in the NTK regime, between each generator step, the training dynamic of the discriminator can be described by the following partial differential equation (PDE):
∂τϕτ = ∇ϕτLt(fτ ). (11) Let f∗Pt = limτ→∞ fτ be the limit of Eq. (11). If the limit exists and is unique, the generator loss is well-defined as Ct(gt) := J(gt, f∗Pt). Note that the output of the ASR-U generator is discrete, which is not a differentiable function per se, but we can instead directly parameterize the generated text distribution as Pgt := PX ◦ Ot for some softmax posterior distribution Ot:
Ot,x(y) := L∏
l=1 exp(hθ,yl(xl))∑ y′l exp(hθ,y′l(xl)) , (12)
where hθ is a neural network, and is assumed to be one layer in our analysis, though it can be extended to multiple layers with slight modifications using techniques similar to those in (Du et al., 2019).
Using such a generator, the generator dynamic can be then described by the following PDE:
∂tθt = ∑
y∈YL b(f∗gt(y))∇θtPgt(y), (13)
where the right-hand side is the term in the gradient of Ct with respect to θt ignoring the dependency of the discriminator f∗gt . Define the NTKs of the discriminator and the generator (distribution) as
Kfτ (y, y ′) = Eϕ0∼W
[ ∂fτ (y)
∂ϕτ ⊤∂fτ (y′) ∂ϕτ
] (14)
Kgt(y, y ′) = Eθ0∼W
[ ∂Pgt(y)
∂θt
⊤∂Pgt(y ′)
∂θt
] ,
(15)
where W is the initialization distribution (usually Gaussian).
Note that the NTKs are |Y|L×|Y|L matrices for ASR-U due to the discrete nature of the generator. A key result in (Jacot et al., 2018) states that as the widths of the hidden layers of the discriminator and generator go to infinity, Kfτ → KD,Kgt → KG stay constant during gradient descent/ascent and we have
∂τfτ = KD (diag(PY )∇fτa −diag(Pgt)∇fτ b) , (16)
∂tPgt = KGbfgt , (17)
where ∇f{a, b} = [ ∂{a,b}(f(y))
∂f(y) ] y∈YL and bf =
(bf (y))y∈YL . However, Eq. (16)-(17) is in general highly nonlinear and it remains an open problem as to their convergence properties. Instead, we focus on the case when the discriminator ft is decomposable with components ft,l, l = 1, · · · , L, and simplify
Eq. (16) and Eq. (17) into PDEs involving only samples at a particular time step:
∂τfτ,l =KD,l ( diag(P Yl )∇fτ,lafτ,l
− diag(P gtl )∇fτ,lbfτ,l ) , (18)
∂tO ⊤ t,x =
L∑
l=1
PXl (x)KOt,xbfgt,l , (19)
for all l = 1, · · · , L, x ∈ X in terms of the stepwise NTKs defined as:
KD,l(y, y ′) := Eϕ0∼W
[ ∂fτ (y)
∂ϕτ ⊤∂fτ (y′) ∂ϕτ
]
KOt,x(y, y ′) := Eθ0∼W
[ ∂Ot,x(y)
∂θτ ⊤∂Ot,x(y′) ∂θτ
] .
We further focus on the special case that fτ,l is parameterized by a two-layer neural network with ReLU activation, though the framework can be extended to network of arbitrary depths:
fτ,l(y) = lim m→∞ 1√ m
m∑
r=1
vlr max{W lry, 0}. (20)
In this case, under mild regularity conditions, we can show that the generator trained with the alternate gradient method minimizes Eq. (8), which under the same condition as in Section 3.2, implies ASR-U is feasible.
Theorem 4. Suppose the following assumptions hold:
1. The discriminator is decomposable and parameterized by Eq. (20), whose parameters are all initialized by standard Gaussian variables;
2. The generator is linear before the softmax layer;
3. The GAN objective is MMD;
4. The linear equation PXO = P Y has at least one solution.
Then we have for any solution Ot of Eq. (19), limt→∞ PXOt = P Y .

heading: 4 Experiments
text: Synthetic language dataset To allow easy control of the eigenvalue spacings of the transition
matrix T and thus observe the phase transition phenomena predicted by our theory, we design six synthetic languages with HMM language models as follows. First, we create the HMM transition graph by treating non-overlapping bigrams as hidden states of the HMM. The hidden state of the HMM will henceforth be referred to as the “speech unit”, while the observation emitted by the HMM will be referred to as the “text unit”. For the asymptotic ASR-U, we control the number of eigenvalues of the Markov transition graph by varying the number of disjoint, identical subgraphs. The number of distinct eigenvalues of the whole graph will then be equal to the number of eigenvalues of each subgraph. For the finite sample setting, we instead select only Hamiltonian graphs and either gradually decrease the degrees of the original graph to its Hamiltonian cycle or interpolate between the graph adjacency matrix and that of its Hamiltonian cycle. Thus, we can increase σmin(PX) by increasing w. For both the subgraph in the former case and the Hamiltonian graph in the latter, we experiments with circulant, de Bruijn graphs (de Bruijn, 1946) and hypercubes, as illustrated in Figure 2. Next, we randomly permute the hidden state symbols to form the true generator mapping from the speech units to text units. To create matched speech-text data, we simply sample matched speech and text unit sequences using a single HMM. For unmatched datasets, we sample the speech and text data independently with two HMMs with the same parameters. Please refer to Appendix B for more details.
Model architecture For finite-sample ASR-U, we use wav2vec-U (Baevski et al., 2021) with several modifications. In particular, we experiment with various training objectives other than the Jensen-Shannon (JS) GAN used in the original wav2vec-U, including the Wasserstein GAN (Liu et al., 2018) and the MMD GAN. All additional regularization losses are disabled. Moreover, we experimentally manipulate two hyperparameters: (1) the averaging strategy used by the generator, and (2) whether to reset the discriminator weights to zero at the beginning of each discriminator training loop. More details can be found in Appendix B.
Phase transition of PER vs. eigenvalue gaps: asymptotic case The phoneme error rate (PER) as a function of the number of eigenvalues of A for the asymptotic ASR-U on the synthetic datasets are shown in Figure 3. For all three graphs, we observe
clear phase transitions as the number of eigenvalues exceeds the number of speech units, and an increase of the number of distinct, nonzero eigenvalues required for perfect ASR-U as the number of speech units increases.
Phase transition of PER vs. eigenvalue gaps: finite-sample case The PER as a function of the least singular value σmin(PX) for the finite-sample ASR-U on the synthetic datasets are shown in Figure 4. As we can see, the ASR-U exhibit the phase transition phenomena in all three graphs, albeit with differences in the critical point and their rate of
approaching the perfect ASR-U regime. While the PER generally decreases as σmin(PX) gets larger, we found a dip in PER in the circulant graph case as σmin(PX) moves from 10−31 to 10−15. Though unexpected, this observation is not contradictory to our theory since our theory does not make explicit predictions about the rate of phase transition for ASR-U. Across different GAN models, we found that JSD generally approaches perfect ASRU at a faster rate than MMD in all three graphs, suggesting the use of nonlinear dynamic may be beneficial. Nevertheless, the overall trends for different GANs remain in large part homogeneous.
Between Wasserstein and MMD, we observe very little difference in performance, suggesting the regularization effect of NTK is sufficient to control the Lipschitz coefficient of the network. Finally, for the MMD GAN in the matched setting, we found the network is able to achieve perfect ASR-U regardless of the spectral properties of the Markov transition graphs, which confirms our theory that a symmetric Markov random matrix tends to have simple eigenvalue spectrum suitable for ASR-U.
Effect of discriminator reset As pointed out by (Franceschi et al., 2021), a discriminator may suffer from residual noise from previous updates and fail to approximate the target divergence measure. We analyze such effects for MMD and JSD as shown in Figure 5. We observed consistent trends that models whose weights are reset to the initial weights every discriminator loop outperform those without resetting. The effect is more pronounced for JSD GAN than MMD GAN and for smaller σmin(PX).
Effect of generator averaging strategy The original wav2vec-U (Baevski et al., 2021) directly feeds the text posterior probabilities O into the discriminator, which we refer to as the “soft input” approach. Alternatively, we can instead calculate a weighted average of the gradient form over the samples y ∈ YL as in Eq. (13), which we refer to as the “outside cost” approach. The comparison between the two approaches are shown in Figure 6. We observed mixed results: for MMD GANs, the softinput approach outperforms the outside-cost approach and performs best among the models in the high-σmin(PX) setting; for JSD GANs, we found that the outside-cost approach performs slightly better than the soft-input approach. Such inconsistencies may be another consequence of the regularization effect predicted by the GANTK. We leave the theoretical explanation as future work.

heading: 5 Related works
text: (Glass, 2012) first proposed the challenging task of ASR-U as a key step toward unsupervised speech processing, and framed it as a decipherment problem. (Liu et al., 2018) takes on the challenge by developing the first ASR-U system with groundtruth phoneme boundaries and quantized speech features as inputs, by training a GAN to match the speech-generated and real text distributions. (Chen et al., 2019) later replaced the ground truth boundaries with unsupervised ones refined iteratively by
an HMM, which also incorporates language model information into the system. (Yeh et al., 2019) explored the cross entropy loss for matching the generated and real text distribution, but it is prone to mode collapse and needs the help of additional regularization losses such as smoothness weight. More recently, (Baevski et al., 2021; Liu et al., 2022) proposed another GAN-based model using continuous features from the last hidden layer of the wav2vec 2.0 (Baevski et al., 2020) model and additional regularization losses to stabilize training. Their approach achieves ASR error rates comparable to the supervised system on multiple languages, making it the current state-of-the-art system.
To better understand the learning behavior of ASR-U systems, (Lin et al., 2022) analyze the robustness of wav2vec-U against empirical distribution mismatch between the speech and text, and found that N -gram language model is predictive of the success of ASR-U. Inspired by the original framework in (Glass, 2012), (Klejch et al., 2022) proposed a decipher-based cross-lingual ASR system by mapping IPA symbols extracted from a small amount of speech data with unpaired phonetic transcripts in the target language.
Our analysis on the sufficient condition of ASRU is based on previous work on the asymptotic behaviour of GAN objective functions (Goodfellow et al., 2014; Arjovsky et al., 2017). Our finitesample analysis takes inspiration from later work extending the asymptotic analysis to the finitesample regimes (Arora et al., 2017; Bai et al., 2019). Such frameworks, however, do not account for the alternate gradient optimization method of GANs and inevitably lead to various inconsistencies between the theory and empirical observations of GAN training (Franceschi et al., 2021). Building upon prior works (Mescheder et al., 2017, 2018; Domingo-Enrich et al., 2020; Mroueh and Nguyen, 2021; Balaji et al., 2021), (Franceschi et al., 2021) proposed a unified framework called GANTK based on NTK (Jacot et al., 2018) to describe the training dynamic of any GAN objectives and network architectures. Our analysis on the training dynamic of ASR-U adopts and extends the GANTK framework to handle discrete, sequential data such as natural languages.

heading: 6 Conclusion
text: In this paper, we develop a theoretical framework to study the fundamental limits of ASR-U as well
as the convergence properties of GAN-based ASRU algorithms. In doing so, our theory sheds light on the underlying causes of training instability for such algorithms, as well as several new directions for more reliable ASR-U training.

heading: 7 Limitations
text: Our theory currently assumes that input speech features are quantized into discrete units, as in (Chen et al., 2019), while preserving all the linguistic information in the speech. As a result, our theory does not account for the loss of linguistic information during the quantization process, as often occurred in realistic speech datasets. Further, more recent works (Baevski et al., 2021; Liu et al., 2022) have shown that continuous features, with the help of additional regularization losses, can achieve almost perfect ASR-U. Such phenomena is beyond explanations based on our current theory and require generalizing our theory to continuous speech features. Further, our model assumes that sufficiently reliable phoneme boundaries are fed to the ASR-U system, and kept fixed during training. It will be interesting to extend our framework to systems with trainable phoneme boundaries, such as the wav2vec-U systems, to better understand its effect on training stability.

heading: Acknowledgements
text: This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2022-0-00184, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics)

heading: A Proofs of theoretical results
text: A.1 Learnability of ASR-U: a sufficient condition
Proof. (Theorem 1)
For simplicity, we assume that the eigenvalues of A are real though a similar argument applies to complex eigenvalues as well. By Assumptions 1 and 2, it can be verified that
PXkN = π ⊤AkE
= π⊤UΛkU−1E,
where E = 1|X|N−2 ⊗ I|X|, where ⊗ denotes the Kronecker product. Define cjk = π⊤ujk. Define r⊤jk to be the k th row of the jth block of the matrix U−1E, i.e., UU−1E = ∑K
j=1 ∑Nj k=1 ujkr ⊤ jk. De-
fine the matrix RK as RK = [r1, · · · , rK ], where rj = ∑Nj k=1 cjkrjk. Then we have:
P⊤XkN = K∑
j=1
λkj r ⊤ j
PX = VL(λ1:K) ⊤R⊤K ,
where VL(λ1:K) is the Vandermonde matrix formed by nonzero eigenvalues λ1, · · · , λK and with L columns, K ≥ |X| by Assumption 1. RK has full column rank of K ≥ |X| by Assumption 2, therefore it is possible to write RK = R̂KL, where R̂K = r̂1, . . . , r̂K ] is a matrix with orthogonal columns, and L is lower-triangular. As a result, we have PX is full rank iff VL(λ1:K) has full row rank of at least |X|, which holds by Assumption 1.
Proof. (Lemma 1)
Use the Rayleigh-characterization of eigenvalues
of the matrix PX⊤PX , we have
σmin(P X)
= √ λmin(PX⊤PX) = √
min ∥w∥=1
w⊤PX⊤PXw
= √
min ∥w∥=1
w⊤RKVLV ⊤L R ⊤ Kw
≥
√√√√ L−|X|−1∑
l=0
λ2lmin min∥w∥=1 w⊤RKV|X|V ⊤|X|R ⊤ Kw
= σmin(P X 1:|X|)
√√√√ L−|X|−1∑
l=0
λ2lmin,
where λmin is the eigenvalue of A with minimum absolute value, and PX1:|X| is the first |X| rows of PX . Therefore, to lower bound σmin(PX), it suffices to lower bound σmin(PX1:|X|). But note that
σmin(P X 1:|X|)
= min ∥w∥=1
∥V T|X|RTKw∥
≥σmin(V T|X|) min∥w∥=1 ∥R T Kw∥
≥ σmax(V|X|)
κ(V|X|) min j
∥r̂j∥
≥ | det(V|X|)|1/|X|
κ(V|X|) min j
∥r̂j∥
= |∏1≤i<j≤|X| |λi − λj |1/|X|
κ(V|X|) min j
∥r̂j∥
≥δ (|X|−1)/2|X| min
κ(V|X|) min j
∥r̂j∥
where the last equality uses the closed-form formula of the determinant of a square Vandermonde matrix, and where the behaviour of κ(V|X|), the condition number of the Vandermonde matrix, has been studied in depth in (Bazán, 2000).
A.2 Finite-sample learnability of ASR-U: matched setup
Theory of small ball probability The proof of Theorem 2 makes extensive use of the theory of small ball probability. Therefore, we briefly provide some background on the subject. First, we define the small ball probability of a vector x as follows.
Definition 2. (Small ball probability) Given a fixed vector x = (x1, · · · , xn), and i.i.d random variables ξ = (ξi, · · · , ξn), the small ball probability is defined as
ρδ(x) := sup a∈R
Pr[|ξ⊤x− a| ≤ δ].
Intuitively, small ball probability is the amount of “additive structure” in x: for example, if the coordinates of x are integer multiples of each other and ξi’s are symmetric Bernoulli variables, the product ξ⊤x tends to have small magnitude as terms cancel out each other very often. Since sparser vectors tend to have less additive structure, small ball probability can also be used to measure how sparse the weights of x are. Another way to look at this is that, if the L2 norm of x is fixed and most of the weight of x is gathered in a few coordinates, the product ξ⊤x has higher variance and is thus less likely to settle in any fixed-length intervals. This is quantitatively captured by the celebrated Offord-Littlewood-Erdös (OLE) anti-concentration inequality (and its inverse) for general subgaussian random variables:
Lemma 2. (Erdös, 1945; Rudelson and Vershynin, 2008; Tao and Vu, 2009) Let ϵ > 0 be fixed, let δ > 0, and let v ∈ Rm be a unit vector with
ρδ(v) ≥ m− 1 2 +ϵ.
Then all but at most ϵm of the coefficients of v have magnitude at most δ.
Note that here we use a slight generalization of the notion of sparsity called compressibility defined as follows.
Definition 3. ((α, δ)-compressible) A vector v ∈ Rn is (α, δ)-compressible if at most ⌊αn⌋ of its coefficients have magnitude above δ.
Note that a sparse vector with a support of size at most ⌊αn⌋ is (α, 0)-compressible.
A more generally applicable anti-concentration inequality requires the following definition of generalized arithmetic progression, which is used to quantify the amount of additive structure of a vector.
Definition 4. (Generalized arithmetic progression) A generalized arithmetic progression (GAP) is a set of the form
Q = {a⊤w : a ∈ Zr, |ai| ≤ Ni, 1 ≤ i ≤ r},
where r ≥ 0 is called the rank of the GAP and w1, · · · , wr ∈ R are called generators of the GAP. Further, the quantity
vol(Q) :=
r∏
i=1
(2Ni + 1)
is called the volume of the GAP.
Lemma 3. (Continuous inverse Littlewood-Offord theorem, Theorem 2.9 of (Nguyen and Vu, 2011)) Let ϵ > 0 be fixed, let δ > 0 and let v ∈ Rn be a unit vector whose small ball probability ρ := ρδ(v) obeys the lower bound
ρ ≫ n−O(1).
Then there exists a generalized arithmetic progression Q of volume
vol(Q) ≤ max ( O ( 1
ρ √ αn
) , 1 )
such that all but at most αn of the coefficients v1, · · · , vn of v lie within δ of Q. Furthermore, if r denotes the rank of Q, then r = O(1) and all the generators w1, · · · , wr of Q have magnitude O(1).
While applicable for any ρ ≫ n−ϵ rather than only those with ρδ(v) ≥ n−1/2+ϵ as required by Lemma 2, Lemma 3 is weaker than Lemma 2 in the sense that rather than showing that the vector is compressible with high probability and thus covered by the set of compressible vectors, it proves that the vector is covered by a small set with high probability.
A related notion that is often more convenient for our analysis is the segmental small ball probability, which is simply small ball probability computed on a segment of the vector:
ρδ,α(x) = inf I⊆{1,··· ,n}:|I|=⌊αn⌋ ρδ(xI),
From the definition, it is not hard to see that ρδ,α(x) ≥ ρδ(x).
Eigen-gaps of symmetric Markov random matrix Armed with tools from the theory of small ball probability, we will establish guarantees of eigenvalue gaps for a symmetric Markov random matrix. First, we shall show that Theorem 2 implies Corollary 1.
Proof. (Proof of Corollary 1) Using Theorem 2 and union bound, the probability that a symmetric Markov random matrix has at least |X| distinct eigenvalues can be bounded as
Pr [ min
1≤i≤|X| |λi − λi+1| ≤ |X|−BN
] ≤
|X|max i
Pr [ |λi − λi+1| ≤ |X|−BN ]
= O(|X|−CN ),
with probability at least 1 − O(exp(−α0|X|N )).
It turns out that a symmetric Markov random matrix enjoys various properties analogous to a symmetric matrix. First, we can show that its eigenvalues are real. This can be proved by noting that for a symmetric Markov random matrix An := D−1n Wn and for any of its eigenvalues λ with eigenvector v,
D−1n Wnv = λv
⇐⇒D−1/2n WnD−1/2n (D1/2n v) = λD1/2n v, (21)
which implies An has the same spectrum as D −1/2 n WnD −1/2 n , which is symmetric and thus has a real spectrum. Further, we can prove a variant of Cauchy’s interlace theorem for symmetric Markov random matrix. Lemma 4. Suppose An = D−1n Wn ∈ Rn×n is a symmetric Markov random matrix with adjacency matrix Wn and eigenvalues λ1 ≥ · · · ≥ λn and Am = D −1 m Wm with adjacency matrix Wm−1 and eigenvalues ν1 ≥ · · · ≥ νm,m < n is formed by successively deleting i-rows and i-columns, then
λi ≤ νi ≤ λi+n−m.
Proof. Using the previous observation in Eq. 21, we can apply the standard Cauchy’s interlacing theorem on A′n := D −1/2 n WnD −1/2 n and A′m := D −1/2 m WmD −1/2 m , then we have
λi(An) = λi(A ′ n) ≤ λi(A′m) = λi(Am)
≤ λi+n−m(A′n) = λi+n−m(An).
Next, we can show that the eigenvalues of a symmetric Markov random matrix and its adjacency matrix are simultaneously distributed within the bounded intervals [−10nγ−1, 10nγ−1] and [−10nγ , 10nγ ] with high probability. For this and subsequent proofs, we will assume γ′ = γ > 1/2.
Lemma 5. Let An = D−1n Wn be a symmetric Markov random matrix with adjacency matrix Wn and properties defined in Theorem 2, then we have with probability at least 1−O(exp(−α0n)),
λi(An) ∈ [−10nγ−1, 10nγ−1], λi(Wn) ∈ [−10nγ , 10nγ ],
for any 1 ≤ i ≤ n and some α0 > 0.
Proof. First, by definition, we can let Wn = Fn + Xn, where Fn is a deterministic matrix with eigenvalues of order nγ and Xn is a symmetric matrix whose elements are independent zero-mean unitvariance subgaussian random variables. Using standard results from random matrix theory (Anderson et al., 2009), we have
{λ1(Xn), · · · , λn(Xn)} ⊂ [−10nγ−1, 10nγ−1],
with probability at least 1−O(exp(−α0n)). Therefore, Weyl’s matrix perturbation inequality then ensures that
{λ1(Wn), · · · , λn(Wn)} ∈ [−10nγ , 10nγ ],
with probability at least 1−O(exp(−α1n)). Suppose this event occurs and use Lemma 4 and the variational characterization of eigenvalues, we have
λi(An) = min Vi−1 max v∈V ⊥
i−1 ∥v∥2=1
v⊤D−1/2n WnD −1/2 n v
= min Vi−1 max v∈V ⊥
i−1 v⊤Dv=1
v⊤Wnv
= min Vi−1 max v∈V ⊥i−1 v⊤Wnv v⊤Dnv ,
where Vi−1 is a subspace of dimension i− 1. Combining the two results, we have with probability at least 1−O(exp(−α1n)),
max v∈V ⊥i−1 ∣∣∣∣ v⊤Wnv v⊤Dnv ∣∣∣∣ ≤ maxv:∥v∥=1 |v⊤Wnv| minv:∥v∥=1 |v⊤Dnv|
= λ1(W )
mini |dii|
Recall that dii = ∑n j=1wij = ∑n
j=1(fij + xij), where wij , fij , and xij are the (i, j)th elements of Wn, Fn, and Xn respectively. Since An = D−1n Wn is a Markov matrix we assume that fij and the distribution of xij are selected to guarantee that wij ≥ 0, e.g., it must be true that fij ≥ 0.
We also know that xij is a zero-mean unit-variance sub-Gaussian random variable, therefore
Pr {wij < δ} = Pr {xij < −fij + δ}
≤ 2 exp ( −1 2 (fij − δ)2 )
Pr {dii < nδ} = Pr    n∑
j=1
wij < nδ   
≤ 2 exp (−α2n)
where α2 = −12(f̄i − δ)2, and f̄i = 1 n ∑ j fij . Therefore, with probability at least 1−O(exp(−α0n)) where α0 = α1 + α2,
λi(An) ∈ [−10nγ−1, 10nγ−1], 1 ≤ i ≤ n (22)
Remark. Lemma 5 ensures that for any symmetric Markov random matrix An = D−1n Wn with properties defined in Theorem 2, we can focus our attention on any eigenvector v whose eigenvalue is no greater than O(nγ−1) and whose ∥Wnv∥2 is of order nγ with high probability. Therefore, we will assume such conditions in later proofs.
Using Lemmas 4-5, we can reduce Theorem 2 to the following statement on small ball probability of the eigenvectors of Xn, analogous to the arguments for symmetric random matrices in (Nguyen et al., 2017).
Lemma 6. Let An = D−1n Wn ∈ Rn×n be a symmetric Markov random matrix with adjacency matrix Wn. Let λi(An) and w = [u⊤, b]⊤ ∈ Rn be the i-th eigenvalue and eigenvector of the matrix An, respectively, where u ∈ Rn−1 and b ∈ R. Then we have
Pr[|λi(An)− λi+1(An)| ≤ δ] ≤ nPr[ρδnγ+1(v) ≥ c0nγ+1δ] + c0nγ+2δ
+O(exp(−α0n)),
for some c0, α0 > 0.
Proof. Let Wn−1 and Dn−1 be the (n − 1)- dimensional minors of Wn and Dn, respectively, then [ Wn−1 wn w⊤n wnn ] [ u b ] = λ [ Dn−1 0n 0⊤n dnn ] [ u b ] ,
where wn is the last column of Wn. Let v be the i-th eigenvector of matrix An−1 := D−1n−1Wn−1, we have
v⊤Wn−1u+ v⊤Wb = λi(An)v⊤Dn−1u
=⇒|(λi(Xn−1)− λi(Xn))| max 1≤i≤n dii ≥
|(λi(An−1)− λi(An))v⊤Dn−1u| = |v⊤wnb|.
Therefore,
Pr[|λi(An)− λi(An−1)| ≤ δ]
≤Pr [ |v⊤wn| max1≤i≤n dii ≤ δ b ] .
By Lemma 4, λi+1(An) ≤ λi(An−1) ≤ λi(An) and we have
Pr[|λi(An)− λi+1(An)| ≤ δ] ≤ Pr[|λi(An−1)− λi(An)| ≤ δ] ≤
Pr [ |v⊤wn| max1≤i≤n dii ≤ δ b ] .
dii is typically O(n), but we have been unable to prove that it is necessarily O(n). Consider that wij = fij + xij , where Fn is a symmetric matrix with eigenvalues λi(Fn) = O(nγ), therefore
n∑
j=1
fij = (Fn1n)i ≤ ∥Fn1n∥2 = ∥Fn∥1
≤ n1/2∥Fn∥2 = O ( nγ+ 1 2 ) .
Wn = Fn +Xn, therefore
Pr { dii ̸= O ( nγ+ 1 2 )}
≤ Pr    n∑
j=1
xij >
n∑
j=1
fij − nδ   
≤ O(exp(−α2n))
Now, by the law of total probability,
Pr [ |v⊤wn| max1≤i≤n dii ≤ δ b ]
≤Pr [ |v⊤wn| max1≤i≤n dii ≤ δ b , max 1≤i≤n dii ≤ O(nγ+ 1 2 ) ]
+ Pr [ max 1≤i≤n dii ̸= O ( nγ+ 1 2 )]
≤Pr [ |v⊤wn| = O ( δnγ+ 1 2
b
)] +O(exp(−α2n)).
By symmetry, we can choose any row and the corresponding column to split the matrix and derive inequality of the same form. Further, suppose for some b1 > 0, with probability at least 1 − exp(−c1n), there are at least nT coordinates of w that are at least b1 and suppose we choose the split index J uniformly at random. Let the J-th column of Wn be W and the J-th coefficient of the eigenvector of Wn be wJ , then we have
Pr[|λi(An)− λi+1(An)| ≤ δ]
≤Pr [ |v⊤W | ≠ O ( δnγ+ 1 2
wJ
) |Nb ≥ nb ]
+O(exp(−c1n)) +O(exp(−α2n))
≤ n nT Pr
[ |v⊤W | ≠ O ( δnγ+ 1 2
b1
) |Nb ≥ nb ]
+O(exp(−c1n)) +O(exp(−α2n)),
where the second inequality can be proved as follows. Define
E = {Nb ≥ nb} , F = {wJ ≥ b1} , G = { |v⊤W | ≠ O ( δnγ+1/2
wJ
)} ,
H = { |v⊤W | ≠ O ( δnγ+1/2
b1
)} .
Then use the above definitions and the fact that F and G are conditionally independent given Nb, we have
Pr [ |v⊤W | ≠ O ( δnγ+ 1 2
b1
) |Nb ≥ nb ]
=Pr(H|E) ≥ Pr(F ∩ G|E) ≥ nT n Pr(G|E)
= nT n Pr
[ |v⊤W | ≠ O ( δnγ+1/2
wJ
) |Nb ≥ nb ] .
Further, to remove the dependency on Nb, notice that
Pr(H|E) ≤ Pr(H) Pr(E) = Pr(H) +O(exp(−c1n)).
Next, by the pigeonhole principle, at least one coordinate of the unit eigenvector w is at least n−1/2, and thus we can let c1 = ∞, nb = 1 and
b1 = n −1/2 and arrive at
Pr [|λi(An)− λi+1(An)| ≤ δ] ≤nPr [ |v⊤W | ≠ O ( δnγ+1 )] +O(e−α0n)
≤nρδO(1)nγ+1(v) +O(exp(−α0n)), (23)
where α0 = c1 + α2. Finally, recall the definition of small ball probability, we have
Pr [ |v⊤W | ≤ δ ] ≤ Pr [ |v⊤W | ≤ δ|ρδ(v) ≤ ϵ ]
+ Pr[ρδ(v) > ϵ]
≤ Pr[ρδ(v) > ϵ] + ϵ,
and thus applying this inequality with δ := c0δn γ+1 on Eq. (23) yields the result.
Remark. We can sharpen the bound in Lemma 6 by extending the delocalization theorem for a symmetric Wigner matrix (see Theorem 4.2 of (Nguyen et al., 2017)) to a symmetric Markov random matrix and using it to choose a larger nb in the proof. This will be left as future work.
With the help of Lemma 6, we can reduce Theorem 2 to the following theorem. Theorem 5. Let An ∈ Rn×n be a symmetric Markov random matrix matrix and v be an eigenvector with eigenvalue λ = O(nγ−1), then for any fixed C > 0, there exists some B > max{4γC + 3γ, 4γ + 1} such that
ρn−B (v) ≤ n−C ,
with probability at least 1 − O(exp(−α0n)) for some α0 depending on B.
Similar to the proof for the perturbed symmetric matrices in (Nguyen et al., 2017), we reduce Theorem 5 to the following.
Theorem 6. Let v be the eigenvector and B be the constant defined in Theorem 5. Then for any n−B ≤ δ ≤ n−B/2, we have with probability O(exp(−α0n)),
n−C ≤ ρnγδ(v) ≤ n0.49ρδ(v). (24)
To show that Theorem 6 implies Theorem 5, we prove the contrapositive of the statement, that is, if ρn−B (v) > n−C , then there exists n−B ≤ δ ≤ n−B/2 such that Eq. 24 holds with probability at least 1−O(exp(−α0n)). To construct such δ, let
δ0 := n −B
δj+1 := n γδj ,
for j = 0, · · · , J − 1 with J = ⌊B/2γ⌋. By construction, we have
n−B = δ0 ≤ δj ≤ δJ ≤ n−B/2
ρδj (v) ≥ ρδ0(v) ≥ n−C . Suppose Eq. 24 does not hold for any δ := δj , or otherwise the result follows, we have
ρδJ (v) ≥ n0.49Jρn−B (v) ≥ n0.49⌊B/2γ⌋−C > 1, if B ≥ 4γC + 3γ, which contradicts the fact that ρδJ (v) ≤ 1. As a result, there has to exist some j such that Eq. 24 holds.
Again similar to the perturbed symmetric matrix case in (Nguyen et al., 2017), we divide the proof of Theorem 6 into the compressible case and the non-compressible case. For the compressible case, we first prove the following lemma. Lemma 7. Suppose v is an eigenvector of a symmetric Markov random matrix An := D−1n Wn with adjacency matrix Wn and the same properties defined in Theorem 2, and suppose there exists δ ∈ [n−B, n−B/2] such that ρδ,α(v) ≥ (αn)−1/2+ϵ, we have with probability O(exp(−α0n)), n−C ≤ ρnγδ(v) ≤ n0.49ρδ(v). Proof. Using concentration inequalities, we have with probability at least 1 − O(exp(−α2n)) for some α2 > 0,
dii = O ( nγ+ 1 2 ) , 1 ≤ i ≤ n (25)
Further, since ρδ,α(v) ≥ (αn)−1/2+ϵ, by Lemma 2, we have v is (O(α), δ) compressible, and thus there exists I of of size O(αn) such that vi > δ only if i ∈ I . Without loss of generality, let I = {n − k, · · · , n} for k = O(αn) and E[Aij ] = 1. Further, split v = [v′⊤, v′′⊤]⊤, then by definition of eigenvalues and eigenvectors, [ Wn−k F F⊤ Wk ] [ v′ v′′ ] = λ [ Dn−k 0 0⊤ Dk ] [ v′ v′′ ] .
Reading off the first line of the matrix equation, we have
∥Fv′′∥2 = ∥(Wn−k − λDn−k)v′∥2 ≤ ∥Wn−kv′∥2 + ∥λDn−kv′∥2.
Notice that assuming Eq. 25 and Eq. 22 occur, we have that all elements v′i of v
′ have |v′i| < δ, therefore ∥v′∥2 ≤ δn−1/2, therefore
∥Wn−kv′∥2 ≤ δn1/2 max v:∥v∥2=1 ∥Wv∥2
= O(n−B/2+1/2+γ)
Furthermore, if we assume that Eq. (22) and Eq. (25) occur, then
∥λDn−kv′∥2 = O(nγ−1 · nγ+1/2 · δn1/2) = O(n−B/2+2γ).
Thus, using the fact that B ≥ 4γ + 1,
∥Fv′′∥2 = O(n−B/2+2γ) = O(n−1/2).
On the other hand, using a standard epsilonnet argument, with probability at least 1 − O(exp(−α3n)),
inf w∈Rk:∥w∥=1
∥Fw∥2 ≥ n−1/2.
Now, define the events
E := {v is an eigenvector of A} Eα,δ := {v is (O(α), δ)-compressible} EI := {∥WIc,IvI∥2 ≫ O(n−1/2)},
then by the previous discussion, we have
Pr(EI |E ∩ Eα,δ) = O(exp(−α2n)) Pr(EcI |E) = O(exp(−α3n)).
Note that to prove the lemma, it suffices to show that the eigenvector v is not (O(α), δ)compressible with high probability, or Pr(Eα,δ|E) is small, since that will lead to ρδ,α(v) < (αn)−1/2+ϵ with high probability and thus a contradiction with high probability. Indeed, we have
Pr(Eα,δ|E) ≤ Pr(Eα,δ ∩ EI |E) + Pr(Eα,δ ∩ EcI |E) ≤ Pr(EI |E ∩ Eα,δ) + Pr(EcI |E) = O(exp(−α0n))
for some α0 > 0.
For the incompressible case, we apply the continuous inverse Offord-Littlewood theorem to discretize the set of eigenvectors, and prove the following result analogous to the symmetric case in (Nguyen and Vu, 2011).
Lemma 8. Suppose v is an eigenvector of a symmetric Markov random matrix An := D−1n Wn with adjacency matrix Wn and the same properties defined in Theorem 2, and suppose there exists δ ∈ [n−B, n−B/2] such that q := ρδ,α(v) < (αn)
−1/2+ϵ, we have with probability O(exp(−α0n)),
n−C ≤ ρnγδ(v) ≤ n0.49ρδ(v).
To prove this result, we need the following useful lemmas.
Lemma 9. For any eigenvector-eigenvalue pair (v, λ) and α > 0 with |λ| = O(nγ−1), suppose n−C < ρδ,α(v) =: q ≤ (αn)−1/2+ϵ, then with probability at least 1 − O(exp(−α0n)) there exists a subset N of Rn × R of size O(n−n/2+O(αn)q−n+O(αn)) such that, there exists (ṽ, λ̃) ∈ N with the properties:
1. |vj − ṽj | ≤ δ for 1 ≤ j ≤ n;
2. |λ− λ̃| ≤ nγδ.
Proof. Split {1, · · · , n} into sets of length differing by at most 1, I1, · · · , Im, m = ⌊ 1 α ⌋ + 1, then we have the length of each set is greater than or equal to ⌊αn⌋, and its small ball probability is
ρδ(vIi) ≥ ρδ,α(v) = q, 1 ≤ i ≤ m.
Therefore, since q ≤ (αn)− 12+ϵ and n−C < q, there exists a GAP
Qi =    ri∑
j=1 aijwij : aj ∈ Z, |aij | ≤ Nij , 1 ≤ j ≤ ri
  
such that
sup j∈Ii\S inf ṽj∈Qi
|vj − ṽj | ≤ δ,
with volume
vol(Qi) ≤ O((αn)−1/2+ϵ/q), 1 ≤ i ≤ m,
for all except at most O(α2n) indices from some exceptional set S. Further, for each Qi, we can quantize its generators wi1, · · · , wiri to the closest multiple of qδ, w̃i1, · · · , w̃iri . This introduces an additional approximation error of at most
∣∣∣∣∣∣ ri∑
j=1
aijwij − ri∑
j=1
aijw̃ij ∣∣∣∣∣∣ ≤vol(Qi) · qδ ≤ (αn)−1/2+ϵ/q · qδ =(αn)−1/2+ϵδ = O(δ).
Next, for the coefficients from the exceptional set S, we also round them to the closest multiple of qδ and let the set of such values be R, which ensures that
sup j∈S inf v′∈R
|vj − v′| = O(δ).
Therefore, for fixed generators wij’s and a given S, we can construct a finite set of vectors
{ṽ : ṽj ∈ ∪mi=1Qi, ∀j ̸∈ S and v′j ∈ R, ∀j ∈ S}
of size at most
( m sup
i vol(Qi)
)n−|S| |R||S|
≤O ( 1
α
(αn)−1/2+ϵ
q
)n ·O((1/qδ)O(αn))
≤O ( n− n 2 +ϵnq−n+O(αn) ) O ( nBαn )
=O(n−n/2+O(αn)q−n+O(αn)),
=O(n−n/2+O(αn)q−n),
that approximates v within O(δ) for every coefficients. The third line uses δ > n−B and α = O(1); the fourth line assumes ϵ = O(α). Further, if we allow the generators to be variable and assume S to be unknown, the quantization mentioned previously and the crude bound of the number of possible S by 2n enlarges the set of vectors by a factor of O ( (1/qδ) ∑m i=1 ri ) ·O(2n) = O(nO(m)) ·O(2n)
= O(nO(1/α)) ·O(2n) = O(nO(αn)).
For the eigenvalue, we also have there exists a set that covers its domain to be within δnγ with a set of size
O
( nγ−1
nγδ
) = O(nB−1) ≤ O(nO(αn)).
with probability at least 1−O(exp(−α0n)). Composing the sets, we find the set N has size O(n−n/2+O(αn)q−n+O(αn)).
Lemma 10. For any eigenvector-eigenvalue pair (v, λ) of an symmetric Markov random matrix An = D −1 n Wn with adjacency matrix Wn and the same properties defined in Theorem 2 and let (ṽ, λ̃) ∈ N be the tuple that well approximates it as defined in Lemma 9, we have
∥AIc,I ṽI − u∥2 = O(δnγ),
where AI,J is the matrix formed by row indices from I and column indices from J and u := (λ̃− AIc,Ic)ṽIc .
Proof. By symmetry, we can let I = {1, · · · , k} for k = ⌊αn⌋. Notice by definition we can split A as
[ Ak G F⊤ An−k ] [ w v′ ] = λ [ w v′ ] ,
where v = [w⊤, v′⊤]⊤, and as a result,
∥F⊤ṽI − (λ̃−An−k)ṽIc∥2 ≤∥F⊤w − (λ−An−k)v′∥2+ ∥F⊤(ṽI − w)∥2 + ∥(λ̃− λ)ṽIc∥2+ ∥(λ−An−k)(ṽIc − v′)∥2 =∥F⊤(ṽI − w)∥2 + |(λ̃− λ)ṽIc∥2 + ∥(λ−An−k)(ṽIc − v′)∥2 =O(nγ−1 · δn1/2) +O(nγδ) +O(nγ−1 · δn1/2) = O(nγδ).
Now we are ready to prove Lemma 8.
Proof. Let E be the event that there exists some δ ∈ [n−B, n−B/2] such that
n−C ≤ ρnγδ(v) ≤ n0.49ρδ(v) =: n0.49q
with q := ρδ(v) and G be the event that
∥AIc,I ṽI − u∥2 = O(δnγ),
where u := (λ̃ − AIc,Ic)ṽIc and (ṽ, λ) well approximates (v, λ) as defined in Lemma 10. Let k := |I| = O(αn), from Lemma 9, we have
Pr(Gc) = O(exp(−α0n)).
On the other hand, if E occurs, define AIc,I = [ak+1, . . . , an]
⊤, u = [uk+1, . . . , un]⊤, then we have
Pr(G|E) ≤ ∑
(w′,ṽ,λ̃)∈N
Pr
[ n∑
i=k+1
|a⊤i w′ − ui|2 = O(δ2n2γ+1) ]
≤ |N |(ρnγδ(v))n−k ≤ |N |(n0.49q)n−k
= O(n−0.01n+O(αn)),
which is O(exp(−α0n)) if α is chosen small enough. As a result, we have
Pr(E) ≤ Pr(G|E) + Pr(Gc) = O(exp(−α0n)).
A.3 Finite-sample learnability of ASR-U: Unmatched setup
Proof. (Theorem 3) Under the assumptions that the discriminator is perfect and decomposable and the GAN objective is MMD with a linear kernel over the embeddings D(Y ) = P̂ Y , Eq. (8) becomes the following least squares regression problem
min O′∈R|X|×|Y|
∥P̂XO′ − P̂ Y ∥2F . (26)
Let Ô be the ERM of Eq. (26) and O be the true assignment matrix, by definition and triangle inequality,
∥P̂XÔ − P̂ Y ∥F ≤ ∥P̂XO − P̂ Y ∥F ≤ ∥P̂XO − P Y ∥F + ∥P̂ Y − P Y ∥F .
Apply the triangle inequality again, we have
∥P̂X(Ô −O)∥F ≤ ∥P̂XÔ − P̂ Y ∥F + ∥P̂XO − P̂ Y ∥F ≤ 2∥P̂XO − P Y ∥F + 2∥P̂ Y − P Y ∥F
Note that if we replace any X(i) → X(i)′ and let the resulting empirical distribution be P̂X ′ ,
∣∣∣∥P̂XO − P Y ∥F − ∥P̂X ′ O − P Y ∥F ∣∣∣
≤∥(P̂X − P̂X′)O∥F ≤ √ 2L
nX ,
and similarly for P̂X and P̂ Y ,
∣∣∣∥P̂X − PX∥F − ∥P̂X ′ − PX∥F ∣∣∣ ≤ √ 2L nX ∣∣∣∥P̂ Y − P Y ∥F − ∥P̂ Y ′ − P Y ∥F ∣∣∣ ≤ √ 2L
nY .
Therefore, we can apply McDiarmid’s inequality to obtain
Pr [ ∥P̂X − PX∥F ≥ √ L|X|√ nX + ϵ ] ≤ e−n Xϵ2 L
Pr [ ∥P̂XO − P Y ∥F ≥ √ L|Y|√ nX + ϵ ] ≤ e−n Xϵ2 L
Pr [ ∥P̂ Y − P Y ∥F ≥ √ L|Y|√ nY + ϵ ] ≤ e−n Y ϵ2 L .
Moreover, let ϵXX := √
L|X|√ nX
+ ϵ, ϵY X := √
L|Y|√ nX +
ϵ, ϵY Y = √
L|Y|√ nY + ϵ, then by a union bound, we
have
Pr [ ∥P̂X(Ô −O)∥F ≥ ϵY X + ϵY Y ] ≤
Pr [ ∥P̂XÔ − P Y ∥F + ∥P̂ Y − P Y ∥F ≥
ϵY X + ϵY Y
2
]
≤ Pr [ ∥P̂ Y XÔ − P Y Y ∥F ≥ ϵY X
2
] +
Pr [ ∥P̂ Y − P Y ∥F ≥ ϵY Y
2
] ≤ e−n Xϵ2 4L +e− nY ϵ2 4L .
Therefore, we have with probability at least 1 − e− nXϵ2 4L − e−n Y ϵ2 4L ,
ϵY X + ϵY Y ≥ ∥P̂X(Ô −O)∥F ≥ ∥PX(Ô −O)∥F − ∥P̂X − PX∥F ∥Ô −O∥F ≥ (σmin(PX)− ∥P̂X − PX∥F )∥Ô −O∥F ,
and combined with the bound on ∥P̂X − PX∥F , we obtain with probability at least (1− e−n Xϵ2
4L − e− nY ϵ2 4L )(1− e−n Xϵ2 4L ),
∥Ô −O∥F ≤ ϵY X + ϵY Y
σmin(PX)− ϵXX .
Assume the correct mapping is deterministic, so that Oxy ∈ {0, 1} and each row has only one nonzero element, then to achieve perfect ASR-U, we need for any x ∈ X and y ̸= G(x),
|ÔxG(x) − Ôxy| > 0 ⇐= 1− |ÔxG(x) −OxG(x)| − |Ôxy −Oxy| > 0
⇐= 1− 2∥Ô −O∥∞ > 0 ⇐⇒ ∥Ô −O∥F < 1
2 ,
which occurs if
σmin(P X) > ϵXX + 2ϵY X + 2ϵY Y .
A.4 Training dynamic of ASR-U To prove Theorem 4, we need the following lemma on the properties of the gradient of the softmax function based on (Gao and Pavel, 2017).
Lemma 11. Let H(x) be the Jacobian matrix of the softmax function σ : Rd 7→ Rd with σi(x) = exi∑d j=1 e xj , then we have H(x) = diag(σ(x)) − σ(x)σ(x)⊤ and H(x) is positive semi-definite (PSD) with the null space span{1d}.
Proof. Apply product rule of calculus, we have
Hij(x) = ∂σi(x)
∂xj
= δijσi(x)− exiexj
( ∑d
j=1 e xj )2
= δijσi(x)− σi(x)σj(x),
and therefore H(x) = diag(σ(x)) − σ(x)σ(x)⊤. To show that H(x) is PSD, notice that
v⊤H(x)v = v⊤diag(σ(x))v − (v⊤σ(x))2
= EI∼σ(x)[v2I ]− E2I∼σ(x)[vI ] = Var(vI) ≥ 0,
where by Jensen’s inequality, achieves “=” if and only if vi = σ⊤v = C, ∀i for some constant C.
Next, we shall establish explicit formula for NTKs of the discriminator and the generator. For clarity, we will copy the formula for the discriminator and the generator used in our analysis:
fτ,l(y) = lim m→∞ 1√ m
m∑
r=1
vτ,lr max{W τ,lry , 0},
(27)
P gtl (y) = EX∼PXl [Ot(y|X)]
:= EX∼PXl
[ exp(U t⊤y x)∑
y′∈Y exp(U t⊤ y′ x)
] . (28)
Lemma 12. For the NTKs of the discriminators defined by Eq. (27), we have KD,l ≡ KD,1, 1 ≤ l ≤ L and 1|Y| is an eigenvector of KD,1.
Proof. For simplicity, we ignore the dependency on τ for the terms in the proof. First, by definition, we have
∂fl(y)
∂W lr = lim m→∞ 1√ m
m∑
r=1
vlrey1[W l ry ≥ 0],
∂fl(y)
∂vlr = lim m→∞ = 1√ m max{W lry, 0}
and therefore
Evl,W l∼N (0,I)
[ ∂fl(y)
∂W lr ⊤∂fl(y) ∂W lr
] =
lim m→∞
1
m Evl,W l∼N (0,I)
m∑
r=1
δyy′v 2 r1[W l ry ≥ 0]
= δyy′ 1
m
m∑
r=1
EW lry∼N (0,1)[1[W l ry ≥ 0]]
= 1
2 δyy′ .
On the other hand,
Evl,W l∼N (0,I)
[ ∂fl(y)
∂vl ⊤∂fl(y′) ∂vl
]
= 1
m Evl,W l
[ m∑
r=1
max{W lry, 0}max{W lry′ , 0} ]
=
{ Ev11 ,W 11 [ max{W 111, 0}2 ] if y = y′,
Ev11 ,W 11 [ max{W 111, 0} ]2 otherwise.
Therefore,
KD,l(y, y ′) =   ( 1 2 + Ev11 ,W 11 [ max{W 111, 0}2 ]) if y=y’,
Ev11 ,W 11 [ max{W 111, 0} ]2 otherwise.
Notice that the sum of every row in KD,l is
( 1
2 + Ev11 ,W 11
[ max{W 111, 0}2 ]) +
(|Y| − 1)Ev11 ,W 11 [ max{W 111, 0} ]2 ,
and thus 1|Y| is an eigenvector of KD,l.
Lemma 13. For the generator defined by Eq. (28), we have
KOt,x = EU1:|Y|∼N (0,I) [ (diag(Ox)−OxO⊤x )2 ] . (29)
Further, the null space of KOt,x is span{1|Y|}.
Proof. For simplicity, we ignore the dependency on t for the terms in the proof. By chain rule,
∂Ox(y)
∂Ux′y′ =
∂hy′(x)
∂Uxy′
∂Ox(y)
∂hy′(x)
= δxx′(O(y|x)δyy′ −O(y|x)O(y′|x))
As a result,
∑
d,y′
∂Ox(y)
∂Udy′ ⊤∂Ox(y′′) ∂Udy′
= ∑
y′ (Ox(y)δyy′ −Ox(y)Ox(y′))
(Ox(y ′′)δy′′y′ −Ox(y′′)Ox(y′))·
= ((diag(Ox)−OxO⊤x )2)yy′′
Take the expectation over U and put everything in matrix form, we obtain
KOx = EU∼N (0,I) [ (diag(Ox)−OxO⊤x )2 ] .
Next we shall study the null space of KOx . From Lemma 11, we have Hx := diag(Ox)−OxO⊤x is PSD with null space span{1|Y|}, and thus
v⊤KOxv = EU∼N (0,I) [ ∥Hxv∥2 ] ≥ 0,
with equality achieved if and only if
Hxv = 0, ∀x ∈ X ⇔ v ∈ span(1|Y|).
We are now ready to prove Theorem 4.
Proof. (Theorem 4) When the objective is MMD, the discriminator can be decomposed as
afτ (y) = fτ (y) =
L∑
l=1
fτ,l(yl),
we have
Lt(f) = L∑
l=1
EYl∼PYl [fl(Yl)]− EY ′l ∼PXl Ot [fl(Y ′ l )],
(30)
and the discriminator dynamic PDE Eq. (18) becomes:
∂τfτ,l = KD,l(P Y l − PXl Ot)⊤.
Without much loss of generality, suppose we initialize f0,l(y) ≡ 0 and stop training the discriminator after τmax steps. The solution for the discriminator PDE is then simply
fgt,l = τmaxKD,l(P Y l − PXOt)⊤. (31)
Plug this expression into the generator loss and apply Lemma 12, we obtain
Ct(gt) := τmax L∑
l=1
∥P Yl − PXl Ot∥2KD,l
= τmax∥P Y − PXOt∥2KD,1 ,
where ∥A∥K = √ Tr(AKA⊤) is the kernelized norm of A by kernel K. Further, plug Eq. (31) into the generator PDE Eq. (19), we obtain
∂tO ⊤ t,x = KOt,x
L∑
l=1
PXl (x)KD,l(P Y l − PXl Ot)⊤
= KOt,xKD,1(P Y − PXO)⊤P̃Xx ,
where P̃Xx is the x-th column of P X . Next, notice that
∂Ct ∂Ot,xy
=2τmaxKD,1(y, ·)(PXO − P Y )⊤P̃Xx =⇒ ∂Ct
∂Ot = PX⊤(PXO − P Y )KD,1.
Then apply the chain rule,
∂tCt = Tr (
∂Ct ∂Ot ⊤∂Ot ∂t
)
= ∑
x∈X Tr
( ∂Ct ∂Ot,x ∂Ot,x ∂t ⊤ ) =
−τmax ∑
x∈X ∥P̃X⊤x (P Y −PXOt)∥2KD,lKG,lKD,l .
Now, apply Lemma 12, we have
∂τf ⊤ τ,l1|Y|
=(P Yl − PXl Ot)KD,l1|Y| =λ(P Yl − PXl Ot)1|Y| = 1− 1 = 0
=⇒1|Y| ⊥ KD,l(P Yl − PXl Ot)⊤,
where λ is the eigenvalue of KD,l associated with 1|Y|, and thus
KD,l(P Y − PXOt)⊤P̃Xx ⊥ 1|Y|.
As a result, using Lemma 13, we conclude that the kernelized residual vector ∂τfτ,l is always perpendicular to the null space of the stepwise generator
NTK KOt,x for all 1 ≤ l ≤ L, x ∈ X, and thus
∥KD,l(P Y − PXOt)⊤P̃Xx ∥KG,l ≥λG∥KD,l(P Y − PXOt)⊤P̃Xx ∥2 ≥λGλD∥P Y − PXOt∥KD,1 ,
where
λG ≥ min 1≤l≤L λ|Y|−2(KG,l) > 0, λD ≥ λmin(KD,1) > 0.
Summing over x, we obtain
∂tCt ≤ −τmaxλGλD∥PX⊤(P Y − PXOt)∥2KD,1 .
Under the assumption that PXO = P Y has at least one solution, we have P Y − PXO is in the range space of PX , which implies
∥PX⊤(P Y − PXOt)∥2KD,1 ≥ λX∥P Y − PXOt∥2KD,1 ,
for some λX > 0. Put together the results, we can bound the convergence rate of the generator loss by
∂tCt ≤ −τmaxλGλDλXCt =⇒Ct ≤ C0e−τmaxλGλDλX t t→∞−−−→ 0,
which implies that limt→∞ PXOt = P Y .

heading: B Reproducibility checklist
text: Synthetic language creation To create a synthetic HMM language, we need to specify the initial probability vector π, the transition probability matrix T , the generator matrix O and the maximal length of the utterances L.
Initial probability: we create π by first uniformly randomly sampling each coefficient between [0, 1] and then normalizing the resulting vector by its sum.
Transition probability: for the asymptotic setting, for all three languages, we control the number of eigenvalues m of its transition matrix using a disjoint union of identical sub-graphs with m eigenvalues, with the remainder of the nodes being self-loops. The parameters and the procedure used to determine them are as follows:
• Circulant graph: only undirected cycles or equivalently, circulant graph with the action set {−1, 1}, are used. Since the distinct eigenvalues of an undirected n-cycle Cn are − cos ( 2πk n ) , k = 0, · · · , ⌊n−12 ⌋+ 1, we can
create a Markov graph with |X|N nodes and n ± 1 eigenvalues by a disjoint union of ⌊ |X|N2n−1⌋ C2n−1 graphs. In our phase transition experiment, we fix N = 2 and vary 10 ≤ |X| ≤ 14 and 2 ≤ n ≤ 20;
• De Bruijn graph: an undirected de Bruijn graph DB(k,m) is a graph with km nodes such that node i connects to any node j whose k-ary numerals v(i) and v(j) satisfies v2:m(i) = v1:m−1(j). Clearly, m is the in/out-degree of the graph. The eigenvalues of DB(k,m) are known to be cos ( iπ j ) , 0 ≤
i < j ≤ m + 1 (Delorme and Tillich, 1998). Therefore, we can create a Markov graph with |X|N nodes and at most n, n ≤ (⌊logk |X|N⌋+ 1)2/2 distinct eigenvalues by a disjoint union of |X| N
k √ 2m−1 DB(k, √ 2n − 1)
graphs. For the phase transition experiment, we set the in/out-degree of the de Bruijn subgraphs to be 2 and the N -gram size N = 3, and we vary 8 ≤ |X| ≤ 11 and 2 ≤ n ≤ 32 with a step size of 2 for the latter.
• Hypercube: an n-cube Qn is a graph with 2n nodes such that node i connects to any node j with Hamming distance between their binary numerals dH(b(i), b(j)) = 1. The eigenvalues of the adjacency matrix of Qn is 1 − 2kn , k = 0, · · · , n. Therefore, we can create a Markov graph with |X|N nodes and n ≤ ⌊N log2 |X|⌋ eigenvalues by a disjoint union of ⌊ |X|N2n ⌋ n-cubes. For the phase transition experiment, we fix N = 4, and vary 5 ≤ |X| ≤ 8 and 2 ≤ n ≤ 9.
In the finite-sample setting, we create transition matrices for phase transition experiments using two different setups:
• For the circulant graph, we vary its action set to be {1, · · · , d}, where d takes values from 2 to 81 with a step size of 8;
• For the other two graphs, we linearly interpolate between the underlying graph TG and its Hamiltonian cycle TC as
T = (1− w)TG + wTC , (32)
with a weight w ∈ [0, 1]. In particular, for the de Bruijn graph, the weight for the cycle w takes 10 different values equally spaced between [0, 1]; for the n-cube, the weight w takes 10 different values equally spaced between [0.98, 1].
Generator matrix O: set by assuming |X| = |Y| and randomly permuting the rows of the |X| × |X| identity matrix.
Sampling: in the asymptotic case, no sampling is needed and we simply set maximal length L = 20 for cycle graph and 10 for the other two graphs. For the finite-sample case, the synthetic speech and text datasets are created independently by sampling from the same HMM twice. For all three graphs, we sample nX = nY = 2560 utterances for both text and speech with L = 40 for the de Bruijn graph and L = 80 for the other two graphs.
Model architecture We use a one-layer linear generator with |X| input nodes and |Y| output nodes, with no bias. Next, for all experiments except the experiment on different generator averaging strategies, we use a one-layer CNN with |Y| input channels, 1 output channel and a 1 × L kernel with no bias. For the experiment on different averaging strategies, we use instead a sequence of 2-layer MLPs with 128 hidden nodes and ReLU activation function, one at each time step, as the discriminators. For all experiments, we disable the logits for special tokens and silences during training and testing.
Training setting SGD with a learning rate of 1.0 is used to train the discriminator, while Adam with a learning rate of 0.005 is used to train the generator. The dataset is used as a single batch for all experiments, though we do not observe any significant drop in performance using smaller batch sizes. No weight decays or dropouts is used. Further, we alternatively train the generator and discriminator 1 epoch each, and reset the discriminator weight to 0 for the linear case and to random Gaussian weights using Xavier initialization in the nonlinear case. All experiments are conducted on a single 12GB NVIDIA GeForce GTX 1080Ti GPU.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 7
7 A2. Did you discuss any potential risks of your work? It is theoretical paper and has no significant risks per se as far as the authors concern
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 1,2,3,4
3 B1. Did you cite the creators of artifacts you used? Section 5, Appendix B
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? It uses open source and publicly available toolkit/data and the license and terms are listed in their websites
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We believe we use all the artifacts with their intended purposes
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Our data are synthetic and do not contain personal information
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? The documentations are available on the official websites of the artifacts
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4, Appendix B
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4, Appendix B
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4, Appendix B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: This will be left as future work. With the help of Lemma 6, we can reduce Theorem 2 to the following theorem. Theorem 5. Let An ∈ Rn×n be a symmetric Markov random matrix matrix and v be an eigenvector with eigenvalue λ = O(nγ−1), then for any fixed C > 0, there exists some B > max{4γC + 3γ, 4γ + 1} such that
ρn−B (v) ≤ n−C ,
with probability at least 1 − O(exp(−α0n)) for some α0 depending on B. Similar to the proof for the perturbed symmetric matrices in (Nguyen et al. , 2017), we reduce Theorem 5 to the following. Theorem 6. Let v be the eigenvector and B be the constant defined in Theorem 5. Then for any n−B ≤ δ ≤ n−B/2, we have with probability O(exp(−α0n)),
n−C ≤ ρnγδ(v) ≤ n0. 49ρδ(v). (24)
To show that Theorem 6 implies Theorem 5, we prove the contrapositive of the statement, that is, if ρn−B (v) > n−C , then there exists n−B ≤ δ ≤ n−B/2 such that Eq. 24 holds with probability at least 1−O(exp(−α0n)). To construct such δ, let
δ0 := n −B
δj+1 := n γδj ,
for j = 0, · · · , J − 1 with J = ⌊B/2γ⌋. By construction, we have
n−B = δ0 ≤ δj ≤ δJ ≤ n−B/2
ρδj (v) ≥ ρδ0(v) ≥ n−C. Suppose Eq. 24 does not hold for any δ := δj , or otherwise the result follows, we have
ρδJ (v) ≥ n0. 49Jρn−B (v) ≥ n0. 49⌊B/2γ⌋−C > 1, if B ≥ 4γC + 3γ, which contradicts the fact that ρδJ (v) ≤ 1. As a result, there has to exist some j such that Eq. 24 holds. Again similar to the perturbed symmetric matrix case in (Nguyen et al. , 2017), we divide the proof of Theorem 6 into the compressible case and the non-compressible case. For the compressible case, we first prove the following lemma. Lemma 7. Suppose v is an eigenvector of a symmetric Markov random matrix An := D−1n Wn with adjacency matrix Wn and the same properties defined in Theorem 2, and suppose there exists δ ∈ [n−B, n−B/2] such that ρδ,α(v) ≥ (αn)−1/2+ϵ, we have with probability O(exp(−α0n)), n−C ≤ ρnγδ(v) ≤ n0. 49ρδ(v). Proof. Using concentration inequalities, we have with probability at least 1 − O(exp(−α2n)) for some α2 > 0,
dii = O ( nγ+ 1 2 ) , 1 ≤ i ≤ n (25)
Further, since ρδ,α(v) ≥ (αn)−1/2+ϵ, by Lemma 2, we have v is (O(α), δ) compressible, and thus there exists I of of size O(αn) such that vi > δ only if i ∈ I. Without loss of generality, let I = {n − k, · · · , n} for k = O(αn) and E[Aij ] = 1. Further, split v = [v′⊤, v′′⊤]⊤, then by definition of eigenvalues and eigenvectors, [ Wn−k F F⊤ Wk ] [ v′ v′′ ] = λ [ Dn−k 0 0⊤ Dk ] [ v′ v′′ ]. Reading off the first line of the matrix equation, we have
∥Fv′′∥2 = ∥(Wn−k − λDn−k)v′∥2 ≤ ∥Wn−kv′∥2 + ∥λDn−kv′∥2. Notice that assuming Eq. 25 and Eq. 22 occur, we have that all elements v′i of v
′ have |v′i| < δ, therefore ∥v′∥2 ≤ δn−1/2, therefore
∥Wn−kv′∥2 ≤ δn1/2 max v:∥v∥2=1 ∥Wv∥2
= O(n−B/2+1/2+γ)
Furthermore, if we assume that Eq. (22) and Eq. (25) occur, then
∥λDn−kv′∥2 = O(nγ−1 · nγ+1/2 · δn1/2) = O(n−B/2+2γ). Thus, using the fact that B ≥ 4γ + 1,
∥Fv′′∥2 = O(n−B/2+2γ) = O(n−1/2). On the other hand, using a standard epsilonnet argument, with probability at least 1 − O(exp(−α3n)),
inf w∈Rk:∥w∥=1
∥Fw∥2 ≥ n−1/2