On the Uncomputability of Partition Functions in Energy-Based Sequence Models

1 Introduction . Modeling discrete sequences is central to natural language processing and bioinformatics . Many common parametric sequence models ? ̃ are ( or can be cast as ) energy-based ( LeCun et al. , 2006 ) : they yield a weight ? ̃ ( x ) for any given string x . Although energy-based models ( EBMs ) of sequences need not represent probability distributions , they have been proposed as such to counter the inexpressivity of autoregressive sequence models ( Bakhtin et al. , 2021 ; Lin et al. , 2021 , §4.1 ) . EBMs with finite partition functions / ? ̃ , ∑ x ? ̃ ( x ) ∈ R > 0 define distributions ? ( x ) , ? ̃ ( x ) // ? ̃ over strings , which are often queried under the principle of probabilistic inference ( Ghahramani , 2015 ) .1 Energy-based sequence models are often parametrized as neural models . Each parameter vector ) in some parametric neural network family ⊆ R3 identifies a parametric model ? ̃ ) , which then defines a parametric distribution over strings ? ) , assuming / ? ̃ ) exists ( Chen et al. , 2018 ) . Contemporary neural networks have been shown to be very powerful . In particular , some popular parametric neural sequence model families , such as RNNs ( Siegelmann & Sontag , 1995 ) and Transformers ( Bhattamishra et al. , 2020 ; Pérez et al. , 2021 ) , have been formally shown to recognize recursively enumerable languages : given any Turing machine `` , there is a parameter vector ) ∈ such that the parametrized sequence model # ) recognizes the same language as `` does . In other words , these sequence model families are Turing-complete . It is therefore intuitive that energy-based sequence models , backed by such powerful neural networks , form expressive families of string distributions ( §3.1 ) : for any decision algorithm `` , there is a parameter vector ) '' in a Turing-complete parametric family , such that ? ) '' exists , and ? ) '' ( x ) is high if and only if `` accepts x . It would seem assuring to work with such expressive family of sequence distributions , : assuming the true string probabilities indeed can be computed in polytime , is well-specified . Moreover , one may assume that given we can sample from ? ) '' , we would be able to use consistent estimators to find ) ′ ∈ , where ) ′ ≈ ) '' . Unfortunately , we find that with such an expressive distribution family , whether the identifiability assumption holds— required for most consistent estimators— itself is undecidable ( Turing , 1937 ) . 1Many popular energy-based sequence models compute normalized probabilities directly ( i.e. , ? ̃ ( x ) = ? ( x ) ) ( Jelinek , 1980 ; Katz , 1987 ; Bengio et al. , 2003 ; Brown et al. , 2020 , inter alia ) . This makes both training and querying with string prefixes much easier , at the cost of expressiveness ( Lin et al. , 2021 ) . Moreover , model selection on any held-out data is also undecidable . Even worse , as long as a parametric family ′ contains one specific vector ) ′ ∈ , model selection will not be possible for ′ either— even when ′ itself is not necessarily expressive ( e.g. , ′ can be fixed-size Transformer EBMs , which can not parametrize all EBMs that require more parameters ) . We construct one such ‘ pathological ’ distribution ? ) ′ as a Transformer EBM . These negative results stem from the uncomputability and inapproximability of / . A main technical contribution of this paper is that there is no algorithm ( either deterministic or randomized ) that can approximate / well . Since statistical procedures are also algorithms , common randomized estimates are not useful in this scenario , either—however , we will see that for less expressive model families , such uncomputability issues do not arise . Our negative results are summarized in Table 1 . To ensure that model selection is still possible ( such that we can compare different parameter vectors with confidence ) ,2 we have no choice but to resort to less expressive string distributions . The paper is structured as follows . In §2 we review definitions and known results of weighted languages , sequence model families , and formalize weighted Turing machines , EC-complete parametric families , and computable estimates . In §§3–5 we describe our main technical results : there exist pathological EBM sequence models that have uncomputable partition functions , which can not be approximated well under stochastic estimates , and do not have asymptotic estimates that have any good guarantees . In §6 we that argue our negative results make model selection impossible for expressive model families , and discuss why common estimation methods fail . Finally , we discuss three ‘ palliative ’ parametrization choices in §7 , which all guarantee computable partition functions , at the cost of expressiveness . 2 Background . 2.1 Energy-based sequence models . Energy-based models ( EBMs ) of sequences ( LeCun et al. , 2006 ; Rosenfeld et al. , 2001 ; Sandbank , 2008 ; Huang et al. , 2018 ; Bakhtin et al. , 2021 ) are a family of discrete sequence models . Given a string x ∈ +∗ over a finite vocabulary + , an EBM ? ̃ computes string weight ? ̃ ( x ) , but not the ( normalized ) string probability ? ( x ) = ? ̃ ( x ) /∑x ? ̃ ( x ) . In this work , we focus on EBMs whose weight is efficiently computable , i.e. , polytime in the length of x . Following Bakhtin et al . ( 2021 ) and Lin et al . ( 2021 ) , we wish to use our EBMs as probability distributions ; this requires normalization . While EBMs are often intractable to normalize ( e.g. , the Ising model ) , the infinite domain of finite strings opens the door to uncomputable probabilities , which invite the impossibility of model selection and comparison . 2.2 Weighted languages . EBMs give weights to strings . Here we formally characterize these weighted strings as a weighted language . An unweighted language ! ⊆ +∗ is a set of finite strings x over a finite vocabulary + . A weighted language ? ̃ is a function ? ̃ : +∗ → R≥0.3 In this work , we discuss Boolean languages , such that + = B , { 0 , 1 } . We also focus on weighted languages where distributions over strings exist . We say a weighted language ? ̃ is normalizable when / ? ̃ , ∑ x∈B∗ ? ̃ ( x ) ∈ R > 0 : the sum over all possible 2Of course , model selection ( and computing / ? ̃ ) are not always needed— for example , simply deciding whether ? ̃ ( x1 ) > ? ̃ ( x2 ) for two given strings x { 1,2 } . In such case the uncomputability issues we discuss are not a concern . 3With a slight abuse of notation , we define support ( ? ̃ ) , { x : ? ̃ ( x ) > 0 } . strings is finite and positive . / ? ̃ is also called the partition function of ? ̃.4 We can then normalize ? ̃ into a distribution ? over B∗ such that ? ( x ) = ? ̃ ( x ) // ? ̃ , and thereby ∑ x∈B∗ ? ( x ) = 1 . The efficiently computable ( weighted ) languages ( EC ; Lin et al. , 2021 ) are those weighted languages ? ̃ , where a string ’ s weight ( which must be non-negative ) is a polytime function of the string . Weighted languages defined by most EBM sequence models fall into this class , since they score every string in finite time ( and usually in polytime ) ; and the scores ultimately are computed by some algorithm . By definition , string weights of an EC language can be obtained as a side product from a Turing machine that recognizes x ( in polytime ) . However , Lin et al . ( 2021 ) did not precisely describe how a Turing machine maps input string x to its ( rational ) weight ? ̃ ( x ) in finite time . In this work , such a construction ( out of many possible ones ) is necessary , as we need to show that this string weighting can be done by parametric sequence model families ( see Appendix A ) . 2.3 Locally normalized distributions . A popular subclass of weighted languages is locally normalized weighted languages ( LN ) , where conditional local distributions given prefix x̂ : ? ( · | x̂ ) can be computed in finite time . Since they automatically define a distribution over strings , we use the term locally normalized distributions interchangeably . If ? ( · | x̂ ) can be computed in polynomial time , we call such distributions efficiently locally normalized distributions ( ELN ) — this is the weighted language class of most autoregressive models , just as its superset EC is the weighted language class of most energy-based sequence models . Locally normalized distributions over sequences have ? ̃ ( x ) = ? ( x ) , and / = ∑x∈B∗ ? ̃ ( x ) = 1 . They are consistent : the probability that a string is infinitely long under such distributions is zero . Equivalently , given any n > 0 , we can approximate / with a finite sum of string weights : Proposition 1 ( Consistency of LN distributions ( Booth & Thompson , 1973 ; Chen et al. , 2018 ) ) . Let ? ∈ LN be a locally normalized distribution over strings . All strings of a given length or longer have their probabilities bounded . That is , for all positive real numbers n , there is a length = at which all strings x of length at least = have ? ( x ) < n . The consistency property of locally normalized distributions implies that they have an exact sampling procedure that almost surely terminates in finite time . Therefore , they ( and in particular ELN distributions ) are an attractive choice as proposal distributions in parameter estimation procedures that alternate between expectation and maximization steps ( Dempster et al. , 1977 ) . 2.4 EC-complete parametric families . We have introduced energy-based sequence models ( §2.1 ) and their characterization as weighted languages ( §2.2 ) . However , we usually do not work with Turing machines directly in machine learning . Here the most common models of computation are ( neural ) sequence model families , such as RNNs and Transformers . While those computational models can appear quite dissimilar to state-machine-based models of computation ( e.g. , Turing machines ) , they have been shown to possess the same computation power ( Siegelmann & Sontag , 1995 ; Pérez et al. , 2021 ) . That is , they are Turing-complete . Just as we extend the definition of Turing machines to weighted Turing machines , we likewise formalize weighted sequence model families . We thus introduce EC-complete parametric families as a sequence model counterpart of the weighted language class EC . At a high level , a parametric family is EC-complete if given any ? ̃ ∈ EC ( as a description of a weighted Turing machine ) , we can construct a parameter vector ) ∈ ⊆ Q∗ which defines ? ̃ ’ s corresponding sequence model . The model produces an output embedding which we then decide in polytime to be either a ‘ halting embedding ’ or not . If it is , we can then extract the weight ? ̃ ( x ) . ( A rigorous exposition is in Appendix H. ) We show that with a slight modification to positional embeddings , the family of arbitrary precision , hard attention Transformers defined in Pérez et al . ( 2021 ) is EC-complete:5 Theorem 1 . The class of one-encoder-layer four-decoder-layer Transformer networks with positional encodings ( = , 1/= , 1/=2 , 2= ) is EC-complete . 4We use the convention that / ? ̃ is the partition function of ? ̃ , and / @ ̃ of @ ̃ , etc . The subscript is omitted in unambiguous cases . 5All proofs omitted from the main text are included in the appendices . Proof sketches for major theorems are in the main text . Proof sketch . We extend the construction from Pérez et al . ( 2021 ) , adding an additional layer to accumulate the string weight over time steps .

Summary: The paper studies the trade-off between expressiveness and computability (of the partition function) for energy-based sequence models. The theoretical results show that the high expressively of unrestricted energy based models comes at the cost of un-computability and in-approximability of the partition function. These negative results further show that rejection and importance sampling are not a panacea either.