Title: A Heavy-Tailed Algebra for Probabilistic Programming
Venue: NeurIPS 2023
Accepted: yes
Keywords: probabilistic programming

static analysis

heavy tails

monte carlo

mcmc

variational inference
Abstract: Despite the successes of probabilistic models based on passing noise through neural networks, recent work has identified that such methods often fail to capture tail behavior accurately---unless the tails of the base distribution are appropriately calibrated.  To overcome this deficiency, we propose a systematic approach for analyzing the tails of random variables, and we illustrate how this approach can be used during the static analysis (before drawing samples) pass of a probabilistic programming language (PPL) compiler.  To characterize how the tails change under various operations, we develop an algebra which acts on a three-parameter family of tail asymptotics and which is based on the generalized Gamma distribution.  Our algebraic operations are closed under addition and multiplication; they are capable of distinguishing sub-Gaussians with differing scales; and they handle ratios sufficiently well to reproduce the tails of most important statistical distributions directly from their definitions.  Our empirical results confirm that inference algorithms that leverage our heavy-tailed algebra attain superior performance across a number of density modeling and variational inference (VI) tasks.

A Heavy-Tailed Algebra for Probabilistic Programming

Abstract

Despite the successes of probabilistic models based on passing noise through neural networks, recent work has identified that such methods often fail to capture tail behavior accurately-unless the tails of the base distribution are appropriately calibrated. To overcome this deficiency, we propose a systematic approach for analyzing the tails of random variables, and we illustrate how this approach can be used during the static analysis (before drawing samples) pass of a probabilistic programming language (PPL) compiler. To characterize how the tails change under various operations, we develop an algebra which acts on a three-parameter family of tail asymptotics and which is based on the generalized Gamma distribution. Our algebraic operations are closed under addition and multiplication; they are capable of distinguishing sub-Gaussians with differing scales; and they handle ratios sufficiently well to reproduce the tails of most important statistical distributions directly from their definitions. Our empirical results confirm that inference algorithms that leverage our heavy-tailed algebra attain superior performance across a number of density modeling and variational inference (VI) tasks.

1 Introduction

Within the context of modern probabilistic programming languages (PPLs), recent developments in functional programming [51], programming languages [3], and deep variational inference (VI) [4] combine to facilitate efficient probabilistic modelling and inference. But despite the broadening appeal of probabilistic programming, common pitfalls such as mismatched distribution supports [32] and non-integrable expectations [53, 55, 60] remain uncomfortably commonplace and remarkably challenging to address. In particular, heavy-tailed distributions arise in a wide range of statistical applications and are known to present substantial technical challenges [37, 55, 60]. Recent innovations aiming to improve PPLs have automated verification of distribution constraints [32], tamed noisy gradient estimates [16] as well as unruly density ratios [53, 55], and approximated high-dimensional distributions with non-trivial bulks [39]. To address the issue of heavy-tailed targets, approaches which initialize with non-Gaussian tails have been proposed [25, 33]. However, these methods typically require the use of optimization and/or sampling strategies to estimate the tails of the target distribution. Such strategies are often unstable, or fail to allow for a sufficiently wide array of possible tail behaviours.

❅❜♠✳/a114❛♥❞♦♠❴✈❛/a114✐❛❜❧❡

❞❡❢

♥✭✮✿

/a114❡/a116✉/a114♥

❍❛❧❢◆♦/a114♠❛❧✭✶✳✵✮

❅❜♠✳/a114❛♥❞♦♠❴✈❛/a114✐❛❜❧❡

❞❡❢

①✭✮✿

/a114❡/a116✉/a114♥

❍❛❧❢◆♦/a114♠❛❧✭✶✳✵✮

❅❜♠✳❢✉♥❝/a116✐♦♥❛❧

❞❡❢

①✷✭✮✿

/a114❡/a116✉/a114♥

①✭✮ ✴

♥✭✮

-→

1.0

HalfNormal

sigma

Sample

operand

Sample

operand

/

left

right

Query

operator

GGA

-- - →

R

2

-→

10

2

10 1

10

6

10

4

10

2

10 0

Fit

Samples

(1)

(2)

(3)

(4)

Motivated by this, we introduce the first procedure for static analysis of a probabilistic program that automates analysis of target distributions' tails. In addition, we show how tail metadata obtained from this procedure can be leveraged by PPL compilers to generate inference algorithms which mitigate a number of pathologies. For example, importance sampling estimators can exhibit infinite variance if the tail of the approximating density is lighter than the target; most prominent black-box VI methods are incapable of changing their tail behaviour from an initial proposal distribution [25, 33]; and Monte-Carlo Markov Chain (MCMC) algorithms may also lose ergodicity when the tail of the target density falls outside of a particular family [42]. All of these issues could be avoided if the tail of the target is known before runtime.

To classify tail asymptotics, we propose a three-parameter family of distributions which is closed under most typical operations. This family is based on the generalized Gamma distribution (Equation (2)), and it interpolates between established asymptotics on sub-Gaussian random variables [31] and regularly varying random variables [35]. Algebraic operations on random variables can then be lifted to computations on the tail parameters. This results in a heavy-tailed algebra that we designate as the generalized Gamma algebra (GGA) . Through analyzing operations like X + Y , X 2 , and X/Y at the level of densities (e.g., additive convolution p X ⊕ p Y ), the tail parameters of a target density can be estimated from the parameters of any input distributions using Table 1.

Operationalizing our GGA, we propose a tail inferential static analysis strategy analogous to traditional type inference. GGA tail metadata can be used to diagnose and address tail-related problems in downstream tasks, such as employing Riemannian-manifold methods [17] to sample heavy tails or preemptively detect unbounded expectations. Here, we consider density estimation and VI where we use the GGA-computed tail of the target density to calibrate our density approximation. When composed with a learnable Lipschitz pushforward map (Section 3.2), the resulting combination is a flexible density approximator with tails provably calibrated to match those of the target.

Contributions. Here are our main contributions.

We propose the generalized Gamma algebra (GGA) as an example of a heavy-tailed algebra for probability distributions. This extends prior work on classifying tail asymptotics, while including both sub-Gaussian / sub-exponentials [31] as well as power-law / Pareto-based tail indices [11]. Composing operations outlined in Table 1, one can compute the GGA tail class for downstream random variables of interest.

We implement GGA as abstract interpretation during the static analysis phase of a PPL compiler. This unlocks the ability to leverage GGA metadata in order to better tailor MCMC and VI algorithms produced by a PPL.

Finally, we demonstrate that density estimators which combine our GGA tails with neural networks (autoregressive normalizing flows [39] and neural spline flows [15]) simultaneously achieves calibrated tails without sacrificing good bulk approximation.

2 The Generalized Gamma Algebra

First, we formulate our heavy-tailed algebra of random variables that is closed under most standard elementary operations (addition, multiplication, powers). The central class of random variables under consideration are those with tails of the form in Definition 1.

Definition 1. A random variable X is said to have a generalized Gamma tail if the Lebesgue density of | X | satisfies

for some c > 0 , ν ∈ R , σ > 0 and ρ ∈ R . Denote the set of all such random variables by G .

Consider the following equivalence relation on G : X ≡ Y if and only if 0 < p | X | ( x ) /p | Y | ( x ) < + ∞ for all sufficiently large x . The resulting equivalence classes can be represented by their corresponding parameters ν, σ, ρ . Hence, we denote the class of random variables X satisfying Equation (1) by ( ν, σ, ρ ) . In the special case where ρ = 0 , for a fixed ν < -1 , each class ( ν, σ, 0) for σ > 0 is equivalent, and is denoted by R | ν | , representing regularly varying tails. Our algebra operates on these equivalence classes of G , characterizing the change in tail behaviour under various operations. To incorporate tails which lie outside of G , we let R 1 incorporate super-heavy tails , which denote random variables with tails heavier than any random variable in G . All operations remain consistent with this notation. Likewise, we let L denote super-light tails , which are treated in our algebra as a class where ρ = + ∞ (effectively constants).

Equation (1) and the name of the algebra are derived from the generalized Gamma distribution.

Definition 2. Let ν ∈ R , σ > 0 , and ρ ∈ R \{ 0 } be such that ( ν +1) /ρ > 0 . Anon-negative random variable X is generalized Gamma distributed with parameters ν, σ, ρ if it has Lebesgue density

where c ν,σ,ρ = ρσ ( ν +1) /ρ / Γ(( ν +1) /ρ ) is the normalizing constant.

The importance of the generalized Gamma form arises due to a combination of two factors:

(i) The majority of interesting continuous univariate distributions with infinite support satisfy Equation (1), including Gaussians ( ν = 0 , ρ = 2 ), gamma/exponential/chi-squared ( ν > -1 , ρ = 1 ), Weibull/Frechet ( ρ = ν +1 ), and Student T /Cauchy/Pareto ( R ν ). A notable exception is the log-normal distribution (see Example 8 in Appendix C).

(ii) The set G is known to be closed under additive convolution, positive powers, and Lipschitz functions. We prove it is closed under multiplicative convolution as well. This covers the majority of elementary operations on independent random variables. Reciprocals, exponentials and logarithms comprise the only exceptions, however, we will introduce a few 'tricks' to handle these cases as well.

The full list of operations in GGA is compiled in Table 1 and described in detail in Appendix A. GGA classes for common probability distributions are provided in Appendix B. All operations in the GGA can be proven to exhibit identical behaviour with their corresponding operations on random variables, with the sole exception of reciprocals (marked by † ), where additional assumptions are required. The asymptotics for operations marked with an asterisk are novel to this work. For further details, refer to Appendix A.

Repeated applications. Provided independence holds, composition of operations in the GGA remain consistent unless one applies Lipschitz functions, logarithms, or exponentials. If one of these operations is applied, the tail becomes an upper bound, which remains consistent under addition, multiplication, and powers, but not reciprocals. Given that we are working with a fixed class of tails, such behavior is inevitable, and it is possible to perform a sequence of operations for which the tail no longer becomes accurate.

Posterior distributions. A primary application of PPLs is to perform Bayesian inference. To cover this use case, it is necessary to prescribe a procedure to deal with posterior distributions . Consider a setup where a collection of random variables X 1 , . . . , X n are dependent on corresponding latent random elements Z 1 , . . . , Z n as well as a parameter θ through functions f i by X i = f i ( θ, Z i ) . For

̸

simplicity, we assume that each f i = f i,k ◦ f i,k -1 ◦ · · · ◦ f i, 1 where each f ij is an elementary operation in Table 1. To estimate the tail behaviour of θ conditioned on X , we propose an elementary approach involving inverses. For each operation f ij , if f ij is a power, reciprocal, or multiplication operation, let R ij be given according to the following:

̸

Letting f -1 i ( x, z ) denote the inverse of f i in the first argument, we show in Appendix A,

where π denotes the prior for θ . Since the inverse of a composition of operations is a composition of inverses, the tail of f -1 i ( x , Z i ) can be determined by backpropagating through the computation graph for X i and sequentially applying inverse operations. Consequently, the tail behaviour of the posterior distribution for one parameter can be obtained using a single backward pass. Posterior distributions for multiple parameters involve repeating this procedure one parameter at a time, with other parameters fixed.

3 Implementation

3.1 Compile-time static analysis

To illustrate an implementation of GGA for static analysis, we sketch the operation of the PPL compiler at a high-level and defer to the code in Supplementary Materials for details. A probabilistic program is first inspected using Python's built-in ast module and transformed to static single assignment (SSA) form [43]. Next, standard compiler optimizations (e.g., dead code elimination, constant propagation) are applied and an execution of the optimized program is traced [4, 58] and accumulated in a directed acyclic graph representation. Abreadth-first type checking pass, as seen in Algorithm 1, completes in linear time, and GGA results may be applied to implement computeGGA() using the following steps:

̸

If a node has no parents, then it is an atomic distribution and its tail parameters are known (Table 5);

Otherwise, the node is an operation taking its potentially stochastic inputs (parents) to its output. Consult Table 1 for the output GGA tails.

3.2 Representative distributions

For each ( ν, σ, ρ ) , we make a carefully defined choice of p on R such that if X ∼ p , then X ≡ ( ν, σ, ρ ) . This way, any random variable f ( X ) , where f is 1 -Lipschitz, will exhibit the correct tail, and so approximations of this form may be used for VI or density estimation. Let X ≡ ( ν, σ, ρ ) and 0 < ϵ ≪ 1 denote a small parameter such that tails e -x ϵ are deemed to be 'very heavy' (we chose ϵ = 0 . 1 ). Inspired by ideas from implicit renewal theory [5], our candidate distributions are as follows.

( ρ ≤ 0 ) If ρ ≤ 0 , then p X ( x ) ∼ cx -| ν | . One such density is the Student t distribution , in this case, with | ν | -1 degrees of freedom if ν < -1 (generate X ∼ StudentT ( | ν | -1)) .

( ρ > ϵ ) For moderately sized ρ > 0 , the symmetrization of the generalized Gamma density (2).

( ρ ≤ ϵ ) If X ≡ ( ν, σ, ρ ) where ρ is small, then X will exhibit much heavier tails, and the generalized Gamma distribution in Case 1 will become challenging to sample from. In these cases, we expect that the tail of X should be well represented by a power law. The generalized Gamma density (Equation (2)) satisfies E X r = σ -r/ρ Γ( ν +1+ r ρ ) / Γ( ν +1 ρ ) for r > 0 . Let α > 0 be such that E X α = 2 . By Markov's inequality, the tail of X satisfies P ( X > x ) ≤ 2 x -α . Therefore, we can represent tails of this form by the Student t distribution with α +1 degrees of freedom (generate X ∼ StudentT ( α ) ).

3.3 Bulk correction by Lipschitz mapping

While a representative distribution will exhibit the desired tails, the target distribution's bulk may be very different from a generalized Gamma and result in poor distributional approximation. To address this, we propose splicing together the tails from a generalized Gamma with a flexible density approximation for the bulk. While many combinations are possible, in this work we rely on the Lipschitz operation in the GGA (Theorem 2) and post-compose neural spline flows [15] (which are identity functions outside of a bounded interval hence 1 -Lipschitz) after properly initialized generalized Gamma distributions. Optimizing the parameters of the flow results in good bulk approximation while simultaneously preserving the tail correctness guarantees attained by the GGA.

0

5

10

0

.

5

1

.

0

1

.

5

2

.

0

Density

(i)

candidate

w/

flow

target

10

0

10

1

10

2

10

-

5

10

-

3

10

-

1

10

1

Density

(ii)

Example 1. Let A ∈ R k × k , x, y ∈ R k , with x i , y i , A ij iid ∼ N ( -1 , 1) . The distribution of x ⊤ Ay = ∑ i,j x i A ij y j is a convolution of normal-powers [21] and lacks a closed form expression. Using the GGA (Table 1), one can compute its tail parameters to be ( k 2 -1 , 3 2 , 2 3 ) . The candidate given by the GGA representative distribution (Section 3.2) is a gamma distribution with correct tail behaviour, but is a poor approximation otherwise. A learnable Lipschitz bijection is optimized to correct the bulk approximation (Figure 2(i)). From the Lipschitz property, the slope of the tail asymptotics in log-log scale remains the same before and after applying the flow correction (Figure 2(ii)): the tails are guaranteed to remain calibrated.

Example 2. Consider ∑ 4 i =1 X 2 i where X i ∼ StudentT ( i ) . While we are not aware of a closed-form expression for the density, this example is within the scope of our GGA. Empirical results illustrate that our method (Figure 3(i)) accurately models both the bulk and the tail whereas Gaussian-based Lipschitz flows (Figure 3(ii)) inappropriately impose tails which decay too rapidly.

4 Experiments

We now demonstrate that GGA-based density estimation yields improvements in tail estimation across several metrics. Our experiments consider normalizing flows initialized from (i) the parametric family defined in Section 3.2 against (ii) a normal distribution (status quo). To further contrast the individual effect of using a GGA base distribution over standard normals against more expressive pushforward maps [15], we also report ablation results where normalizing flows are replaced by affine transforms, as originally proposed in [30]. All experiments are repeated for 100 trials, trained to convergence using the Adam optimizer with manually tuned learning rate. Additional details are available in Appendix D. All target distributions in this section are expressed as generative PPL programs: Cauchy using a reciprocal normal; Chi2 (chi-squared) using a sum of squared normals; IG (Inverse Gamma) using a reciprocal exponential; normal using a sum of normals; and StudentT using a normal and Cauchy ratio. Doing so tasks the static analyzer to infer the target's tails and makes the analysis non-trivial.

Our results in the following tables share a consistent narrative where a GGA base distribution rarely hurts and can significantly help with heavy tailed targets. Standard evaluation metrics such as negative cross-entropy, ELBO, or importance-weighted autoencoder bounds [6] do not evaluate the quality of tail approximations. Instead, we consider diagnostics which do: namely, an estimated tail exponent

(i) GGA

(ii) Gaussian

10000

10000

7500

7500

W

low

5000

5000

F

F

2500

2500

0

0

0

200040006000800010000

0

200040006000800010000

Target

Target

ˆ α , and the Pareto ˆ k diagnostic [60]. Except for when targets are truly light tailed ( α = ∞ in Chi2 and normal), GGA-based approximations are the only ones to reproduce appropriate GPD tail index ˆ α in density estimation and achieve a passing ˆ k below 0 . 2 in VI. Less surprising is the result that adding a flow improved approximation metrics, as we expect the additional representation flexibility to be beneficial.

Density Estimation. Given samples { x i } N i =1 from a target density p , we minimize a Monte-Carlo estimate of the cross entropy H ( p, q ) = -E p [log q ( X )] ≈ -1 N ∑ N i =1 log q ( x i ) . The results are shown in Table 2 and Table 3 along with power-law tail index estimates ˆ α [11]. Closeness between the target Pareto tail index α [11] and its estimate ˆ α in q ( x ) suggest calibrated tails. Overall, we see that normal (resp., Cauchy) based flows fails to capture heavy (resp., light) tails, while GGA-based flows yield good tail approximations (lower NLL, ˆ α closer to target) across all cases.

Variational Inference. For VI, the bulk is corrected through the ELBO optimization objective E q log p ( X ) q ( X ) ≈ 1 N ∑ N i =1 log p ( x i ) q ( x i ) , x i ∼ q . Since the density p must also be evaluated, for simplicity, experiments in Table 4 use closed-form marginalized densities for targets. The overall trends also show that GGA yields consistent improvements; the ˆ k diagnostic [60] indicates VI succeeds ( ˆ k ≤ 0 . 2 ) when a GGA with appropriately matched tails is used and fails ( ˆ k > 1 ) when Gaussian tails are erroneously imposed.

Bayesian linear regression. As a practical example of VI applied to posterior distributions, we consider the setting of one-dimensional Bayesian linear regression (BLR) with conjugate priors, defined by the likelihood y | X,β,σ ∼ N ( Xβ,σ 2 ) with a Gaussian prior β | σ 2 ∼ N (0 , σ 2 ) on the coefficients, and an inverse-Gamma prior with parameters a 0 and b 0 on the residual variance σ 2 . The posterior distribution for β conditioned on σ 2 and X,y is Gaussian. However, conditional on the pair ( X,y ) , σ 2 is inverse-Gamma distributed with parameters a 0 + n 2 and b 0 + 1 2 ( y ⊤ y -µ ⊤ Σ µ )) , where µ = Σ -1 X ⊤ X ˆ β for ˆ β the least-squares estimator, and Σ = X ⊤ X + I . Since σ 2 is positive, it is typical for PPL implementations to apply an exponential transformation. Hence, a Lipschitz normalising flow starting from a Gaussian initialization will inappropriately approximate the inverse Gamma distributed p ( σ 2 | X,y ) with log-normal tails. On the other hand, Lipschitz flows starting from a GGA reference distribution will exhibit the correct tails. We assess this discrepancy in Figure 4 under an affine transformation on four subsampled datasets: super (superconductor critical temperature prediction dataset [23] with n = 256 and d = 154 ); who (life expectancy data from the World Health Organisation in the year 2013 [41] with n = 130 , d = 18 ); air (air quality data [14] with n = 6941 , d = 11 ); and blog (blog feedback prediction dataset [7] with n = 1024 , d = 280 ). In Figure 4(i), the GGA-based method seems to perfectly fit to the targets, while in Figure 4(ii), the standard Gaussian approach fails to capture the tail behaviour.

(i)

(ii)

10

1

10 0

10 1

10 2

200

150

100

50

0

log10

p

(

|

X

,

y

)

supers

whos

air

blogs

10

1

10 0

10 1

10 2

200

150

100

50

0

log10

p

(

|

X

,

y

)

super

who

air

blog

Invariant distribution of SGD. For inputs X and labels Y from a dataset D , the least squares estimator for linear regression satisfies ˆ β = min β 1 2 E X,Y ∼D ( Y -Xβ ) 2 . To solve for this estimator, one can apply stochastic gradient descent (SGD) sampling over independent X k , Y k ∼ D to obtain the sequence of iterations

for a step size δ > 0 . For large δ , the iterates β k typically exhibit heavy-tailed fluctuations [24]. In this regard, this sequence of iterates has been used as a simple model for more general stochastic optimization dynamics [22, 24]. In particular, generalization performance has been tied to the heaviness of the tails in the iterates [47]. Here, we use our algebra to predict the tail behaviour in a simple one-dimensional setting where X k ∼ N (0 , σ 2 ) and Y k ∼ N (0 , 1) . From classical theory [5], it is known that X k converges in distribution to a power law with tail exponent α > 0 satisfying E | 1 -δX 2 k | α = 1 . In Figure 5, we plot the density of the representative for β 10 4 obtained using our

algebra against a kernel density estimate using 10 6 samples when σ ∈ { 0 . 4 , 0 . 5 , 0 . 6 } and δ = 2 . In all cases, the density obtained from the algebra provides a surprisingly close fit.





















=0.3





















=0.4





















=0.5

5 Related Work

Heavy tails and probabilistic machine learning. For studying heavy tails, methods based on subexponential distributions [18] and generalized Pareto distributions (GPD) (or equivalently, regularly varying distributions [49]) have received significant attention historically. For example, [35] presents closure theorems for regularly varying distributions which are special cases of Proposition 1 and Theorem 2. Heavy tails often have a profound impact on probabilistic machine learning methods: in particular, the observation that density ratios p ( x ) q ( x ) tend to be heavy tailed has resulted in new methods for smoothing importance sampling [53], adaptively modifying divergences [55], and diagnosing VI through the Pareto ˆ k diagnostic [60]. These works are complementary to our paper, and our reported results include ˆ k diagnostics for VI and ˆ α tail index estimates based on GPD.

Our work considers heavy-tailed targets p ( x ) which is the same setting as [25, 33]. Whereas those respective works lump the tail parameter in as another variational parameter and may be more generally applicable, the GGA may be applied before samples are drawn and leads to perfectly calibrated tails when applicable.

Probabilistic programming. PPLs can be broadly characterized by the inference algorithms they support, such as: Gibbs sampling over Bayes nets [13, 48], stochastic control flow [19, 58], deep stochastic VI [4, 52], or Hamiltonian Monte-Carlo [8, 59]. Our implementation target beanmachine [50] is a declarative PPL selected due to availability of a PPL compiler and support for static analysis plugins. Similar to [4, 46], it uses PyTorch [40] for GPU tensors and automatic differentiation. Synthesizing an approximating distribution during PPL compilation (Section 3) is also performed in the Stan language by [30] and normalizing flow extensions in [57]. We compare directly against these related density approximators in Section 4.

Static analysis. There is a long history of formal methods and probabilistic programming in the literature [26, 29], with much of the research [10] concerned with defining formal semantics and establishing invariants [54] (see [3] for a recent review). Static analysis uses the abstract syntax tree (AST) representation of a program in order to compute invariants (e.g., the return type of a function, the number of classes implementing a trait) without executing the underlying program. It has traditionally been applied in the context of formalizing semantics [29], and has been used to verify probabilistic programs by ensuring termination, bounding random values values [44]. As dynamic analysis in a PPL is less reliable due to non-determinism, static analysis techniques for PPLs become essential. As recent examples, [32] proposes a static analyzer for the Pyro PPL [4] to verify distribution supports and avoid -Inf log probabilities. More relevant to our work are applications of static analysis to improve inference. [38] and [12] both employ static analysis to inform choice of inference method. However, both works do not account for heavy tails whereas the primary goal of GGA-based analysis is to ensure tails are properly modelled.

6 Conclusion

In this work, we have proposed a novel systematic approach for conducting tail inferential static PPL analysis. We have done this by defining a heavy-tailed algebra, and by implementing a threeparameter generalized Gamma algebra into a PPL compiler. Initial results are promising, showing that improved inference with simpler approximation families is possible when combined with tail

metadata. While already useful, the generalized Gamma algebra and its implementation currently has some notable limitations:

The most significant omission to the algebra is classification of log-normal tails. Addition may be treated using [20], but multiplication with log-normal tails remains elusive.

Since the algebra assumes independence, handling of dependencies between defined random variables must be conducted externally. This can be addressed using a symbolic package to decompose complex expressions into operations on independent random variables.

Scale coefficients σ for conditional distributions may often be inexact, as exact marginalization in general is NP-hard [28]. Treatment of disintegration using symbolic manipulations is a significant open problem, with some basic developments [9, 45].

Compile-time static analysis is only applicable to fixed model structures. Control flow, openuniverse models [36], and PPLs to support them [4] are an important future research direction.

References

Søren Asmussen and Hansjorg Albrecher. Ruin probabilities , volume 14. World scientific, 2010.

Søren Asmussen, Enkelejd Hashorva, Patrick J Laub, and Thomas Taimre. Tail asymptotics of light-tailed Weibull-like sums. Probability and Mathematical Statistics , 37(2):235-256, 2017.

Ryan Bernstein. Static analysis for probabilistic programs. arXiv preprint arXiv:1909.05076 , 2019.

Eli Bingham, Jonathan P Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D Goodman. Pyro: Deep universal probabilistic programming. The Journal of Machine Learning Research , 20(1):973978, 2019.

Dariusz Buraczewski, Ewa Damek, Thomas Mikosch, et al. Stochastic models with power-law tails. Springer Ser. Oper. Res. Financ. Eng., Springer, Cham , 10:978-3, 2016.

Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519 , 2015.

Krisztian Buza. Feedback prediction for blogs. In Data analysis, machine learning and knowledge discovery , pages 145-152. Springer, 2013.

Bob Carpenter, Andrew Gelman, Matthew D Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. Stan: A probabilistic programming language. Journal of statistical software , 76(1), 2017.

Kenta Cho and Bart Jacobs. Disintegration and Bayesian inversion via string diagrams. Mathematical Structures in Computer Science , 29(7):938-971, 2019.

Guillaume Claret, Sriram K Rajamani, Aditya V Nori, Andrew D Gordon, and Johannes Borgström. Bayesian inference using data flow analysis. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering , pages 92-102, 2013.

Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. Power-law distributions in empirical data. SIAM review , 51(4):661-703, 2009.

Marco F Cusumano-Towner, Feras A Saad, Alexander K Lew, and Vikash K Mansinghka. Gen: a general-purpose probabilistic programming system with programmable inference. In Proceedings of the 40th acm sigplan conference on programming language design and implementation , pages 221-236, 2019.

Perry de Valpine, Daniel Turek, Christopher J Paciorek, Clifford Anderson-Bergman, Duncan Temple Lang, and Rastislav Bodik. Programming with models: writing statistical algorithms for general model structures with nimble. Journal of Computational and Graphical Statistics , 26(2):403-413, 2017.

Saverio De Vito, Ettore Massera, Marco Piga, Luca Martinotto, and Girolamo Di Francia. On field calibration of an electronic nose for benzene estimation in an urban pollution monitoring scenario. Sensors and Actuators B: Chemical , 129(2):750-757, 2008.

Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. Advances in neural information processing systems , 32, 2019.

SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neural Information Processing Systems , 29, 2016.

Mark Girolami and Ben Calderhead. Riemann manifold Langevin and Hamiltonian monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 73(2):123-214, 2011.

Charles M Goldie and Claudia Klüppelberg. Subexponential distributions. A practical guide to heavy tails: statistical techniques and applications , pages 435-459, 1998.

Noah Goodman, Vikash Mansinghka, Daniel M Roy, Keith Bonawitz, and Joshua B Tenenbaum. Church: a language for generative models. arXiv preprint arXiv:1206.3255 , 2012.

Archil Gulisashvili and Peter Tankov. Tail behavior of sums and differences of log-normal random variables. Bernoulli , 22(1):444-493, 2016.

Rameshwar D Gupta and Ramesh C Gupta. Analyzing skewed data by power normal model. Test , 17(1):197-210, 2008.

Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. The heavy-tail phenomenon in SGD. In International Conference on Machine Learning , pages 3964-3975. PMLR, 2021.

Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of a superconductor. Computational Materials Science , 154:346-354, 2018.

Liam Hodgkinson and Michael W. Mahoney. Multiplicative noise and heavy tails in stochastic optimization. In International Conference on Machine Learning , pages 4262-4274. PMLR, 2021.

Priyank Jaini, Ivan Kobyzev, Yaoliang Yu, and Marcus Brubaker. Tails of Lipschitz triangular flows. In International Conference on Machine Learning , pages 4673-4681. PMLR, 2020.

Claire Jones and Gordon D Plotkin. A probabilistic powerdomain of evaluations. In Proceedings. Fourth Annual Symposium on Logic in Computer Science , pages 186-187. IEEE Computer Society, 1989.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques . MIT press, 2009.

Dexter Kozen. Semantics of probabilistic programs. In 20th Annual Symposium on Foundations of Computer Science (sfcs 1979) , pages 101-114. IEEE, 1979.

Alp Kucukelbir, Dustin Tran, Rajesh Ranganath, Andrew Gelman, and David M Blei. Automatic differentiation variational inference. Journal of machine learning research , 2017.

Michel Ledoux. The concentration of measure phenomenon . American Mathematical Soc., 2001.

Wonyeol Lee, Hangyeol Yu, Xavier Rival, and Hongseok Yang. Towards verified stochastic variational inference for probabilistic programs. Proceedings of the ACM on Programming Languages , 4(POPL):1-33, 2019.

Feynman Liang, Liam Hodgkinson, and Michael W. Mahoney. Fat-tailed variational inference with anisotropic tail adaptive flows, 2022.

Arakaparampil M Mathai, Ram Kishore Saxena, and Hans J Haubold. The H -function: theory and applications . Springer Science & Business Media, 2009.

T Mikosch. Regular variation subexponentiality and their applications in probability theory, 1999.

Brian Milch and Stuart Russell. Extending Bayesian networks to the open-universe case. Heuristics, Probability and Causality: A Tribute to Judea Pearl. College Publications , 2010.

J. Nair, A. Wierman, and B. Zwart. The Fundamentals of Heavy Tails: Properties, Emergence, and Estimation . Cambridge University Press, 2022.

Aditya Nori, Chung-Kil Hur, Sriram Rajamani, and Selva Samuel. R2: An efficient MCMC sampler for probabilistic programs. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 28, 2014.

George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research , 22(57):1-64, 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Kumar Rajarshi. Life expectancy (who), 2018.

Gareth O Roberts and Richard L Tweedie. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli , pages 341-363, 1996.

Barry K Rosen, Mark N Wegman, and F Kenneth Zadeck. Global value numbers and redundant computations. In Proceedings of the 15th ACM SIGPLAN-SIGACT symposium on Principles of programming languages , pages 12-27, 1988.

Sriram Sankaranarayanan, Aleksandar Chakarov, and Sumit Gulwani. Static analysis for probabilistic programs: inferring whole program properties from finitely many paths. In Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation , pages 447-458, 2013.

Chung-chieh Shan and Norman Ramsey. Exact Bayesian inference by symbolic disintegration. In Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages , pages 130-144, 2017.

N. Siddharth, Brooks Paige, Jan-Willem van de Meent, Alban Desmaison, Noah D. Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr. Learning disentangled representations with semi-supervised deep generative models. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30 , pages 5927-5937. Curran Associates, Inc., 2017.

Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning , pages 5827-5837. PMLR, 2019.

David Spiegelhalter, Andrew Thomas, Nicky Best, and Wally Gilks. BUGS 0.5: Bayesian inference using Gibbs sampling manual (version ii). MRC Biostatistics Unit, Institute of Public Health, Cambridge, UK , pages 1-59, 1996.

Nader Tajvidi. Confidence intervals and accuracy estimation for heavy-tailed generalized Pareto distributions. Extremes , 6(2):111-123, 2003.

Nazanin Tehrani, Nimar S Arora, Yucen Lily Li, Kinjal Divesh Shah, David Noursi, Michael Tingley, Narjes Torabi, Eric Lippert, Erik Meijer, et al. Bean machine: A declarative probabilistic programming language for efficient programmable inference. In International Conference on Probabilistic Graphical Models . PMLR, 2020.

David Tolpin, Jan-Willem van de Meent, Hongseok Yang, and Frank Wood. Design and implementation of probabilistic programming language anglican. In Proceedings of the 28th Symposium on the Implementation and Application of Functional programming Languages , pages 1-12, 2016.

Dustin Tran, Matthew W Hoffman, Dave Moore, Christopher Suter, Srinivas Vasudevan, and Alexey Radul. Simple, distributed, and accelerated probabilistic programming. Advances in Neural Information Processing Systems , 31, 2018.

Aki Vehtari, Daniel Simpson, Andrew Gelman, Yuling Yao, and Jonah Gabry. Pareto smoothed importance sampling. arXiv preprint arXiv:1507.02646 , 2015.

Di Wang, Jan Hoffmann, and Thomas Reps. Pmaf: an algebraic framework for static analysis of probabilistic programs. ACM SIGPLAN Notices , 53(4):513-528, 2018.

Dilin Wang, Hao Liu, and Qiang Liu. Variational inference with tail-adaptive f-divergence. Advances in Neural Information Processing Systems , 31, 2018.

George Neville Watson. A treatise on the theory of Bessel functions . Cambridge university press, 1995.

Stefan Webb, Jonathan P. Chen, Matrin Jankowiak, and Noah Goodman. Improving automated variational inference with normalizing flows. In ICML Workshop on Automated Machine Learning , 2019.

David Wingate, Andreas Stuhlmüller, and Noah Goodman. Lightweight implementations of probabilistic programming languages via transformational compilation. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 770-778. JMLR Workshop and Conference Proceedings, 2011.

Kai Xu, Hong Ge, Will Tebbutt, Mohamed Tarek, Martin Trapp, and Zoubin Ghahramani. Advancedhmc. jl: A robust, modular and efficient implementation of advanced hmc algorithms. In Symposium on Advances in Approximate Bayesian Inference , pages 1-10. PMLR, 2020.

Yuling Yao, Aki Vehtari, Daniel Simpson, and Andrew Gelman. Yes, but did it work?: Evaluating variational inference. In International Conference on Machine Learning , pages 5581-5590. PMLR, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a static analysis technique for probabilistic programming languages, which annotates random variables with metadata characterizing their tail behavior. In particular, generalized Gamma distributions are used for this purpose. It is shown that they are closed under a number of operations, including addition, multiplication, and, under some conditions, reciprocals. This gives rise to an algebra which is used to statically infer the tail behavior of random variables throughout the probabilistic program, including posterior distributions. At runtime, the random variables are then estimated by neural splice flows initialized to the inferred Gamma distributions, which guarantees the correct tail behavior while allowing flexibility for the bulk of the probability mass. It is shown that this approach yields the correct behavior in many instances in which conventional methods fail.\n\nSTRENGTHS:\n- The paper addresses an important and interesting problem in probabilistic programming.\n - The static analysis approach to analyzing tail behavior is novel to my knowledge.\n - The chosen approach is competently executed: The provided algebra covers a wide variety of distribution types and operations, many of which utilizing newly derived theoretical results.\n - Combining the inferred Gamma distribution parameters with neural splice flows is a clever way to maintain the correct tail behavior while allowing flexible estimation of the distribution's bulk.\n - Encouraging experimental results are provided for a number of tasks, including density estimation, variational inference, and Bayesian linear regression.\n - The paper is well written and easy to follow. Detailed derivations for the new theoretical results are provided in the appendix.\n\nWEAKNESSES:\n- The models used for the experimental results are somewhat small. While simple monovariate distributions are fine to get a sense for the behavior of the method, it would be good to include some larger programs to demonstrate the robustness of the method.\n- The paper mentions a number of limitations of the approach: It does not cover log-normal tails, operations between dependent variables, or dynamic model structures. Given these limitations, it would be especially useful to highlight some set of more complex models used in practice to which the method is nevertheless applicable.\n\nQUESTIONS:\n- How does the system behave when the assumptions discussed above are not fully met, e.g., when there is an operation between dependent variables? Is it possible to make use of partial results for a given program?\n- The discussion on posterior distributions mostly focuses on two random variable connected by a chain of operations, but notes that other cases may also be covered. Is it really the case that any posterior distribution may be estimated this straightforwardly? Can the system as implemented cover complicated multivariate computation graphs?\n\nLIMITATIONS:\nThe limitations have been well addressed, though their practical implications could be stated more directly (as discussed above).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper develops an algebra which acts on a three-parameter family of tail asymptotics based on the generalized Gamma distribution. The algebraic operations are closed under addition, multiplication, powers, and a full list is given in Table 1. With this algebra, tail calculation can be done automatically in probabilistic programming instead of analytically. The paper also proposed a method of splicing bulk and calibrated tails with experimental verification.

STRENGTHS:
1. The paper develops a heavy-tailed algebra for probabilistic programming and also proposed a method of splicing bulk and calibrated tails with experimental verification.
2. Example 4 in the appendix had a calculation for the product of two random variables.

WEAKNESSES:
I did not notice any glaring weaknesses in the paper but I am also not a subject matter expert on probabilistic programming.

QUESTIONS:
1. In the case of multi-dimensional random variables and dependent tails, would you consider using an extreme-value copula?
2. In the case of dependent tails and bulk to tail splicing/ calibration/ extrapolation, would you consider using an Archimax copula?

LIMITATIONS:
The limitations were clearly defined and addressed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDuring inference, we are often interested in the behavior of the tails of the distributions we are analyzing. Heavy or light tails may necessitate switching algorithms so that inference remains stable for example. This paper describes a calculus by which a probabilistic programming language may calculate the tails of distribution objects derived from operations performed on elemental distributions. They apply their method to perform some simple probabilistic programming applications.\n\nSTRENGTHS:\nThe authors apply their method to a huge number of distributions and their transformations. They summarize their rules helpfully in charts. It seems as though their method may also simply be added to other probabilistic programming languages. I would certainly appreciate being able to call the tail parameters of a distribution object in Pyro for example.\n\nWEAKNESSES:\nThe authors in section 3 suggest a method for fitting a density that consists of fitting its tail using their calculus and then fitting the bulk using a flow model. This is a reasonable choice for their simple experiments and there are a number of other choices they could have made to fit the bulk. The exposition of the paper sets it up in figure 1 and section 3 as essential to their probabilistic programming method, but I don't believe that to be that case. It would clarify and strengthen the paper if authors clarified the relation of their flow method for estimating densities to the rest of their contributions.\n\nQUESTIONS:\nIn line 134 you describe the representative distribution of a distribution with a tail with a small $\\rho$. You motivate your construction by matching moments, but could you describe more rigorously why this is a reasonable \n\nI also wonder why the authors don't consider incorporating discrete distributions such as Poisson and Geometric distributions, which can be treated as continuous in their setting, say by adding a uniform random variable, so a geometric would have a $(0, 0, \\log(p))$ tail for example. There are often cases where discrete data is treated as continuous, such as for RNAseq counts.\n\nLIMITATIONS:\nThe authors describe limitations in section 6.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of density estimation of probabilistic models (expressed as probabilistic programs), with a focus on their tails. This is important for several Bayesian inference methods: importance sampling can exhibit infinite variance if the proposal has a lighter tail than the target, many black box variational inference methods cannot change their tail behavior and MCMC algorithms may lose ergodicity if the tail of the target is outside a particular class. This paper proposes a static analysis pass on probabilistic programs consisting of sampling from primitive distributions and arithmetic operations on the sampled variables (addition, multiplication, division, logarithm, exponential, and Lipschitz functions). The analysis is based on generalized Gamma distributions, which capture many tail behaviors of primitive distributions (e.g. Gaussian, Gamma, Weibull, Student-t, Pareto, but not the log-normal) and are closed under the above operations on random variables (or can be bounded if not represented exactly). As a consequence, a compiler pass can compute the parameters of a generalized Gamma distribution with the same (or at least providing a bound on the) tail behavior as the posterior distribution of the probabilistic program. Since the bulk of this distribution may be very different from the posterior, the paper combines it with neural spline flows: effectively, this computes the pushforward distribution under a Lipschitz function, resulting in a better bulk approximation while preserving the tails.\nThe experimental evaluation demonstrates that this approach to density estimation usually improves the tail estimation compared to status-quo initializations with a normal or Cauchy distribution, both for density estimation using normalizing flows and variational inference, where the target distributions are simple Cauchy, Inverse Gamma, Student-t, χ^2 and normal distributions. Similarly, the density estimation of the variance parameter of a Bayesian linear regression model is improved compared to the standard Gaussian approach. Finally, they observe that their method (without normalizing flows) provides a very close fit for the density estimation of the iterates of stochastic gradient descent applied to a least-squares linear regression.\n\nSTRENGTHS:\nThe paper's main idea of using a static analysis to compute the tails of the posterior distribution of a probabilistic program is very interesting because tails are difficult to estimate. Such an ahead-of-time analysis with correctness guarantees is therefore very useful. In order to arrive at this result, the paper proves several new theoretical results on generalized Gamma distributions, which may be of independent interest. The experimental results show that in combination with normalizing flows, this often performs better than existing methods that assume a Gaussian base distribution. The presentation of the paper is generally clear and readable.\n\nOverall, this paper has an innovative idea with promising results, which are of interest to the probabilistic programming community.\n\nWEAKNESSES:\nWhile the paper's idea is very interesting, I have a few concerns about the soundness and presentation. I will update my rating if the author's answers address my concerns.\n\nGuarantees and assumptions: What exactly are the guarantees (mentioned in line 149) and assumptions of the static analysis? It would be good if this could be stated in one place, ideally as a self-contained theorem. Table 1 mentions that \"additional assumptions are required\" for several operations, but I was not able to find them in the paper. The assumption that the operands are independent is also very important and only really mentioned at the very end (line 270). It is also unclear what exactly the guarantees are for programs involving reciprocals, exponentials, and logarithms because the generalized Gamma distributions are not closed under these operations. In this case, will the computed tail be an upper or a lower bound? What happens if such operations are applied repeatedly? While there are theorems and proofs for each individual operation, I was not able to find a theorem clarifying the assumptions and guarantees about the composition of such operations, especially if the resulting tail cannot be represented exactly in the GGA.\n\nBulk correction: Section 3.3 (the bulk correction via normalizing flows) is informal and light on details. I assume the neural spine flows preserve the tails because they are 1-Lipschitz maps? This is not obvious to me, and I think it would be good if the paper could elaborate on why the normalizing flows don't affect the tails. Again, putting the statement in the form of a theorem with all necessary assumptions would be helpful.\n\nProbabilistic programming language: The grammar of the probabilistic programming language is only introduced by example, there is no comprehensive description, not even in the appendix. Furthermore, the name \"programming language\" sounds like there is support for control structure (branches, loops), which doesn't seem to be the case in this approach. Rather, the \"programming language\" seems to be a simple expression language with the grammar `E ::= sample(D) | op(E, ..., E)` where `E` is an expression, `op` is a supported operation, and `D` is a supported primitive distribution. It would be good if this could be clarified in the paper.\n\nIndependence assumption: a main assumption is that the operands to all operations are independent random variables. This seems to be difficult to guarantee in practice because posteriors of parameters are often correlated in probabilistic models. Does this mean that (for now), all operands of all operations need to be hand-checked for independence? Does this severely restrict the class of probabilistic models that can be handled by this approach? If so, this should be discussed. It is only briefly mentioned in line 270.\n\nThe following points are weaknesses, but less important than the previous points:\n\nExperimental evaluation: the experiments demonstrate benefits for very simple benchmarks (one-dimensional primitive distributions, one-dimensional Bayesian linear regression). If more realistic experiments (e.g. higher-dimensional examples) could be performed, this would strengthen the paper. However, I don't consider this a condition for publication because the results are already interesting.\n\nAbstract interpretation: the static analysis pass, as I understand it, performs abstract interpretation (a common technique in static analysis) where the abstract domain is given by the generalized Gamma algebra. It would be good to mention this keyword, as it helps put this method into context.\n\nQUESTIONS:\n- Line 92 claims that closure under some operations is known. Could you provide references for this?\n- What exactly are the guarantees and assumptions for operations under which GGA is not closed? What happens if such operations are applied repeatedly? (cf. Weaknesses)\n- The operations in Table 1 only apply to equivalence classes $(\\nu, \\sigma, \\rho)$. What happens if the result of an operation is $\\mathcal L$ or $\\mathcal R_1$? Will those be treated as $(?, ?, \\infty)$ or $(-1, ?, 0)$ in subsequent operations? If so, what values are assumed for the question marks? Why don't the infinities lead to problems?\n- Section 3.3: could you elaborate on why the normalizing flows don't affect the tails (cf. Weaknesses)?\n- In Table 4, GGA Flow does not seem to work for IG and StudentT. Do you know what the reason for this is? It seems unexpected given all the claimed benefits.\n\nTypos:\n- Equation (1): should be $= cx^\\nu ...$ or $\\sim x^\\nu ...$\n- line 111: missing indices $f_{ij}$, $R_{ij}$.\n- line 131: shouldn't it say $\\rho \\le 0$ instead of $\\rho \\le -1$?\n- line 201: it is unclear where the \"conditional on\" clause ends, please rephrase\n\nLIMITATIONS:\nLimitations are discussed in the Conclusion. I believe a few items should be added to this list:\n- no control structure in the programming language (it is an expression language, cf. Weaknesses)\n- tails are not exact (only bounds) if certain operations are involved", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}