The Cross-entropy of Piecewise Linear Probability Density Functions
Authors: Tom S. F. Haines
Venue: Accepted by TMLR ()

Abstract: The cross-entropy and its related terms from information theory (e.g.~entropy, Kullback–Leibler divergence) are used throughout artificial intelligence and machine learning. This includes many of the major successes, both current and historic, where they commonly appear as the natural objective of an optimisation procedure for learning model parameters, or their distributions. This paper presents a novel derivation of the differential cross-entropy between two 1D probability density functions represented as piecewise linear functions. Implementation challenges are resolved and experimental validation is presented, including a rigorous analysis of accuracy and a demonstration of using the presented result as the objective of a neural network. Previously, cross-entropy would need to be approximated via numerical integration, or equivalent, for which calculating gradients is impractical. Machine learning models with high parameter counts are optimised primarily with gradients, so if piecewise linear density representations are to be used then the presented analytic solution is essential. This paper contributes the necessary theory for the practical optimisation of information theoretic objectives when dealing with piecewise linear distributions directly. Removing this limitation expands the design space for future algorithms.

Published in Transactions on Machine Learning Research (04/2024)
The Cross-entropy of Piecewise Linear Probability Density Functions
Tom S. F. Haines
tsfh20@bath.ac.
Deportment of Computer Science University of Bath
Reviewed on OpenReview: https: // openreview. net/ forun? id= AoDs92gdsv
Abstract
The cross-entropy and its related terms from information theory (e.g. entropy, Kullback- Leibler divergence) are used throughout artificial intelligence and machine Icarning. This includes many of the major successes, both current and historic, where they commonly appear as the natural objective of an optimisation procedure for learning model parameters, or their distributions. This paper presents novel derivation of the differential cross-entropy between two 1D probability density functions represented piecewise linear functions. Implementation challenges are resolved and experimental validation is presented, including a rigorous analysis of accuracy and demonstration of using the presented result as the objective of a neural network. Previously, cross-cntropy would need to be approximated via numerical integration, or equivalent, for which calculating gradients impractical. Machine learning models with high parameter counts are optimised primarily with gradients, so piecewise linear density representations are be used then the presented analytic solution is essential. This paper contributes the necessary theory for the practical optimisation of information theoretic objectives when dealing with piecewise linear distributions directly. Removing this limitation expands the design space for future algorithms.
1 Introduction
Information theory (Shannon 1948 provides mathematical toolbox, that, while originally motivated by the problem of communication over noisy channels, has proved essential to artificial intelligence. The set of information theoretic terms that can be obtained from cross-entropy often appear within training objectives for machine learning (ML), e.g. information gain for random forests (Sethi & Sarvarayudu 1982). They are integral to the evidence lower bound (ELBO) in variational inference (Jordan et al. 1999). ) as used by techniques such as variational autoencoders (Kingma Welling 2014) and diffusion models (Sohl-Dickstein et al. 2015). Cross-entropy loss (Hinton al. 1995) is often preferred over the classical mean squared error [Prince 2023 for neural networks, c.g. in transformers (Devlin et al. 2019). It also plays part in thcory, e.g. the maximum entropy principle for selecting distributions of Jaynes (1957).
In practice the uses of information theory are relatively simple. Most algorithms use Monte Carlo integration (Metropolis et al. 1953) to calculate the mismatch between data and model, with the training set acting as a fixed sample from the target distribution. It is regularly used as a loss function for discrete classification, but in the context of continuous regression alternate objectives are common, as the differential cross-entropy of the error distribution is hard calculate in general. Variational methods often limit themselves to the exponential family (Wainwright & Jordan 2008), for which analytic expressions known. This motivates this paper: adding to the list of probabilistic objects with a known cross-entropy increases the design space available for practical algorithms.
For the purpose of open access the author has applicd Creative Commons Attribution (CC-BY) licence. A complete implemennattin,, including code to generate the included figures, in the supplementary material and also available from There data. This research made Hex, the GPU Cloud in the Department
https://github. com/thainee//rograaa of Computer Science the University of Bath.

---

Published in Transactions on Machine Learning Research (04/2024)
This paper's main novel contribution is analytic expression for the cross-entropy between two 1D piecewise linear probability density functions (PDFs),
Pi log(g)) + +p++1 log(g(+))
P=+197 -Pi9 (log(0++) - -log(4))
-(P,Q)=-> Σd; 6/
=
2
2(ge+1 9:)²
(3p; + Pi+2)9141 - (p: + 3p:-1p1100 1)9:
(1)
4(1 4.)
It sums over every linear segment, indexed by i, with P3 the PDF P evaluated at the start of the segment and Pi+1 evaluated at. the end, these being change points. The same relationship holds for and Q while 5 is the width of segment i. Entropy (= H(P, and the Kullback Leibler divergence (Kullback & Leibler 1951 (= (P, - H(P, P)) follow immediately and are also novel contributions.
This result supports the futuure development of MI. algorithms that use piccewise linear PDFs. Potential advantages include support for step changes and multi-modality, which are poorly supported by the ex- ponential family. This could be particularly valuable within the context of variational inference [Jordan et al. 1999), , where cross-entropy appears as the objective. Optimal transport (Bonneel et al,, 2016) is also trivial with this representation. Within the context of neural networks, and gradient-based optimisation in general, piecewise linear PDFs can be included only if laking their derivative is practical'. Doing so with the presented result is computationally efficient, while using an alternative built around technique such as numerical integration is inefficient to the point of being implausible. Whatever their use, piecewise linear PDFs can have an arbitrary and tunable parameter count, allowing greater expressiveness relative to the typical text book distributions, and are simpler than many nonparametric approaches. Tt remains the case that using numerical integration is simpler and will work for many applied problems, but in an AI/ML context, where cross-entropy is often optimisation objective, there is a need for an analytical (and differentiable) result. Numerical integration is typically slow however, so speed advantage may be observed even for straightforward problems. Related work follows, with the derivation of the result in Section 3 This is followed by numerical validation in Section then a demonstration and, finally, a conclusion.
2 Related work
A history of piecewise linear PDFs is presented, building up to various uses within AI/ML. Quadrature based approaches are discussed briefly. The triangular distribution (Simpson 1755 Johnson 1997), arguably the simplest, is the only piccewise lincar density function for which information theoretic calculation appears to be available: differential entropy Lazo & Rathie 1978) This result adapts to the trapezoidal distribution
RReé van Dorp & Kotz 2003).
A histogram Pearson 1895) can be a piccewise lincar PDF if it represents binned continuous data and area has been normalised. lised. Its cross-entropy is trivial. While suitable for many problems it has no gradient, e.g. you can't immediately train a neural network to output data with a specífic distribution if that distribution is represented by a histogram, because infinitesimal changes data position leave the cross-entropy unchanged. The frequency polygon (Scott, 1985b) connects the centres of regular histogram, and is probably the earliest example of constructing general piecewise linear PDF2 This is only consistent for a regular histogram however, because otherwise the maximum likelihood solutions differ between the two representations. History appears to have omitted or forgotten the obvious correction for this.
A variant is the edge frequency polygon (Jones et al. 1998). which connects the midpoints between bins (half way between bin heights at the odges between them) with lincar segments. It confers no advantage over the frequency polygon, having the same convergence in mean squared error sense. This convergence is matched by the kernel density estimate (Rosenblatt) piecewise linear kernel is used, such as triangle. Alternatively, averaging many histograms with different origins (Scott 1985a), such that the bins are misaligned, achieves a comparable effect with computational ad advantage. It is piecewise linear though has an excessive number of change points. Another approach is to
1956 Parzen 1962), which generates piecewise linear PDFs if a
Of relevance here, the gradient which connects data point position to the eross-entropy their distribution Farlier mentions exist but lack detail, e.g. Pearson in 1925 Tarter & Kronmal 1976).
2

---

Published in Transactions on Machine Learning Research (04/2024)
take kernel density estimate with any kernel and then fit a linear representation to it this is an approximation and requires finite or truncated kernel.
[Ln et al. 2006
Some approaches go directly to a piecewise linear representation. Beirlant et al. 1999 fit a linear segment to each bin of a histogram; this does result in discontinuities. Alternatively, Karlis & Xekalaki (2008) fit a mixture of triangular distributions, which generates an arbitrary polygon without discontinuities; they refer to it as the "polygonal distribution". A least squares approach with the segments fixed but their heights allowed vary has been proposed (Wielen & Wielen 2015), has maximum likelihood estimator (Nguyen) & McLachlan 2016 The segments remain connected for both. Nguyen & McLachlan (2018) have shown that the maximum likelihood estimator is consistent.
Perron & Mengersen (2001) take a non-parametric Bayesian approach, estimating a non-decreasing function as the cumulative distribution function of a mixture of triangular distributions. This approach models an explicit. Poisson draw of the mixture count followed by Dirichlet over membership, using reversible jump MCMC for inference. Alternatively, Ho et al. (2017) use Dirichlet process prior (Ferguson) 1973) for doing Bayesian density estimate with a mixture of triangular distributions. They are motivated by the problem of making Bayesian estimate of an unknown distributions mode and utilise a Gibbs sampler with the stick breaking construction (Sethuraman 1994b.
Beyond explicit models built. around piecewise linear PDFs there are also incidental uses within AL. Nu- merous models output histograms as discrete representations of contimuous values. As an example stereo disparity for which examples utilising dynamic programming (Cox al. 1996), belief propagation (Felzenszwalb & Huttenloche 2006), graph cuts (Boykov et al., 2001 and convolutional neural networks (Zbontar & LeCun 2015) exist. Finally, commonly taught when introduc- ing neural networks (Prince 2023), a network that only uses regularised linear units (ReLU) (Fukushima 1975 generates multivariate piecewise linear function.
algorithms for rectified images output disparity,
For the purpose verification numerical integration (Gibb 1916) and Monte Carlo integration (Metropolis et al.] 1953) are used in Section [ These are the main alternatives to the presented analytic approach if only calculation is required. Other choices exist, primarily those based on quadrature (Gonnet 2012) This requires family of functions for which the relevant information thcoretic terms be calculated. For PDFs the Gram-Charlies Charlier/Edgewoone evaluate the integrand directly with a more general technique (Place & Stach 1999). Hyvärinen 1997) has proposed specific quadrature scheme designed for calculating differential entropy. While not the focus of this paper, the equation contributed does enable 1D quadrature with piccewise lincar approximations for cross-entropy. Examples of these approximations being used within machine learning include independent component analysis (Jutten & Herault, 1991) and projection pursuit (Huber 1985).
series (Wallace 1958 are commonly chosen, but you also
3 Derivation
The cross-entropy will now be derived. Note that, much like entropy has to define 0log(0) = 0, similar issucs occur. In the interest of clarity these are considered after the initial derivation. Differential cross-entropy is defined as (Jaynes 1963
P(x) log Q(z)dz, H(P,Q)=
(2)
where and Q are two PDFs and æ integrated over their domain. Taking P and Q to be piecewise linear and 1D we can consider the integral to be the sum of many linear sections,
- -E 8, ((11 (1l-t)pp + tp#11 log((1 - t)q. + tqt
(3)
where Pi is evaluated at the start of section i and Pi+1 is P evaluated at the end; likewise for g and Q. 5, is the section width, Z+41 - 2z. If the linear segments of P and Q are not in alignment then extra change points can be added.
3

---

Published in Transactions on Machine Learning Research (04/2024)
Consider single section,
= - S ((1 (11-t)p: 1og+)) - t)g: + tqb11ddt.
(4)
Define
Ap: Pi+1 Ps:
Aq: = qi+1 9i
(5)
and introduce
q= t)q + tq;+1 = 9: Aqit.
(6)
then simplify Equation 4 with a change of variables,
6; Aq: 9h
Pi (q 4.) log(E)dg.
p:
(7)
Aqi
Separate the two integral forms,
d Ag:
{
pi
Api Agj
qi 1
log(
Api Mi++
Ag. {
á (log())dá)
(8)
and slot in solutions to both,
5i Api Aui"
(p:
9:
(a++1 (log(g++1) - 13 qi {log(a:) - 1})
Dg.
Api
(lo((1+1))
1
log((:)
4 1))
+ Ag.
9i+1
9i
2
(9)
4
then rearrange to obtain
2
5i = {
Api
(Aqipi - Apage) (gi+1 log(g++) - gi log().))
+
(g log(9)+) - 9 log(g:))
2
+ (Aq.p: Apiq:) 9+11+ Ap;
(g) r2tz)
(10)
Separate out the first two terms within the curly brackets, ① and ?
(App: Apra:) (g+41 log(g(11 log(g))) =
Pi+19, log(g:) + p:git log(g++) 1 + pi+1 log(g++1)), (11)
Api (9 log(g(41) of log(q:))
ps+19 log(g)
pige. log(9(+1)
Pi419h 1og(9-1)
+ pige log(a:)
(12)
2
2
2
2
Introduce
l:+1
(gi+1 - -/))
+
9.9++1
(13)
2
4

---

Published in Transactions on Machine Learning Research (04/2024)
and use it to merge 1 and @, to obtain
pi+ log(a:) polh+1 log(gi-1 + (9+1 10:1--981 9:)) (og(())) Pe+1 log(9(+1)) P log(1.)
1
Pi+1 log(4i+1)
2
(14)
where the third term above corrects for the mismatch of the second term with @. Simplify to get
p: Pi+1 of
(log(u+1) log(a)) 2 (A log(a.) p++1 log(u++)).
(15)
Now rearrange 3 and 4 from Equation
10
Aps
Aq: [(pp, + p1+1)9++11 (pi + -pp++)9gl
(Ag.Pi - Apig.) (a: - 9(+1)
(g) (241)
(16)
and bring all of the terms from Equation 10 back together to restate the equation for a single segment,
1 poff+1 pi+ i9 (log(a(+1) log(g:)) + (2 (ogg(( + P log(s++1)
(Ag:)² 1
2
2
Ag
[3p; + P:41)4411 (p. + 3ptta)ail, ) (17)
4
which rearranges to
Pi log(g))) + P.+1 log(9-1) 8,
Pi+10e - p:ch+ 2( -x,²
(log(1+1)) log(g:)) + (3p; + P=+1)9:41 (p; + 3p )4:
-
2
4(144 9:)
(18)
completing the derivation needed for Equation
3.1 Singularity
As expected, Olog(0) = O has to be specified for Equation to work, iee. this is Lebesgue integration. However, there also a singularity when qi = 4++1. To ignore the second and third term of Equation when dc+1 = qi it must be the case that
Pi+1 - p. (log(0+1) - log(g:)) + 8, (3p; + P++1)9++1 (ps + 33p+11 ) lim 8,
= 0.
(19)
2(9 92
4(g qt)
The two terms have to be considered simultaneously for this to be the case. Use the series 2010 Equation. 4.6.4.),
Olver et al.
log(a) = Σ
1
(a=1))"
(20)
a +
(1.3,,,....
to convert the first term the limit into an infinite sequence,
(git1 g:)m
1. - pi
Σ
(21)
(g++1 + 9:)m
(1335.55....
and note that the limit trivially true for n = onwards. Put the n 1 term only back into the limit
(Equation 19
(4++1 -1
(git1 + 91)) + (3p, Pi+ 1):41 (p; + 3p )46
lim -
6e(p:1197 p.
(22)
4(g.+1 - 9:)
and rearrange to obtain
(pi+1 p:)((441 9i)
lim 8: g++1 7gg
(23)
4(g+t1 + (i)
which is simply zero, satisfying the requirement.
5

---

Published in Transactions on Machine Learning Research (04/2024)
1e-1042.3028599e-
Le- 11+9. 1629073e2
4,5
Numerical integration
22
Numerical integration
Segimet cross crosserrtopp entropy
Fast
Segment cross antropy
Fast
Stable
21
4.0
20-
19-
18-
-1.5
-1.0
0.0
2.0
-2.0
-1.
0.0
2.0
9: 46
10-5
Q1 93
10 5
Figure 1: Zoomed in plots of the cross-entropy of linear segment, showing the approached. Number top left is the scale of the y axis, bottom right the scale of the axis. Calculated with double precision. For both plots Po = P1 = 0.1 and 6, = 1; for the left plot 40 0.1 while for the right f0 0.4. The axis shows the difference between the two q values, i.e. they cross such that the singularity is in the middle and the mean remains constant. This demonstrates how switching from Equation to 24 only needs to occur when within 1e-5 of the singularity, with this range increasing with q. To obtain this precision with numerical integration required 2 samples with averages organised over three levels to avoid underflow; is over 90000 slower than the stable equation.
the singularity is
3.2 Implementation
Computing cross entropy with Equation 18 is numerically stable when a safe distance from the singularity, but becomes unstable due to the limits floating point operations (EEE 2019 when loo close. stable everywhere alternative
-6,-
P: log(g:) + P.+1 log(4--) + 6, (pi+1 - p:)(((t1 - 4)) + o, P++1 p. 4(gi+1 + +.)
Σ
gi+l qi
2
(46+1 + 9:)
n
gi+1 + qi
DE (1,,5......
(24)
which is obtained from the limit calculation above. The infinite series converges quite slowly; use Equation 20 and assume,, without loss of generality, that g:+1 g: hence
0
Σ n+r 06+1 + gi
9i+1 9
"
1
log
gi+1
(25)
2
qi
(13
where 0, and 2 gets bound on Equation 24 then
If e is the error after rumning to N - 2, inclusive,
Σ NNNNN-2 +.....
9i+1 96
9i+1 96
Σ
9++1 "
(26)
9i+1 + qi
0:+1 gi
N +
g++1 + qi
ne
such that
N-1
(+1 gi
a++1
OSe<
4s+1 + log
(27)
1- is almost always small and the bound is quite loose", so you obtain multiple bits of In practice 9--1+% precision with each iteration.
Using a basic Python/nunn implementation Equation 24 is 40x slower than Equation is use the fast equation where possible. This comparison obtained using an implementation that keeps calculating terms within the summand in Equation 24 until one evaluates as less than 10-64, Figure [ explores where the transition between approaches should occur: within [go - ?ıl S le-5 is the rule selected for the implementation in the supplementary material. A more complex rule may make sense depending on the hardware/ hardwarrelikellhood likelihood of being close to the singularily.
the preference
3tt made tighter with polylogarithms

---

Published in Transactions on Machine Learning Research (04/2024)
0.4
0.5
Monte-Cario integration
Numerical integration
0.3
KL-divergence
True
Analytic
0.3
0.2
0.1
0.1
0.0
0.0
-3
1
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75 1.00
No
Ar
Figure 2: The standard Gaussian (u= = 0, = 1), represented as piecewise linear PDF and hence truncated. Which, due to how curve rendering works in the digital realm, is identical to rendering standard Gaussian without explicitly construct- ing piecewise linear PDF. The piecewise linear approximation made by matching the area der the linear segments with the area under the Gaussian's CDF, with a regular spacing.
Figure 3: The Ballback--eibler divergence, calcu- lated via cross-entropy with H(P, H(P, of two standard width Gaussian distributions as their means are varied. The & axis shows the difference between the means. Four approaches have been used to calculate this result: the known solution ("Truc"), and then three approaches based on piecewise linear approximation with " "Analylic" the presented approach. The lines all overlap.
3.3 GPU implementation
Computing the infinite series of Equation 24 a GPU not reasonable: the loop will have to be run until all of the parallelised values have converged, wasting computation, plus reverse mode automatic differentiation will struggle. Fortunately, this stable version only has to be used when 1 - gi is sufficiently small that the infinite series obtains float (32 bit) precision with the first two terms only. This means in practice that
- 8, P: log(()) (1)) + Pi+1 log(g++1) 8, (Pit1 (:)(1 9:)
4(g++1 + +(:)
+ 5 Pi+ P.q.+1
4++1 4i
+ 5 gi+1 q '
Mi+1
(28)
9.+1 + 9:)2
3
41+1 + 9:
is used for computation on a GPU when in the g++1 2 qi situation. Approximation will still occur where an infinity or large value is expected, but as these break optimisation this proves convenient. Finally, due to the inefficiency of branching on a GPU a fourth variant of the result proves useful when | - gi is large,
- 5, pi (g)())) + Pi+1 log(g++) + 8, (pi+1 pi) 9c) + 8, 2(g 9.)2
Pi+1 P.g.+1
log(a++) - log(4:)
4(g.+1 + 9:)
Pit P.g..1
8, (git1 ()(( q:)
(29)
This can be understood as the main result, Equation 18 rewritten to share many terms as possible with Equation 28 to minimise the code that appears within branches. Both of these versions are compatible
with standard automatic differentiation libraries; for completeness an example implementation is included in Appendix as used for the demonstrations in Section

---

Published in Transactions on Machine Learning Research (04/2024)
1.0
2.5
0.8
0.6
PDJ
1.5
Carle integration Numerical Integration Analytic
1.0
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
0.0
0.2
0.4
0.6
0.8
1.0
D.Q
0.5
1.0
Figure 4: The top left graph shows a mixture distribution: a uniform distribution on the left and triangular distribution on the right, with equal probability, plus (low probability) wide Gaussian to avoid infinities in the following, converted into piecewise linear PDF. As shown by the bottom row, the top left graph is showing & = 0: at t ==1 1 the cube and triangle have switched places, with them moving lincarly for the transition, c.g. at t = 0.5 the cube is wearing a hat. The top right graph plots the cross-entropy, H(P, Q): where P is always at / = 0 while Q varies from / = 0 to = 1. At the right we have the entropy of the top graph, in the centre the shapes are poorly aligned, giving the highest cross-entropy, then, when they have swapped places, it's a better match, if imperfect. The analytic solution shown alongside numerical and Monte Carlo integration, for verification.
4 Numerical validation
Three demonstrations have been selected to provide numerical validation. In all cases numerical integration Gibb 1916) and Monte Carlo integration (Metropolis al. 1953) have been used to verify the result. Operations have been done with float (32 bit) precision for the presented approach, consistent with the precision commonly used GPUs. For the first two demonstrations there is also direct solution (* True", because they use Gaussian distributions, but note that the Gaussians have to be converted to piecewise linear PDFs, introducing some error that is then reflected by the proposed approach (" ("Analytic") as well as both of the sampling integrators. The conclusion throughout is that the presented approach is accurate.
First, Figure shows a truncated standard Gaussian distribution (Gauss 1823). has been represented as a piecewise linear PDF: the area under each linear section was matched to the Gaussian and then was renormalised, to account for the truncation. Entropy of the standard Gaussian known to be 1.4189385. The presented approach gives 1.4189711, noting that some variation is expected due to the truncation and linear approximation. Numerical integration gives 1.4189712, identical to float precision. Monte Carlo integration gives 1.4191646, but is using the same number of samples as numerical integration (224) which is not enough to match the precision it's a poor choice of integrator for a 1D function.
Figure 3 shows a sweep of the Kullback-Leible divergence between two standard width Gaussians as the delta between their mcans varies. As before there is a known equation for this ( ("True"), which is again expected not match exactly due to the distributions being represented with linear sections. The maximum difference between numerical integration with 24 samples and the analytic solution is 0.000594; increasing the numerical integration sample count made no further difference. Based on the results elsewhere it is reasonable to believe that numerical integration is converging poorly.
Finally, Figure the cross-entropy directly, calculated for a sweep as two parts a mixture distribution swap position. This reflects the more arbitrary distributions that justify the use of a piecewise lincar distribution. There is no known equation for this, but the maximum difference observed between numerical integration with 2 samples and the presented analytic approach is 66
8

---

Published in Transactions on Machine Learning Research (04/2024)
Q(x)
0.3
Figure 5: The gradients of points drawn from P(x) to move towards Q(z), where the cost of the differ- ence to be minimised is expressed with the Kullback- Leibler divergence, D || Q). The vertical layout of the arrows for visual clarity only, and the seg- ments of the piecewise linear distribution of ((z) are delincated by the short vertical lines.
P(x)
0.2
Plx)
0.1
0.0
2
0
2
X
Start Goal
End
0.4
Goal
0.2
p(X)
P(x)
0.2
0.1
0.1
0.0
0.0
-2
0
1
2
3
-3
-2
1
X
Figure 6: On the left is the initial distribution of a draw of 1024 points from a standard Gaussian. A subset of the points have been shown with their gradients as arrows; their vertical positioning has been randomisod to reduce visual overlap. The two bumps, from the central region of a rectified sine curve (normalised), constitute the target of the optimisation. On the right is the result, where the points have been moved to broadly match the goal distribution using Nesterov's accelerated gradient descent.
5 Demonstration
One use case of cross-entropy is objective for continuous optimisation. This is demonstrated in Figure where 1D set of points have been sampled from P(x) and are being moved towards matching the distribution Q(x), both represented as piecowise linear probability density functions. The gradients of the points, as represented by arrows, are calculated in terms of the Kullback Leibler (KL) divergence bet the two distribut,ons, as defined in terms of the cross-entropy given Equation [ with DKL(P || Q) ((P,Q) - (IP P). The piecewise linear distribution of the points is constructed similarly to histogram, except point mass is linearly interpolated between bin centres; this generates piecewise linear distribution (equivalent to mixture of triangular distributions) where point positions have gradient relative to the bin heights. Gradients have been calculated using reverse mode automatic differentiation (Linnainmaa 1976).
The central region of Figure 3 has no gradient because both distributions are uniform, while at the edges the points are being moved inwards, to where Q(x) has mass. On the right hand side a set of points with no gradient can be observed; this bocause their segment and the adjacent segment both lack probability mass, i.e. ((x) = 0. Note however that gradient still exists for Q(x) = 0 segments when they are adjacent to segments where Q(x) # 0. It is therefore necessary to merge adjacent zero probability segments ensure gradient exists everywhere.
The ability to calculate gradients relative to an objective enables continuous optimisation. This is demon- strated in Figure 0 where a draw from one distribution is optimised (moved) to match with another. Nes- terov's accelerated gradient descent (Nesterov) 1983) used, with 2048 iterations reducing the KI-divergence from 0.740 0.007. In the centre, where the probability is zero, erratic behaviour can be observed. This is because the absolute gradients can get excessively large due to the zero, causing instability in this region.
9

---

Published in Transactions on Machine Learning Research (04/2024)
0.40
Input
Transformed
0.35
Output
0.25
0.20
0.15
0.05
0.00
-3
-2
2
3
The input the standard Gaussian distribution while the output is a mixture of assorted dis- tributions, sampled and represented with a piecewise linear probability density function. A residual neural network is trained to convert the simple imput distribution into the more complex output distribution. The "transformed" graph the result of this conversion; it generated density estimate of 32768 points from the input distribution after they have been passed through the trained neural network. KL-divergence has again been minimised.
Figure 7:
To avoid this the probability should be adapted not get low, either using wide "slab" distribution by simply hacking the values.
The preceding demonstration is achievable with the inverse CDF transform, avoiding substantial complexity. A more realistic use case is as the training objective of a neural network. In Figure 7 network has been trained to distort draws from the standard Gaussian to match an arbitrary mixture, as represented with a piecewise linear distribution. In addition the three mixture components (triangle, uniform and Gaussian) the output distribution includes a low probability uniform slab distribution ensure stability. The network has two hidden layers of width 32, with Gaussian activations on all layers except the last, which remains linear. It is used as offset (residual) for point positions, such that the final layer can be initialised with small values so it starts close to an identity transform. ADAM (Kingma & Ba 2015) with 8192 iterations reduces the KL-divergence from 1.207 to 0.009. Stochastic gradient descent is used, L.e. each iteration a new sample of 256 points is drawn and pushed through the network for calculating the gradient.
6 Conclusion
This paper has introduced an important operation for a flexible distribution representation, including a rigorous exploration of how to implement it within practical algorithms. Consequentially, piecewise linear PDF representations can be used when constructing AI/ML algorithms where, previously, it was either computationally impractical or impossible. Potential applications can be found throughout the introduction and its use objective for neural network has been demonstrated..
A substantially extended (more rigorous) version of the derivation is presented in Appendix A including an alternative path for completing the derivation. The supplementary material contains generic library for calculating the differential cross-entropy, KT-divergence and entropy of piecewise linear PDFs, alongside standard operations such that it constitutes a complete library for working with such distributions'. Within this library code to generate all of the figures in this paper.
This library named orogram; same construction as kistogram except the wooden posts ("histo") have been switched for mountains o"o"")) reflect the triangular nature of piecewise linear PDF.
10

---

Published in Transactions on Machine Learning Research (04/2024)
References
Jan Beirlant, Alain Berlinet, and László Györfi On piecewise linear density estimators. Statistica Neer- landica, 53(3):287 308, 1999.
Nicolas Bonncel, Gabriel Peyré, and Marco Cuturi. Wasserstein barycentric coordinates: Histogram regres- sion using optimal transport. ACM Transactions on Graphics, (4), 2016.
Yuri Boykov, Olga Veksler, and Ramin Zabih. Fast approximate energy minimization via graph cuts. Pallern Analysis and Machine Intelligence, 23(11):1222-1233 2001.
Ingemar J. Cox, Sunita L. Hingorani, Satish B. Rao, and Bruce M. Maggs. A maximum likelihood stereo algorithm. Computer Vision and Image Underslanding, 63(3):542 567, 1996.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. North American Chopter of the Association for Computational Linguistics, 2019.
Pedro F. Felzenszwalb and Daniel P. Huttenlocher. Efficient belief propagation for early vision. International Journal of Compuler Vision, 70:41 54, 2006.
Thomas Ferguson. Bayesian analysis some nonparametric problems. The Annals of Statistics, pp- 209-230, 1973.
Kunihiko Fukushima. Cognitron: A self-organizing multilayered neural network. Biological Cybernetics, 20 (3-4):121-136, 1975.
Carl F. Gauss. Theoria combinationis observationum erroribus minimis obnoriae. Henricus Dicterich, 1823.
David Gibb, Course in Interpolation and Numerical Integration for the Mathematical Laboratory. G. Bell & Sons, Limited, 1916.
Pedro Gonnet. A review of error estimation in adaptive quadrature. Computing Surveys, 44(4), 2012.
Geoffrey E. Hinton, Peter Dayan, Brendan J. Frey, and Radford M. Neal. The 'wake-sleep' algorithm for unsupervised neural networks. Science, 268(5214):1158 1161, 1995.
Chi-san Ho, Paul Damien, and Stephen Walker. Bayesian mode regression using mixtures of triangular densities. Journal of Econometrics, 197(2):273 283, 2017.
Peter J. Huber. Projection pursuit. The Annals of Statistics, 13(2):435 475, 1985.
Aapo Hyvärinen. New approximations of differential entropy for independent component analysis and pro- jection pursuit. Advances Newral Information Processing Systems, 10:273-279, 1997.
IEEE. 754-2019 IEEE standard for floating-pois arithmetic, 2019.
Edwin T. Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620-630 630, 1957.
Edwin T. Jaynes. Information theory and statistical mechanics (lecture). Brandeis University Summer Instātute. Lectures in Theoretical Physics, pp. 181 218, 1963.
David Johnson. The triangular distríbution a proxy for the beta distribution in risk analysis. Journal of the Royal Statistical Society: Series D (The Statistician), 46(3):387 398, 1997.
M. C. Jones, M. Samiuddin, A. Н. Al-Harbey, and T. A. H. Maatouk. The edge frequency polygon. Biometrika, 85(1):233-225 1998.
Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. An introduction variational methods for graphical models. Machine Learning, 37:183-233 233, 1999.
11

---

Published in Transactions on Machine Learning Research (04/2024)
Christian Jutten and Jeanny Herault. Blind separation of sources, part i: An adaptive algorithm based neuromimetic architecture. Signal Processing, 24(1):1-10, 1991.
Dimitris Karlis and Evdokia Xekalaki. The polygonal distribution. Advances in Mathematical and Statistical Modeling, pp. 21-33, 2008.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2015.
Diederik P. Kingma and Max Welling, Auto-encoding variational Bayes. International Conference on Learn- ing Representations, 2014.
Solomon Kullback and Richard A. Leibler. On information and sufficiency. Annals of Mathematical Statistics, 22(1):79-86, 1951.
Aida C. G. Verdugo Lazo and Pushpa N. Rathie. On the entropy of continuous probability distributions. Transactions of Information Theory, 24(1):120 122, 1978.
Chien-Tai Lin, Jyh-Shyang Wu, and Chia-Hung Yen. A note on kernel polygons. Biometrika, 93(1):228-225 234, 2006.
Seppo Linnainmaa. Taylor expansion of the accumulated rounding error. BIT Numerical Malkemalics, 16: 146-166 1976.
Nicholas C. Metropolis, Arianna W. Rosenbluth, Marshall N. Rosenbluth, Augusta H. Teller, and Edward Teller. Equation of state calculations by fast computing machines. Journal of Chemical Physics, (1)):: 1087 1092, 1953.
Yurii Nesterov. A method of solving a convex programming problem with convergence rate o(1/k²). Soviet Mathematics Doklady, 27:372-376 1983.
Hien D. Nguyen and Geoff J. McLachlan. Maximum likelihood estimation of triangular and polygonal distributions. Communications in Statistics - Theory and Methods, 102:23-36 2-,23,6,622 2016.
Hien D. Nguyen and Geoff J. McLachlan. Some theoretical results regarding the polygonal distribution. Computational Statistics & Data Analysis, 47(0)):5083-5000 2018.
Frank W. J. Olver, Daniel W. Lozier, Ronald F. Boisvert, and Charles W. Clark. The NIST Handbook of Mathematical Functions. Cambridge University Press, 2010.
Emanuel Parzen. On estimation of probability density function and mode. Annals of Mathematical Statistics, 333)):0065 1076, 1962.
Karl Pearson. X. Contributions to the mathematical theory of evolution. - ii. Skew variation in homogeneous material. Philosophical Transactions of the Royal Society A, 156:343-414, 1895.
Francois Perron and Kerrie Mengersen. Bayesian nonparametric modeling using mixtures of triangular distributions. Biometrics, 57(2):518-522 2001.
Jerry Place and Jerry Stach. Efficient numerical integration using Gaussian quadrature. Simulation, 73(4): 232 232-238, 1999.
Simon J. D. Prince. Understanding Deep Learning. MIT Press, 2023.
Johan René van Dorp and Samuel Kotz. Generalized trapezoidal distributions. Metrika, 58:85-97, 2003. Murray Rosenblatt. Remarks on some nonparametric estimates of a density function. Annals of Mathematical Statistics, 27:832-837, 1956.
David W. Scott. Averaged shifted histograms: effective nonparametric density estimators in several dimen- sions. The Annals of Statistics, 13:1024-1040, 1985a.
12

---

Published in Transactions on Machine Learning Research (04/2024)
David W. Scott. Frequency polygons: theory and application. Journal of the American Statisticol Associa- tion, 80(390):348 -344, 1985b.
Ishwar K. Sethi and G. P. R. Sarvarayudu. Hierarchical classifier design using mutual information. Pallern Analysis and Machine Intelligence, 4:441-445, 1982.
Jayaram Sethuraman. A constructive definition of Dirichlet priors. Stalistica Sinica, 4(2):639 650, 1994.
Claude E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3): 379-423, 1948.
Thomas Simpson. On the advantage of taking the mean of a number of observations, in practical astronomy. Philosophical Transactions of the Royal Society, 49:82-93, 1755.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervisod learning using uonequilibrium thermodynamics. International Conference on Machine Learning, 32:2256
2265, 2015.
Michael E. Tarter and Richard A. Kronmal. An introduction to the implementation and theory of nonpara- metric density estimation. The American Statistician, 30(3):105-113 112, 1976.
Jure Zbontar and Yann LeCun. Computing the stereo matching cost with convolutional neural network. Computer Vision and Pattern Recognition, pp. 1592-1599, 2015.
Martin J. Wainwright and Michael I. Jordan. Graphical models, exponential families, and variational infer- ence. Foundations and Trends in Machine Learning, (1-22)11-30)) 2008.
David L. Wallace. Asymptotic approximations distributions. Annals of Mothematical Statistics, 29(3): 355-654, 1958.
Michael Vander Wielen and Ryan Vander Wielen. The general segmented distribution. Communiccations in Statistics - Theory and Methods, 44(10):1994-2009 2015.
A Full derivation
The process of doing the derivation was digital. Specifically, Atom² was used to edit Markdown with inline Latex equations using Mat Consequentially, it is excessively detailed: each step was done by copying and pasting the previous line then editing it. Below are the raw proofs; they probably double as a statement about mathematical paranoia. While inclusion necessary to ensure complete rigour, they are probably more valuable as an example of the difference between what goes into the body of paper and what's actually done, to inform and/or terrify future researchers. It does still serve the purpose of clarifying the steps of the derivation. Editing of the surrounding text has been done for "professionalism" and clarity, mostly adding details that were originally omitted and adapting to formatting changes, but the mathematical steps have not been edited at all.
There are two proofs; for completeness both are included. While the shorter derivation, as presented in the main paper, was considered at the start the original proof it was not explored, because the division by zero appeared problematic. It was only after finishing the original derivation, and resolving the limit, that approaching the problem with change of variables anonymous reviewer; special thanks are extended to whomever they may be. The limit calculation within the main paper utilises the techniques/fore of the original derivation.
reconsidered. This was at the urging of an
A defunct text editor; some later steps were done with inferior tools.
13

---

Published in Transactions on Machine Learning Research (04/2024)
A.1 Shorter derivation
Differential cross-entropy is defined as,
H(P.Q) - P(x) log Q(z)dr,
(30)
where the integral is over the (shared) domain of the two distributions, P and Q. In the linear case this becomes
=(P:Q)=- 8 ((L t)ps + 1p441) t)g + tq(+1)dt.
(31)
where the sum is over each piecewise linear section, with & that sections width (i to i+ 1). Lowercase letters indicate an evaluation the respective uppercase PDF, at the edge a linear segment: subscript i at the start of segment subscript i the end. Hence, we need to be able evaluate
-6: / ((1 ttpp + tps+1) log((1 - t) + tqb+1ddt. dtt.
-6
(32)
Define
Ap; Pi+1 Pt: Ags 4+41 48
(33)
and consider
-= (1 t)qs + t9:41 9: + Aqit
(34)
which can be used for a change of variables,
6; Aqi
91+1
Api (á 4:) log(O)dg. Aqi
Pi
(35)
Separate the two different integral forms,
di
{
Api
Qi
/ 94+1
log( +
Api (
[ glog(@)dg)
(36)
Ag:
Aq:
Ags Jq.
and slot. solutions to both (omitting the unknown offsets, as they cancel)
5;
{
(pt
Api
App
q² log (
1 2++1
[ (logg(() 11
(37)
Aqi
Aq.
q
{
4
5e Aq.
{
Ap;
9i (9+1 (log(g++1 - 1} 9i flog(q:) 1})
pt
Aq:
Ap:
log(g++)
1
(lo((s)
1 }))
+
9i+1
2
of
(38)
qi
2
2
then rearrange,
d:
{
Ap;
Aqi qi ) (qi+1 log(g:1) - 96+1 - q: log(g:))) + q:)
p::
- Aq
=
Ap: 2Agi
+ })} (39)
1 oh
444 log(a44))
o² log(g:)
2
2
6; {(Agip - Ap:g.) (gi+1 log(g:41) qi log((;)) + g. - qi+1)
(Aq.)²
+ log(g-1))))) -2 d
Api
2 }
(40)
14

---

Published in Transactions on Machine Learning Research (04/2024)
de
Api (gb+1 log(ui+1) log(a:))
(AqiPi - Aprq.) (qi-1 log(g++1) 9i log(g:)) + 2
(Ag.)²
Ap:
(af - -hiii (41)
+(Aqp: - Ap.ai) - 90+1) + 4
Explode the first term (within the curly brackets) and clean up,
(Ag.P: - Ap:q:) (9 log(9+1) - log(9)))
(42)
4oppit+1 10g(4+1)) - 929 Aq.p log(a:) + Apia, log(ai);
(43)
(++1 9:ppit+11 log(1++1) (pi+1 ps)4 10)9::4-+110(99+11 - (gi+1 q: p.4% log(g:) + (pi+1 - p:)ge log(g:),
(44)
P: log(4++1) P:0:0:+11111111 log(g:+) + P.4:4011 log((++)) 1 P1+10:0+11 log(1++1)
+ p. log(g: p2000+1 log(a) + 1i log(u:) p.dk log(us)
(45)
p. log(1++1) 11+19:91+111111 log(g=+ -: log(a) + P++ log(g:),
(46)
P1-1 log(4:) + pigē+1 log(1(+) - 4. 40 log((:) + pi+1 log(n=1)).
(47)
Same again for the second term,
Ap: (et1log(aa+)) - d² log(4:))
(48)
2
Pc+1 Pi
l log(9++1) Pi+1 Pi o log(a))
(49)
Pi+1 9:t1 log (4(11))))))))))))))()))
p.g.+ log(9++))
Pit
(og()))(())) log(1))
+ p:g log(a)
(50)
2 Pit 1 of log(g:)
2
2
2
Pog log(g++1)
P14194+1 log(9)+)
pigf log((;)
(51)
2 Note that, with reference to the last part of both of the above exploded terms,
2
22t1
qf 9.9++1
(4++1 - -1)²
(52)
Using this, both terms can be mergod get
P. log(g:)
p:g.+: log(g++1) + (g441 11)=1-4)220 4:) (p, log(a) + log(4(1)) pige+ log(gs)
Pi+1 log(9++1)
2
(53)
where the third line corrects for the mismatch of the second line. Simplify to get
pigZai
Pi+1 2
(log(u++1) log(4))) + 1 (An))(p log(g) ps+ log(g++))..
(54)
Now rearrange the last two terms from within the curly brackets,
(Au.p Apiai)a q(t1) Api (0² (2+1)
(55)
-(Dq.p App Api (q: + gi+t) Agy.
(56)
Aqs |L0q.P: 4Apiq. + Ap.ii + Ap.g+1),
(57)
4
Aqi
[4p.
-p.4. 4p:+19 + 4p:9. + Pi+19: p.q. + Ps+19:+1 - Peqi41)
(58)
4
15

---

Published in Transactions on Machine Learning Research (04/2024)
Aii [3p:4++ 3p:+19: - Piti + P++19++11,
(59)
Aq [(3p: + pi+1) (pi + 3p:+1)al
(60)
Bring all of the simplified terms back into the main equation,
d4
1 pi - -Pi+1% (log(1+1)) - log(g:)) + (Aq)) log(4:) + Pi+1 log(a++))
(Ag;)²
2
2
Ag:
[3p: + P1+1))0-11 (p= + 3ptti)al }.
(61)
4
and rearrange,
= pitl+1 - Di+1 (log(g:+1) log(a:)) + (p; log(g:)) + Pit1 log(1:+1) 2(D
[3p + Ps+1 1141)/6-11 (Pr + 3pitiDal.. (62)
4 4Aq
obtain the final form,
= { (p, log(g:) + 10g(9++1)
2
Pi+19? -Pili+1
(log(g++1) - log(4;))
2(gi+1 q:)²
(3p: + P 10-1)4+111 (p: + (.3p+1) )4:
4(q (:) S (63)
A.2 Original derivation
Start from the need to evaluate
/ ((1 t)p, + (p:41) log((1 - t)gs + ttt+1) dt.
(64)
This time, consider the series
log(a) =2
Σ
1
a
(65)
n
a
nE (1,,,,,,,....)
defined for a > O. Use it to rewrite the objective as
8, I ((1 t)p, tp+t1) 2
1
(1 t)q + tqp+1
n
Σ
dt.
(66)
n
(1 t)g + tq +
(,,,,,,,,...
((1 t)p; + (pi+1)
(1 - t)gi + tggi1111111111 - 1
n
-25
Σ
dl,
(67)
n
11 t)gi + 145+1 +
nE (1.
-28
Σ
1
(ps + (pt+1 p.)t)
q: + (g++1 9:) dt.
+ (gi+1 9:){ n
(68)
n Jo
(1,,,..5....
Use
d Dit+ (p--1- - pp))t
= + (pi+1 - p.)t,
(69)
Equaaion 4.6.4, P.108 of NIST Handhook Mathematical Functions, 2010
16

---

Published in Transactions on Machine Learning Research (04/2024)
plus
(g: - 1 + (gi+1 g:)()m
(70)
dt (% + 1 + (4i+1 - 4:)t)m
de (q: 1 + (9++1 - 96 ())
d
+ (g: 1 + (4t+1 - q:))))² dt (g: + 1 (+++1 - 9: ):)
(71)
(g: + 1 + (git1 4))
m(g: 1 1 (gi+1 9:) 9:) mn (gi+1 91))
- (q: 1 + (at+1 ):)0) n(g: + 1 (gi+1 )t) mm (9i+1 93)
(72)
(q: + 1 + (qi+1 - q:)()m
(4) 1 + (gi+1 - 9: 0z)1)2n
m(1 1 + (+++1 - 9:)m) n(4 1 + (f++ - -))))
(gi+1 - 4:)
(73)
(a: + 1 (4+1 )t)² (9 + 1 + (9+1 9.)t)=+1
where some generalising has been done pre-emptively by introducing m S in anticipation of later steps, though for this first use n = m. Apply integration by parts to get
- 22
Σ
1
[[mm+
mi++
(pi+) - -Pi)(²)
9: - 1+ (gi+1 - 9:)6
n
2
+ 1 (gi+1 - 9:)t
nE (1,55,,5.....
- -fo'
pit + (pi+1 - =pi)
(q: 1 + (gi+1 q1)
(q: 1 + (gi+1 - q:)t)
n(gz-1 -g)
dt
2
[g: + 1 + (gi+1 9:))t)
(g: + 1 + (9 q:)t)atT
(74)
- 20;
pi + Ps-1
Σ
1
q: 1 + (4++1 - 9:)) )r}
2
9: + 1 (g++1 9t))
(,,,,,,,...
{/o- pit
(pi+1 - -p:)(?
-9:+11 (i))
Σ
net 1,,,,,,5.....
(y) + (yi+1 2;)()==1
(g: 1 1 (gi+1 9:)()
11
x
(qi + 1 + (gi+1 - q:)t))
(q: + + (4i+1 93 (1)())+)
dt ƒJ
(75)
Pi Pi-
log(9++1) + 28: - -.) Σ (pst (pit1 - -D:)(2)
(g: 1 + (gi+1 -=))))-1
8,
(y) + 1 + (g++1 2.)(m dt
2
2
x.51,3.5,....
Uh' (pit+ (D4+1- -D()e
(q:
(q++1 q,)t)"
- -26,(94+1- - qi)
Σ
(gi + 1 + (gi+1 - q:) g:)e)m+z dt j. (76)
2
DE (
Pi + 1 log(g++1) 26, qi) Σ
8c
(-1)a {1'0
(pil+ (pi+1 - -p:)(3)
(g: 1 + (gi+1 q:) g;))
-8,
(pil+
(gi 1 + (git1 - 13)2)² dl
2
72
(77)
Consider the remaining integration,
(pot+ (pi41 - p.){2) 10:1-2p)t2
(qi 1 (gi+1 9.)t)m-1
(g: + 1 + (gi+1 - ()) dt,
(78)
2
and prepare for integration by parts again. Need
pal2
(pi+I- pi) = Pel + (pi+1 p2))(
(79)
dt 2
6
2
in addition to the above, where this time the generalisation is used. Stepping through,
8. Pi + 1
28 1og(g--1)
9:)
Σ
(1-07/(( pit+ (p++1- - pi)) (
(-1)
(((ptt
(gi 1 (qi+1 4:)1)²²1
(g. + 1 + (9++1 - g)) dt
2
2
(80)
17

---

Published in Transactions on Machine Learning Research (04/2024)
Di + Pi-1
8,
log(zi+1)
2
[
pitt²
2 + (pi+1 - -P:)tt)
(gs - 1 + (96+1 9:) q:)
-
-20:((i+1 43)
(-1)
(q: + (gi+) - 7:)1)²) 10 -(ic+1 -%a
6
n=1
pit2
(pi+1 - p))
{
(n
l) - +1 (gi+1 q:) q:)()==² n(q: 1 + (gi+1 (;)6)m=11
(q: + 1 + (9t+1 - 9;))) } 1ll
x
+
2
6
(9i + 1 + (9t+1 - 9:: ())
(81)
Pi + Pi-1
log(g+41) 20:(01+1 - qi) Σ (-1)² {
Pi +
(pi+1 - pi))
(a: - 1 + (gi+1 9:)))--
8,
2
6
(q: + + (gi+1 - 4:))
n=1
8
1- {1/²
ppl22 (No+1 - -Pi))))
(n () - 1 + (9h+1 - 1// 1))
+ 20:(1 - -2i²2
Σ
dt
2
(g: + +1 + (gi+1 9t)
Σ (-1) {1-
Pite
(pit1 (aii-pj) p.)00))
{
n(a: + (q:+1 q:)()(=11 (q: + + (gt+1 - 4: (t)
di }
(82)
6
Note first entry of sequence of second line is zero, and offset to align powers with sequence of third line,
8.
Pi + Pi-1
- 5:(1:41 log(zi+1)
-4)
Pi +
(pi+1 pp.)
Σ
(-1)) { (gi+1 - 1)m-1
{ (7++1 1)m
2
3
n=1
1)a (1/'
1
1p.12
(Pi+1 - 1:)13
{ [n(8 - -1 (96+1 - -4:)) (2:)))=11
- 25: - -
+
dt
2
6
(9i + 1 + (gi+1 -4:) -b))))m+
1==1
pit²
(pi+1 - pi)
n(a: - 1 + (a++1 - q1)1)a²1
(q: + (get1 (1)tm++1 dl } (83)
+ n=1
(-1)²
2
6
8,
Pi + Pi-1
log(0++) - 5, 9i+1 9i
(pp + Pi-1
Σ
00
(-1)
(git1 - 1)m
2
9i+1 +
3
(at+1 + 1)a-I
n=1
20: - -q:)2
Σ
00
1mm {1/0
(nd?+
(pi+1 pt)tt)
Inq: - -1+ (gi+1 - g.) (g)t)-)))t)mm)
di |di
(84)
3
(g/ +1+ (gx-1 9: tm)+
n=1
Focus on second term on first line, which is an infinite geometric sequence,
8, 9++1 9
2pp + Pi+1
Σ
(gi+1 1)"-1 (gi+1 1)a
(85)
1641 +
3
n=1
+ + 96+1 - 4i
(2p; + Pi+1
80
(-1)²
(gi+1 - 1)m
(86)
h++1 + 1
3
[gi+1 + 1)
n=0
+6, 4:41 qi
(2p; + Pi+1
4i+1 1 n 4++1 +
(87)
6+1
+
and use Σx= p. T- as it is the case that |r| < by construction,
+8, 1:+1 qi
2pp + Pi+1
(88)
4+11 + 1
3
1 - Ti 111t1
+ 9++ di
2pp + Pi+1
4s+1 + 1
(89)
4:+1 +1 1
3
1:+1 + 1 + 9:41 1
18

---

Published in Transactions on Machine Learning Research (04/2024)
(pp + p141))(1+11 q:)
+6,
(90)
69.+1
Now put it back into the original equation,
8, Pi + P
(2ps + P141)(9+11 9i)
log()i+1) + 6,
2
69i+1
Σ n(-1)) (%/ (pi2+ + (D141- pt)63)
00
{
(g= 1 + (gi+1 9.)))a=1
29 - q:)²
(q: + +1+ (4++1 - g:)t)att dt
(91)
Time for yet another integration of parts, which needs
d (pit² (pi+1 p:)t* pit (Ps-1 pt)
(92)
dt 3
12
3
plus the generalisod derivative from above; consider just the final term,
220 - -41)² Σ nx(1))² {%' [ptf+ (pi+1 -p.)t² ) {
00
(q: - 1 (qi+1 4:)t)
(g: + 1 (g++1 9: )t)) dt
(93)
{((2) pits
(Pi+1 - pi)ct)
( (g: + (qi+1 - )z)t)=
- -60((gi+1 - -2:)2 Σ n 11-1)"
3 +
12
(qi + 1 + (qi+1 - 2: )) m
nm
/-²
1
pit3
3 + (pi+1 pi)(
(g++1 (t)
12
(n 1) 1 + (gi+1 q: (;)))²
(n + 1)(g: 1 + (gi+1 9.)t)
dt } (94)
do
x
(y: + 1 + (gi+1 - ())()
(16 + 1 + (+++1 - 9: m;)(mm+
8
n (-11) { Pt + (pit1 - pi))
(qi41 + ( 1)a+1 }
( (16+1 - 1)m
25 - 9:)² Σ
12
(a(-1) 1)a² (1''
p1t33
3 + (pi+1 piitt
(n 1)(4: - 1 (gi+1 g:)t)m=
+ +20.(9.-1 91) 9b)
dt
12
(g: + (gt+1 - 9:
n=1
- -26, - 2:33
Σ n 1-1"
(1'
pit
(p++1 - pi)tt)
{
(n+ 1)(9: + (9 - 9:) 9()))
dt ) (95)
3
12
(g. + 1 + (gi+1 - g))
n=1
Second line is zero for first entry in sequence, so shift along to align with last line,
n 1(-1)" { [pp; + Pi+1
(gi+1 1)m-111 }}
- -28:(gi+1 - 4: Σ
11)+i)
12
(gi+1 1)m+I
aml
a(n + 1)(-1) {%-
+ (pi+1 - pi)(
(q: - 1 + (9++1 - 43:4)) 4;)()mm-1
-25,(93-1 - 9: 3
{
dl
3
12
(q: + 11 + (9:+1 9: )t) ))
n=1
xc
nnn + 1)(-1) {T-
pet3
(pit1 pp))
{ (g: - + (gi+1 - 9:: m))
26,((9+1 - 9:)3
(q + +1+ (9++1 - g.)t)aa22 dt (96)
12
3p: + Pi++1
8
(gi+1 1)m
-25:(ge+1 - 4:)2
Σ
(n + 1)(-1)"
12
(g++1 + 1)
4
n(n + 1)( {T/² (p.t3 + (pit1 - pi) {
(g: 1 1+ (gi+1 - 9:)
5,(1 9;)3
(g. 1 + (gt+1 g.) dt (97)
3
4
n=1
19

---

Published in Transactions on Machine Learning Research (04/2024)
3
2 gi+1 qi
3p; (3p; + Pi+1
Σ
20
(n (m+1) (- 4i+1 - 1) "
gi+1 + 1
4
9++1 +1
4
(:
9:)3 n=1
n(n + 1-(1)"
{fo
(1it3=
(Di+1 - pi))
)
{
(9: 1+ (g++1 - 9:)1)m=11
(g +++ (9++1 (.) dt ). (98)
3
4
The infinite sequence in the first line is example of a polylogarithm, specifically, it is equivalent to two evaluations of, such that
2
9:+1 qi
2
3p; + Pi+1
1
9++1+1
9:
3
9i+1
+ 1
4
++
8:11-1
1
2: 111-1
2
94+++
9141+1
4
n(n + 1)1" {%/- (pic8
(pit1 - ps))t*
{
(q: 1 1+ (gi+1 - 9.)t)a=1)
=
6,(9/+1 9:)² Σ
[g: ++ (4i+1 - (:) dt (99)
3
4
n=1
2
(3p; + Pit1
9i+1 -9: qi
2
9++1 +1
(96-1 + 1)((+11 - 1)
=
4
li+1 + 1
24141
406=11
4
8
(ne + (Dit1
- -
{
(g: - 1 + (g+41 - (1)()==1 dl dll (100)
5i((ct11 -4:33
Σ
n(n + 1)(-1))
3
4
(g: + + (git1 - (:)1)
(3p, + + P141)
-2)2
+ 8,
24
4
(h'² (pet + (pit1 -mvee)
{
(q: - + (gi+1 - q;)())m=1
5.g0t1 -1)² C(nn+11 1)(-1)²
(gi + + (gi+1 u+2 dt j. (101)
3
4
n=1
Now bring back the rest of the equation.
Pi
+ Pi-
1og(1++1) + +0 (2p; + Pi+ (9tt1 - 9 + 0:
(3pi + Pi+1)(91+11 - 99
- 8,
2
69u+1
24
4
Σ n(n+ In 1)(-1)" {1'' (110* (Pi+1 - pi)ta -Ditt) n=l
(g) - 1 (gi+1 m1) dt j. (102)
did+11 9:)3
3
4
(gi + 1 + (gi41 (2)))
This is suggestive, but not enough terms to be sure, hence repeat again. Going to need
dt (4 4 + (pi+1 pc)1
d (Pplt
=p,t + (pi+1 - p2:)
(103)
20
4
and the generalised derivative above. As before, consider only the final line,
4
n(n + 1)(-1) ( (1eos + (pit1 -P:))
)
{
(g: 1 + (a:+1 - g:)t)n=1 dt
- 0,9111 -;)3
[g: 1 (g++1 (:))))+z d (104)
4
n=1
4
5: - -
Σ n=1
xc
nnn + 1)(-1)
{[(2))
(pitt
+ (piti - -1:)c5)
(g: + (git1 - 9:)
3
4
20
(q: + 1 + (q++1 g))
L-¹
Dit
(pi+1 p:)ts
+
(git1 4:)
4
20
(n 1) + (4++1 - )22
(n 2) + (gi+1 9:)()==1 dt } (105)
(q) + 1 + (git1
(g: + +1 + (git1 m)
20

---

Published in Transactions on Machine Learning Research (04/2024)
4
x n(n + 1)(-1)) {(
pi + (pi+1 - ps))
(q: - 1 + (gi41 9:))a
3 S 9:)² 3
20
(g: + + (git1 - :q:))
(n + 1)(-)) {1/0 ( Pitt + (pi+1 - -Px)t
(
(n 1)( - 1 + (gi+1 9:)1 g1)1)m=- (g: + 1 + (9i+1 9: 2:)()u+2
+ =0 0,(91+1 -;)²*
dt
3
4
20
n=1
4
20
{['
patt + (pi+1 - pi)tt)
(n + 2) 1 1 (9x+1 - 9:)6)m=1
Σ m(n+ nn(++)(-11)
3
20
(gs + ++ (9i-1 - 9: 0:))))+33
1=
(106)
Note that first entry of the second line's summation is zero, so shift to align with third line,
4 8s (p(t ( +
(pit1 -pi))
{ (9 - 1)m-1
(0t+1 - -%)²
n + 1)(-1)"
12
5
(9 + 1)m+t2
n=1
nnn + 1) + 2-) (1'²
pitt
(pi+1 - -Pi)t
( (9: 1 1 (9i+1 q:)t)==
6,(9 - 9i)²
Σ
dt
3
4
20
(g: + 1 + (gi+1 9: )
n=1
4 3
0: -44 4 Σ
nc
n(n (1++))(+2))(11)) { //'²
(pttt
(pi+1
p.)t
(g. 1+ (g++1 91)
(os + 1 + (mh-1 - y: )))+3 dt
20
(107)
4
5,
( Ap; + Pi-1
(#i+1 9:)3² Σ
20
(n+ 1)(n + +))(-1)² { (9++1- - 1)"
12
5
(gi++1 + 1)at3
n=0
8 3
6 -q,)²
Σ
(nn+1)(++2((
{%/0
(4
(pit"
(De+1 -pi)0)
(q: + 1 + (9.-1 gt m)) dt
(g:
1 + (g:+1 9: 9;)0)a=1
20
(108)
4
8,
4p: + Pi-1
(9++1 9: 00
Σ (2 + 3n + n2?
2(
(_ht1 1 n
12
5
(gi+1 + 1)3 n=
91+1 +
2
(
-9:²*
Σ
n(n+ +) (1/ (Dit* (pi+1 - -pitte)
(q: - 1+ (9 - 9:) dt
(4: + 1 (g:-1 93 (ot+3
1=1
(109)
The infinite scries of the first line is equivalent to three polylogarithms of different order,
1
6/
4p;
+
Pi+1
(g+t1 - 33
2
3
9++++
9441-1
9 11 9++++I
4++1+1
3
5
(I++1 + 1)3
1+
9++++I
9:11-
1
0++++
9:11-1
2
1+
9:11-1
1++1+1
2
d - 9:)4 Σ
a(n+ 1)()+)
{1/-
(pit+ + (pe+1 - -p()t
(g. 1 1+ (g++1 9:)
dt
(q + 1 + (4h-1 - 9:) ()++3
(110)
1
[4p, + Pi+1
(M41 - 43)3
1141 +1
3(0 - 1) + +12
2(1441 - 1)( + 1)31
+
5
(/+11 + 1)3
9--1
4(g:41 1)0
8(1:+1 + 1)2
2
((n+1)(++2) 21) {T'
(pit + (Ni+1 - pi))
(q: - 1 + (9h+1 - m))
di
3
5
(gs + 1 (9h-1 -t)+33
==1
(111)
21

---

Published in Transactions on Machine Learning Research (04/2024)
1
4p, + Pi+1
(9++1 - 9:)²
44141(9441 +1)2 132--(9:+1 1)(9-11 +12 2(9 1) 1)88
6/
3
5
(x+1 + 1)3
4(gi+1 + +1)
8(9 - 1)2
2
20
((((xtt (pit*+ (Dit1 - -pi)tt)
(g: 1 + (9++1 - -1i)))==1)
9:)*
a(n+ + (2(+2)(211))))))
dt
3
5
(as + 1 + (gh-1 - ::)t)ntt3
==1
(112)
1.
4pp + Pi+1
(9t+1 -
[9-1+ + 3)(411 + +1)
2(9++1 (9 + 1)
+
8
5
(9++1 1)33
4(qi+1 + +1)g2t1
8(gi+1 1))0//11
2 3
{%/- (pit* + (Di+1 - pule))
(1: -1+ (9s+1 - 9:) (=)1)m==
n(n+ + 1)(n+)))(++))))))))))))))
(g: + 1 1+ (96-1 96 tt)+3 dt
5
1=1
(113)
1
[pp; + Pi+1
(gi+1 - -
24441(9441 + +3)(111 + +1)3 - -2(()-1 - 1)((111 + 1)3
5
(q++1 + 1)3
8(9++1 + =1)74'+1
2
(9 - 9:)*
8
n(n+ 1)(n (2)=)))-))))))
{f (pitt (ptttt (pi+1 - -Di)t*)
)
( ((9) 1 (g++1 (:)6)m-1>
(%: + 1 + (16-1 - 9z m) dt
(114)
1
App + Pit1
(at+1 - 4:
(gi+1 + 1)
+
5:
3
5
(+++1 + 1)3
[4(9it1 + 1)2
a(n+ + 1) (((((=2)--)))))) {%/' (pat* (pi+1 -
( (4: 1 + (g++1 9;)()m-1 dt
5:(1111
9:)*
3
5
(g: + 1 + (9h-1 - (:)t)a+3
==
(115)
(4p; + P141) 43
8,
604241
2 (: - 41)* Σ n(n+ (a(++1) m(n+1)(x+2)(11))) ( (pit + (pit1 - -)))e)
(
(g: 1 + (g++1 9:)6)m=1
(4: + 1 (96-1 9: )))++ dl
5
(116)
Now bring back the rest of the terms to get
P: PP+1
(g(()+1) +5, (2p; + P141) - 9i) +6, (3p: + P.P11 (9 - q1)2
+6, (4p: + +141) 9:)
8.
2
69i+1
24
60g;'
2
8
n(n + 1)(()) (()(==2)(())))) { {1' (2it* (pit1 - pi)tt)
(9: 1 (9++1 - 9:)6)m=11
8,
(g 9:)
Σ
dt
3
(qi + 1 + (9i-1 - 9::): ())33
#=1
(117)
From this see the sequence is?
Pi + Pi+1
(n + 1)p; + Pi+1 n(n + 1)(n 2)
qi+1 -
8,
log(g++) + 8,
(118)
2
4+
This is a pretty good solution, at least computationally speaking, and it has been verified by comparison to numerical integration. It is unstable however and that last infinite sequence can be removed through the use of polylogarithms.
Divisor https:/ /ttps://oeis.org// 0rt00//ssor//0/03311 A007531
22

---

Published in Transactions on Machine Learning Research (04/2024)
Start with
(n - (n + + 3) + (n 1)n(n +2)
( 33
+ n(n + 1)(n+t + 3) 4(n + 2)
(119)
n(n 1n +
4n
4(n +)
as derived in Subsection
A.2.2
This allows the goal to be rewritten
8
Pi
P
log(gi+1)
2
(n (n + 2)((n + 1)ps + +Pt1))
gi+1 46
+ 6
4n
9i+1
==1
00
2 (n (n 1)p; + P(+1) (n 1)n(n 2)((n 1)pt + Pi+1)
9++1 qi gi+1
6,
4(n +
n(n + 1)(n +3)(()) + 1)p: pit1) 4(n + 2)
9++1 qi n
+ 8
(120)
a++1
and then offset the sequences so the denominator just 4n,
8
pi
P
log(fi+1)
2
(n- 1n + +22 + 1)p; + Pit1) 9i+1
6
4n
gi+1
==1
nn 1)²(n + ()(p + Pp--1) + (n 1)(n + 1)(pp, + pi+1) 9i+11 - 96
n-1
5
4n
46+1
n=2
8
(n 2) 1) 1)((n 1)p; + D++1)
qi+1 - qi
n-2
6
(121)
4n
4i+1
7=3
Now correct the sequences to start from one by noting that the early terms are zero and correct for the exponents by pre-multiplication. It should be noted that there is a dependency on Olog(0) = being the case here (Lebesgue integration), as expected for cross-entropy.
Pi P:-1 log(4++1) 8,
2
(n 1) + 2)((n + 1)p; + P1+1)
[i+1 Lb
4n
4i+1
gi+1
(n 1) + 2)(p + P(+1) + (n 2) 1)(n 1) + Pi+1)
[i+1 qi
6:
01+1
4n
di+1
nml
+ 5g
4i+1
00
(n - 2n 1)(n 1)((n - 1)px p1+1)
[i+t - Li
(122)
9i+1
qi
4n
gi+1
==1
Now explode each summed fraction in turn. First:
(n (n + 2) + 1)p; + Ps+1) (n 1)2(n + 1)(n +)pp + (n 1)2(n 2)pp++1
(123)
4n
4n
(n²
2n + 1)(n² +3 + 2)p; + (ni
2n + 1)(n + 2)p++1
(124)
4n
2n + n3
3n.
+ 2)p;
(n3 3n + 2)p+++
(125)
4n
n Pt
2 Pi
3mp;
Pi
Pi
n
Ps+1
3pit1
Pi+1
+
+
+
(126)
4
4
4
2n
4
4
2n
23

---

Published in Transactions on Machine Learning Research (04/2024)
n Pt (p + Pit1)
3np;
pj + 3p;+1
+ pi + P++1
+
(127)
4
4
4
4
2n
Second:
(n 1) n + ()pp + P1+11) + (n-
2)(n - 1)(n + 1)(pp; + P1+1)
(128)
4n
(n²
2n + 1)m(n + 2)p; + (n²
2n + 1)(n + 2)p++1
4n
(n² 3n + 2)m(n + 1)p; + (n² - 3n + 2) + 1)p++1
(129)
4n
- (n²-3n"
+ 2n)pp + (n -3n + 2)pi+1 + (n 2n + 2n)p; + (n" - 2n²² 2)p++1
(130)
4n
(2m -2n² - -nn² + +4n)p: + (2n2 2n² - An + 44pi+1
(131)
4n
ni n'pp
n² np(+1+ - pi)
n(2p; + Pi+1)
Pi+1
+ (p: -Pi+1) +
(132)
2
2
2
n
Third:
(n - (n (n + 1)((n - 1)p; + pi+1) (n²² 3n + 2) + 1)(n 1)p; + (n²² - 3n + 2)(n + 1)p++1
4n
4n
(133)
(n - -33 n2 + 3n 2)p; (n² - 2n n 2)p
(134)
4n
nea p:
n² (pi+1 3p;)
n(p: 2pi+1)
(3ps - Pi+1)
Pi+1 Pz
(135)
2n
For notational convenience define pro-filled in polylogarithm of order
Li.()) = Σ
9s+1
(136)
n
9i-1
n=1
then put it all together and rewrite the equation as
6
Pi +
Pi+1
log(g(+1) +
Pepi+1
Piq
Li
2
2(g(41 9.)
4(g 9:)3
+5,
Pi
P++1
(pi+1
Dt)
(pi+1 3p,)gk+
Li
2(g:+1 9c)
4(g++1 - 9,)2
3p;
(-2p;
pi+1)9i+1
(pi - (3pi+1)9...
+0,
Li
4
2(gi+1 9i)
4(g:+1 - 93)
+6;
Pi
3pit1
(p:
1i+1)g441
(3p; Pi++ P1+1)4241
Lio(.)
(g/+1 (c)
4(d - 9.)2
(pi+1 -P:)
+5;
Pi pi+1
Pi+1
Lir(-)-
(137)
2
(gi+1 4b)
2(g++1 - q:)2
This has been coded and verified, and unlike the carlier equation is numerically stable. But it's mess, so simplify much as possible.
Consider each term in turn, starting by dropping closed form definitions for the polylogarithms in, starting with order 33:
8,
Ps
Pid++1
P. Li_3(),
(138)
4
2(g+1
9.)
4(g.41
9:)3
24

---

Published in Transactions on Machine Learning Research (04/2024)
2111
(1 + 4 12:11-
d pilli41 - 4:)² - 2p;.i 1 (qi41 q:) + prgi+1
9++1
1:13
(139)
4(9i+1 -qi q1)²
gi+1 -0:
9:11
pigi+1 2p:9:94+1 p.g. 2p:94+1 + 2p.q:9++1 p.oi-1) (git1 4:) (1 + 19+11110
+
1:
4(9++1 - -;)²
Di+1
9:+1
(140)
Piq.
Lit1 + +4 (9++1 - 9:)) + 9i+1 - -i)2
6
(141)
4(q 4:)
pigp
Vi+1 + 44
44.
+ uht11 -2q,4/+1 184411
(142)
4(qi+1 - 95)
p.ge
6a
69: + g2
6,
(143)
4(1i+1 9c)
Sp.gf99++ (9t+1 9i)
P:9 96-1
8,
+ 49. (9++1 - -gt)
(144)
4q 19++1 - -gi)
3p:
Piqi+1
8. 20
+ 8i
(145)
A(git1 -ii)
Now order -2:
8,
pj + Pi+1
(pi+1 - p: p:)gi+1
+ (pi+1 - 3p: 3p:)
Li=)(-),
(146)
4
2(gi+1 - -1))
4(g 91)2
9-+1-9
2-+1-9
(pi + pi+1))(++1 - 9:)2 - 2(pi+1 - pi) (git1 (.) + (pit1 3/::)0f41
8
9:11 (1
1:
(147)
4(qi+1 - 42
26+1 -Q4
gi+1
prsi+1 2p/4/44111 + put + p1+14/+11 211419:0441 + Ps+19
221+10.+11 + +2pigf+1 + 2pi+19:9it1 2p:gide+1 + P141 - 3pr
(ait1 9i) (1 11
9--1 9g
8,
0:41
4(gi+1 -q:)2
9i+1 0
(148)
8, Pog. + Pi+19. - 4pi4/9i+1
R2- + 1041(19-1) 9:)
(149)
4(9i+1 9:)
(p + Ps+ u 111 - 4pr0: i -4)
+ *P++1 2220 4p
6.
+
(150)
4q (gi+1 -9:) 9:)
4q (9++1 - qi)
Polib+11 P1+19:0++1 4pr0f+1
+ Pigeti+1
Pi+ 1i+11900221
4p:
8.
(151)
4g
4g (g++1 - 9i)
4p.9.-
P:0.0++1(944 qi) + P1+19191++1
(9.+1 (9.41
q:) + p.g:0.+1 + Pi+11.9941 - 4p
8,
+
8,
(152)
4q
49 49141 9t)
Pag..1
2
2p:9.92
2 p:g. + 2p:+19:02.1 -1it19 4p192.1
8:
(153)
4q (ai+1 - 93))
-iifit1 5,
-
pof211
Di+ 192-1
-Peli+1
+ 6, -
+ 6;
+
5x
+ +. 2q (ai+1 - -qi)
+ 0 4(gi41 - -qi) 4(g:41 9i)
(154)
d
(2(44
-4)
2q,
(qit1 -4))
Order -1:
-3p; 3pp
(-2p: - -P++119411
(p: (p:-22 10:+1p )9 Ii
5,
+
(155)
4
2(9.+1 - 9))
4(g.41 9:)2
25

---

Published in Transactions on Machine Learning Research (04/2024)
3pil9t+1 9:)² 2(2p + *Pt+ 1 - 9:) + (pi 2pct
9.+-1
6,
0:
(156)
4(0i+1 9:)2
2
16+11
0i++1
3p + 6p:9:94+1 3p:q; 4p:1.+1 2p++19i+1
4p:4//1+1 -2p++1 + p:uj+1 33p+419741
g++ gt)
(157)
4(9i+1 - 9:)2
96+1 0f
2p:0f+1 + 2p.9:0i41 2pi+1 -3pid 95-1
8,
(158)
4(gi+1 9:)
q
2p10.+1 + 2p:9.0 2p1+19:01+1 - 3pp
8,
(159)
49, (gi+1 9i)
P:
P+0l+1
Pit 11+10:+1
-3p10++1
8:
+ 8
+ 8. 24 (9i-1 9/))
8,
(160)
24 [gi+1
9:)
29: (1i+1
9:)
4(qi+1 - 9.)
And then order
6
Pi
3pi41
(pi pi+ 1)94+1
(3p; Pi+1 )o2t1
Lio(.),
(161)
4
(9++1 - q:)
4(gi+1 9:)²
(p: + 3ps+ (16+1) 9/)²
1(p: Pi+1 1 (i) + (3p; - )4+21
0:41
6,
9:11
(162)
4(1i+1 - 9:)3
21++1 -Q
9c+1
pi 2pi psg? 3p + 66p+1999441 - -3p+19
4p:0i+1 + 4pig:0i+1 + Ap:+107 4pi+10:9441 + +3:.9 Pi+14i+1
(gi+1 ②i)
6,
(163)
4(9 )
gi+1 9:11
2prg2+1 p.
3pi+ of + 6p: + 2p;-19:9--
8,
(164)
4q.(9 9:)
8 09:(9/+11
P-
+ 5 A(g,+1
Piqe
8,
3pi+1%
+ 6, 3ptd++1
+ Pi+19i+1
(165)
9:)
91)
4(gi+1 - -9.)
2(9 96)
2( - -%)
Finally, order ==1 1:
di
Pi + P++1
Pi+10++1
+ (pi+1 pt)9 Lir()),
(166)
2
(9:41 - 9:)
2(gb+1 9:
(pr (pi+Pettl)) + qi)²
2pt+
10++1 (9++1 - qi) + (pi+1 - pi)get1
gi+1 qi
6i
log
(167)
2(9:+1 91)2
gi+1
2p14:0i+1 + piot Pi+ 14 2p + Pi+
2pp+ 02-1 + 2p; 1444c+1 + 914107-1 Peg
-6
log
45
(168)
2(g+11 - 9:
gi+1
2p.9194+1 - -P:9, Pi++1
6
(log(a) - log(g+1))
(169)
2(gh41 9.)2
26

---

Published in Transactions on Machine Learning Research (04/2024)
Now stick all of the above terms together,
Pi P:-1
3p.
Piq++1
6,
log(9++1)
+5,
6,
2
29h
4(q++1 96)
+ 5,
Pig
poq
+ +b; 29: 1i11
Pi+19++1
-P:6++1
+ 4(9:+1 9:)
-Pt+19
o
9: 1itt1
91)
29: gitt1 - 9i))
9i)
4(qi+1 - 9:)
Piti+1
pid.+1
Di+1
Pii+11+t1
1.t11
-3p;%+1
+ 8 24 (i++1
+ 6
+ 6: 29,
+ 6:
- -91)
2q
(it11 - -%i)
-%:
4(91+1 9t))
+ 6
Piql+1
+ 6: 4(gi+1 - 4i)
+ 6: A(git1 9:)
-3p:+19;
8;
3p1dt+
+ 6: Pi+1 9i+1
2q:
-9:) 9c))
2 - 9i)
2( 11 9:)
2p:9:94+1
p.q. Pi+19
+ 6
(Iog(q)) log( (170)
2(qi+1 - 9:)²
Start by considering only the log terms,
+ Pi-1 log(d:+1) + ò, 2p:0:0it1 p.g. Pi+108 (log(u:) - log((:+1))
6,
(171)
2
2(1(+1 9,)2
2p:0:0++1 log(g:) p.q. log(4;) - 1i+1 log(2:) 2pr404+11 log(z++1)
+.g (og(g++1) + Pi+1 log(9)+1) - (p: + ps+1 (9+1) - 9:)3 log(gi+1)
52
(172)
2(x+11 - 9:)2
6:
2(9+11 - q.)² (2p:9;9--1 p.q log(4)) -i+1 log(4:) 10949441ogg944111 + p: log(9:11)
+P++ of log(g++1) -pedt log(9.+1) + 2p.4./+41 log(g++1 p: log(1:+1) P.+ P.+19 log(g:+1)
+2p log(9+1)) P++ log(a:1))) (173)
6;
(2p:40h--1 log(()))))))))) - p.g log(4) - 1i log(4:) + 100411004111 log(1++1)
2(g++1 - 4:)2
- -P....1 log(9-1)) - p1+1 log(a+1))) (174)
Note that
- (p, log(a:) + Pc+1 log(1+1))) - -;)²=
- -p:q + log(g:) pig.41 log(g;) pitig log(1++) +2p log(ge1)) -ace)))>>ee4p0 Pe+19 log(g++1)
(175)
is almost, but not quite, a match for the numerator two of the logs are swapped (underlined). This can be corrected by adding
pioz_1 log(u:) + Pi+ of log(ge+1) - prgitz log(gi+1) Pi log(us),
(176)
(pi4h+1 1 log(u)) (pi+1 p( log((++)))
(177)
(p-- 2 pigitz) (log(g+11) log(g:)).
(178)
and rewrite the log terms as
ps log(()) + pi+1 log(q(+1)
+ Pi-10 peti+1 (log(g++1) log(g:)),
- -s
(179)
2
2(g:+1 9:)²
27

---

Published in Transactions on Machine Learning Research (04/2024)
which is quite elegant considering the beginning (there is the question of if it has geometric meaning, which could lead to a much more elegant derivation), with numerical stability.
Now work through all of the non-log terms:
3piRt+1
Diqi+1
+
24f
A(gi+1
9i)
+ 5,
piqltt
8; qi (qit1 - 9:)
-P.P.+1
6:
Piqi+1
d; Pi+1
8;
-p:4e+1
+ 6; 4((i+1 9:)
1
9
2q:(9:t 9i)
29: 9i)
*((9+1 - 9ii)
pige+1
Pigp+1
-ii+19++1
-3p:9-1
+ 8. 2/7 (i+11 - -Q,)
+ 6 2// a1t11
+ 6: 2011 (1t1 - -%:)
+ 6: A(gi+1 - -21)
- -qi)
-P...+1
-Pifi
-3p++19: qi
3ptqc+1
Pi+10i+1
+ 8 29 (9t+1 -91 9c:)
+ 8: A(gi+1 -ii)
+ A(9:-1 9t)
+ 5, 2(q - -9,)
+ 5, 2((4+1 9,)
(180)
3poR2+1
-2pi2+1
-2p:1
P:
6,
06;
+ 6, 2 (9i+1
+ 8:
29
20
9i)
24 (+++1 9:)
P
pig.+1
pi
-PI+1
-P:
+ 8,
+ 8: 29: (9t1 91+1 -9:) 9x))
+ 8:
+ +8; 29: -9t)
+ 8, 29: (96+1 9t
(9t+1
29:
(9t(
9i)
+ 6: 4lgi+1 - -qi) + -p++19-4
-pi9i+1
+ + 4(qi+1 - 9:)
pi9++1
+ 8:
-pI9t
4(1i+1 - 9())
4(ui+1 - -i))
-3p,4i+1
-3p++19; + 6,
6pidit1 +
2p;+
+ 6: A(di+1 -4:)
+
(181)
A(gi-1
91))
A(gi+1 -q:)
*(1i+1 95)
-P:4f+1
p:4f+1
(3p: + Pi+1 1 (p. + 3pp41)9:
8.
+ 6 2g7( (gi+1 9i)
+ +6:
+
(182)
20
24 (9i+1 - -gi)
4(1i+1 - -q:)
P.
p.
P.9.+1
(3p; + Pi+1 ]itt1 (pi + 1-ppp)1 )4:
6,
+ 6;
+ 6
(183)
2q (gi+1 - -96)
2
(9 - -%,
4(g.-1 - 9:)
-pi92+1
p292+1
(3ps + Pi+ 1)4411411111 - (pi + --p41)0
5.
+ +5:
+ +5:
(184)
24. (9:41 - -%:)
2q:(9:+1 9c)
4( 9:)
(3p, + Pi+1 1)1111 - (p) + (ppp+))
8
(185)
4(yb+1 (:))
Finally, putting it all together gets
- 8. //'' ((1 t)pp + 1p log((1 - -t)q: + tq:+1)dt= =
pi log(g:) Pi+1 log(g;-1) +0;- 2(gi+1 q:)²
Po+19 pigh+1
(log(1+1) - log():))+o, (3p, + +P441)9447 (p, + 3p )4;
8.
2
4(g++1 - q:)
(186)
where only the first term is needed if 4++1 = qi This works, as verified by comparison to numerical integration. Need to switch evaluation strategy when too close to singularity.
A.2.1 Behaviour around singularity
First demonstrate that it's mathematically, not computationally, stable. To ignore the second and third term when q++1 = qi it needs to be shown that
6, Put1 -.g.+11 (log(g+1) - log(a:)) + 8, (3pr + P++ 11+114++1 - (p: + 3pp+1) = 0. (187) 2(g:+1 9.)2
lim d:11>g;
4(g.41 %6)
28

---

Published in Transactions on Machine Learning Research (04/2024)
This is not the case if the two terms are considered independently - it is only the case if they are merged. To show this limit start by converting the log(.) of the second term back into an infinite sequence,
26: Pi+ o- p:0
Σ
1
9+t1
91
(188)
2 2(9 - 9:)²
n
9++1
1
ne (1,,5,,
91
8, P(+19: -Pi9h-1
[gi+1 9i)
(189)
(9s-1 =;)²2
12 (9i+1 + qz)
(1.33..5....
1++1 qt) n-2
8, (pt+ 19 - p(llt1)
Σ
(190)
n (g++1 + 9))
1,,,,,,,....
Note that the limit is trivially true for n = 3 onwards, so you only need to consider the n = 1 case, which can be dropped into the rest of the limit statement
10 - pight1) (9++1 9,) + 6, (3p; + Pi+1)9++1 (p: + 104:
(191)
(gi+1 + (;))
4(gi+1 - 4()
8: 1p - Apiif+1 + (3pi + Pi++)) (16+1 + qi)) - (pr + 3ptt1)q:(1i + 9i)
(192)
4(g+41 (;((1 + qi)
4pr4198 4proh+1 + +3pp + 1=+1)09411 + (2p: - 2pi+1)9s0++1 - (pr + 11-ppp1)) ))
6,
(193)
4(1i+1 - (:c41 + 4:)
8, (pi+1 p:)9241 2(p++1 p:)q:0++1 + (pi+1 p:)g?
(194)
4(1+t1 - (:(((c11 + +.)
6, (pit1 - ps) (9)(1-11 - 9:)²
(195)
4(g (1(+1 + t,)
8, (Di+1 - (1)+1 qi)
(196)
4(g++1 + 9:)
for which the required limit is simply true. The above is suggestive of an alternate way to write the equation out; consider the unused terms from the second equation,
(gi+1 - 9:))
1.(p: - DitHt1)
Σ
(197)
(q++1 + 9%)m
ne (3,5.7,...
(gi+1 - -q,)
8, (p pi
Σ
(198)
n2 n 2 (9:+1 + m+
n (1,3,5...)
Pi+19 P.
6 (9 + q2)2
Σ
9i-1 - qi
(199)
n 2
9--1 + qi
(1,,,,,)....
which appears to be the end of the line. But writing it all together gets
6: / ((1 - t)pp: + 1p log((1 - t) + tg:+1)dt=
pi log(g:) + P 1og(9--),
40, (pi+1 pi) qi) 4(gi+1 + q:) 48, (gi+1 + 9:)2
Pi+19h - -:gffu
Σ
1
9i+1 li
2
n + 2
gi+1 + 9i
(13,5
(200)
which is good fallback when in the q g++1 condition, as the terms of the infinite series move towards zero quickly in typical usage (see main text for quantification.
29

---

Published in Transactions on Machine Learning Research (04/2024)
A.2.2 Minor details
For completeness, results from The On-Line Encyclopedia of Integer Sequences are now proven and then combined to get the term needed by Equation 120 Firstly
n(n + 3)
(n - 1) ( + 4n(n + 1)
(201)
n(n + ()(n+)
4(n 1) +2
4 n2 +3) (ni 1)(n + 2)²
(202)
4= 4 3n² (n 1)(n" 2 4n + 4)
(203)
4 = n + 3n n 4n 4n + n² 4n +
(204)
4=4
(205)
Then 9
1
n - n
(206)
n(n+ + 1) n
1 n²
(n 1) (n +1
(207)
1= n² n² +1
(208)
1=1
(209)
Combine the above to gel
1
n(n+ +3)
n +3)
(n-1)n(n+)) +
+ + (o-1))(nr + +2)
(210)
n(n + 1)(n+ +2) 2)
4(n + 2)
4(n + 1)
4n
+ 3) (n -n- 1) ( 3), 4 = - 2 (,,p::1""|"|2"32333111"110""1"4"
= (n () 1) () ()
(m
(211)
+3),
(212)
(n² 2n htap://....aaaaa................
(213)
4 (n +3n n
5m
5m (n
mx) (n 5n 7m 3n
7
(214)
4 = nt + = 7
+4- 5n 6n - n + + 5 + 3n
(215)
4 = (1 - 11-111111 1)n + (3 3 5)m
+ (-1 6 7)n (-7 3)n +
(216)
4=4,
(217)
which can also be written
(n (n- 10) 2)
n (n + 3) + (n - 1)n(n + +2)
n(n + 1)(n + 3) 4(n + 2)
+
(218)
n(n + 1) 22
4n
4(n + 1)
Phetps:/ /otps:///eei.org/// A007531
Phetps: /oeis.org/ A002378
30

---

Published in Transactions on Machine Learning Research (04/2024)
B GPU code
The below Python code is for Jax! and has been developed with version 0.4.25 45. Validation, including of gradients, has been performed and may be found in the supplementary material alongside code for the demonstrations within the main text.
ajax.
def crossentropy (p. 9, delta):
"**Returns cress entropy betueen two regular orograma aligned evenly spaced bin centers, given as P and q, delta the apacing between bins. Will be approximate in some aituatione, as dodges around infinities and singularities to remain stable whatever you give it.
# First term requires what is effectively the relative area... Salved_ends jap. ones (p. ahape shape [0])
halved_ende balved_ends at [0] set (0.5) halved_ends halved_ends.ate [-11] set 50
# Assorted basic terms...
log_9 jup. log (jnp. maximum (q, 1e 32))
pdelta p[1:] p[:-1]
qdelta q[1:] -[]=-]] qaun q[:-1] =ql1:]
qaqr jnp. square (q)
top p[1:]*qaes [1-1] p[:-1]+qsqu [1:]
Inner term of infinite loop (used elsewhere), done in a atable way, # plus variant with extra qaus...
notzero qaum>1e-5 seum_safe
(qaun, 1e-5)
inner qdelta qaum_safe inner_ds2 qdelta / (jnp. (qsun_safe) squm__safe)
# Do the stable parts...
@chalved_enda P 1og_q). 0.25 (pdelta inner) sun
# Do the branches, with stability hacks for unstable #s Unstable sccurate qdelta high... abs abs_qdelta fab (jnp. signbit (qdelta) 0.5)
eign_qdelta
Shelta__qr_satt safe jap. maximum (jap. aquare (qdelta), 1e-10)
Beeelta_quun_eee siga_qdelta. jup maximum qdeltaa qeum, 1e-10)
safe ret_unstable top (0.5 (1og-q(1:) log-q(:-1)) / qdelta_sqr_stt safe adelta_qsun_sate
## Stable but only accurate when gdelta is lov...
ret_approx = top (1 / 3 jnp square (inner) (inne)) 5) inner_ds2
## Pick the right branch for each and sum in... jax lax. select
sex.ett (abs, (aaeeaaaassseeleeaeeeeeeeeeeeeee ge ret_unstable, ret_approx) sum ()
return delta+ret
grad_croesentropy jax. jit (jax grad (crossentropy, (0,1)))
10 "httpp::/wwwppytattt .rg/
htc.h//www.ooooogg///ee// google/jax
com
31


Reviews:
[{"review_id": "l1PSUAsijt", "reviewer": "Reviewer_u8K1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\nThe problem addressed is rather of rather general interest.\n\n**Weaknesses:**\n- The core of the contribution is the first part of section 3, which describes a derivation. The clarity of this section may be improved to better follow the derivation. I wrote some suggestions in the Requested Changes section.\n\nNote: The Appendix presents the full derivation, which, in the authors' own words, \"probably doubles as a statement about mathematical paranoia\" and \"it’s probably more valuable as an example of the difference between what goes into the body of a paper and what’s actually done, to inform and/or terrify future researchers\".\n- The method requires that the two piecewise linear densities have equal segments, i.e., equal number and location of change points. At one point in the paper, the authors add: \"though if the linear segments are not aligned, extra splits can be added\": do you mean that the union of the change points of the two piecewise linear densities should be considered?\n- Based on the related work section, I did not get a good understanding of what are the advantages/disadvantages of the present contribution with respect to pre-existing methods for tackling this problem. Could you please elaborate more on this?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper's main contribution is an analytical expression for the cross-entropy of two unidimensional probability density functions which are piecewise-linear. This may be relevant for machine learning tasks where differentiating such an objective is needed.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Given that this is a very maths heavy paper, it is important that equations are readable and the passage-by-passage derivation is easy to follow. \n\nReadability of the maths could be improved, for example by highlighting relevant terms in the equations---e.g., with the `underbrace` command, `\\underbrace{<term in the equation>}_{\\text{(i)}}`, to then refer to individual terms in the text ---- \"term (i) in equation ...\"\n\nMinor: Some of the text could be made a bit smoother, e.g., before eq. (5), \"combine (3) and (4) to obtain\"; \"Apply integration by parts and clean up\" could be rephrased as \"through integration by parts and rearrangement of the remaining terms, we get...\". I found this specific passage of (3)-(4) to (5) hard to follow.\n\nTypo: \"Computing with Equation 15 is stable when a safe distance from the singularity\" missing word?", "broader_impact_concerns": "No broader impact concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "npY874hF2I", "reviewer": "Reviewer_9bRM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThe detailed derivation appears to be a non-trivial result. The two expressions together constitute a way to perform the computation efficiently across many break points.\nThe paper provides a detailed look at numerical precision and improvements in precision (caption of Figure 1).\n\n# Weaknesses \n\nClear motivation is lacking in the introduction. The claim that this is useful to ML is not supported without concrete examples connecting the formulation with ones based on empirical samples encountered in ML problems. The related works describe the triangle kernel for kernel density, if this is a motivation example then how the piecewise linear function(s) would be defined in this case should be given. \n\nRelatedly, in the numerical evaluation, there is a lack of connection between actual ML application and the proposed approach. \n\nOverall, the paper’s organization should be changed with a clarified motivation up front, then the related work that connects this motivation to the proposal, and some examples on ML tasks. Otherwise, it seems more like a mathematics papers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a detailed derivation of two analytic expressions (one closed form but unstable at singularities and one requiring an infinite series that can be truncated as an approximation) for cross-entropy between two piecewise linear probability distribution functions. The choice of expression can be done per interval if the difference in the probability density function is too small causing a singularity. Some numerical experiments demonstrate the expressions behavior. Computationally, to get the same level of precision the truncated expression is much faster than Monte-Carlo integration.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Major:**\n\n1. The paper should be clear up front under what conditions would piecewise linear PDFs exist in a machine learning context. Clear statement of assumptions around how this would be useful should be introduction. Later, the paper should include specific examples of how a piecewise linear PDF is generated from data and when the target distribution can be approximated as piecewise linear. For example, the related works describe the triangle kernel for kernel density, if this is a motivation example then how the piecewise linear function(s) would be defined in this case should be given. \n\n2. Relatedly, the paper should include some comparisons of the cited examples of entropy estimation that have already been published.  I'd also like to see examples of comparing mixtures of Gaussians (with linear approximations) as this is task that motivates the use of other divergences. \n\n3. It is not clear to me what value of $N$ is used in practice to ensure precision at a specified tolerance. Is the bound used in the code to give specified tolerance?  \n\n**Minor:**\n\n1. In the equations, it is typical convention to have operands at beginning of line rather than end. Also full stops (e.g., equation 10) or commas are missing in some cases and spurious equal sign (in Appendix). \n\n*Introduction*\n\n2. I'd prefer to stick with machine learning in the first sentence rather than calling everything AI.\n\n3. In second sentence, \"It\" is ambiguous from context.\n\n4. Clearly cross-entropy is between distributions, so it needs to be clarified to the reader when it is applicable to the range considered in continuous regression. I guess this under a parametric assumption of the distribution around observed but it must be clarified. \n\n5. Before equation 1, the description of the PDFs with breakpoints should be given. \n\n6. The mix of English and symbols like \"Entropy ($=H(P,P)$)\" is hard to parse, this should be revised.\n\n*Related work*\n\n7. Does \"regular histogram\" have uniformly sized bins?\n\n8. The use of \"excessive number of change points\"  isn't the number of breakpoints relative to the number of data points in the kernel density estimate? So averaging across histograms won't necessarily create more breakpoints.\n\n9. \"course\"->\"coarse\"\n\n10. \"unknown distributions\" -> \"unknown distribution's\"\n\n11. Regarding a neural networks with ReLU functions, even if the input is 1D how would this property be useful for input and output pairs used in machine learning training?\n\n*Derivation*\n\n12. \"cross entropy\" not hyphenated. \n\n13. The steps leading to Equation 7 should be clarified with insights from Equation 38 from appendix. Otherwise the indexing going from odd positive integers to all positive integers isn't clear.\n \n14. The \"argument has been prefilled\" is not clear. Is the argument $i$?\n\n15. \"floating point maths\" -> \"floating point operations\"\n\n*Numerical validation*\n\n16. The scale of Figure 3 is such that the precision is not demonstrated. Perhaps the error can be plotted separately.\n\n*Appendix*\n\n17. Some of the text in the supplement is unprofessional: \"terrify\", \"horrifying\", \"poking up like an enormous boil\", and \"so it’s time to scream into the void again\". The paper should avoid the attempts at humor—even the \"(yay!)\" is unnecessary. The appendix section title \"Hulk smash fraction of minor irritation\" is indecipherable cultural jargon.", "broader_impact_concerns": "no concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AfSw5pvHzQ", "reviewer": "Reviewer_aUx1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is clearly written, the objective is clear.\n* I verified that the main result is correct.\n\nWeaknesses:\n* The authors argue that a closed-form formula for this cross-entropy is needed specifically to employ cross-entropy as a loss function. From reading the article, it is not clear to me which application would need to compute the cross-entropy between piecewise linear PDFs, as opposed to more standard piecewise constant PDFs (i.e. histograms) for which evaluation of the integral would be straight-forward.\n* The derivation of the formula for the integral in this paper is overly complicated.\n* I'm not entirely convinced that the topic of this paper is of particular interest to the TMLR readership; this is in part related to the first point above, but it is also the case that the main result of this paper can also be obtained in a few lines of Mathematica code (or derived by hand if one is willing to go through a little bit of algebra). But I'm willing to accept that having a searchable resource available which immediately gives the result of the paper could be of value in some cases.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper derives a formula for the cross-entropy of two 1-dimensional, piecewise linear probability distribution functions p(t) and q(t); ultimately, this simply amounts to computing the integral\n$$\n\\int_0^1 (p_0(1-t) + p_1 t) \\log (q_0(1-t) + q_1 t) dt,\n$$\nfor coefficients $p_0,p_1,q_0,q_1>0$. The authors present a detailed derivation of a closed-form formula. They further discuss potential numerical issues with this formula, which arise because the formula contains near-vanishing denominators in the limit $q_0\\to q_1$ which formally cancel, but do not cancel numerically due to round-off errors. They also discuss an implementation which avoids numerical instability, and present examples which numerically verify their derivation by comparison to quadrature/MC integration.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to ask the authors to \n* Expand a bit on why and where this formula finds applications. \n* Provide a shorter derivation of the formula. The infinite series expansion in equation (4) is unnecessary. Instead, one can use a simple change of variables $q = q_0 + (q_1-q_0)t = q_0 + \\Delta q t$ to write the integral in the form,\n$$\n\\int_0^1 (p_0(1-t) + p_1 t) \\log (q_0(1-t) + q_1 t) dt = \\int_{q_0}^{q_1} \\left(p_0 + \\frac{\\Delta p}{\\Delta q} (q-q_0)\\right) \\log(q) \\frac{dq}{\\Delta q}\n$$\nand observe that this is simply a sum of two terms: The first is a constant multiple of \n$\\int_{q_0}^{q_1} \\log(q) \\, dq $, the second is a multiple of $\\int_{q_0}^{q_1} q\\log(q) \\, dq $. Both of these integrals have closed-form formulae.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two of the reviewers advocated for accepting the paper, with one leaning reject.  To quote the more negative reviewer:

"The paper would be much more acceptable with concrete example of using the cross-entropy with piecewise linear assumptions for optimizing a data representation or processing function. For instance, simply moving data points to match a target distribution using gradient flows, which would illustrate the stability of the gradients.

Another reviewer suggestions along with my limited analysis points to some of the stability being resolved by careful pairings of terms different then the paper uses. Thus, the highlighted numerical stability may be of less concern than another one that may have actually better tolerance. This casts doubt on the result which requires series calculation if the bins are two small such that the heights are constant."

This echoes to some degree the other reviewers as well, who all felt that the connection to the ML community could be made stronger.  This does not entail any major change to the paper, but what would help is improve the motivation and to use a concrete example, as suggested above.

The other requested change, which was also noted by multiple reviewers, is to ensure that the language be cleaned up to be more "professional" and less distracting.

 There was some seemingly unresolved discussion on stability, which the authors should examine more closely.  It's unclear to me if this requires any additional change to the paper but please have another look.
Decision: Accept (poster)