Floating-Point Neural Networks Can Represent Almost All Floating-Point Functions
Authors: Geonho Hwang
Yeachan Park
Wonyeol Lee
Sejun Park
Venue: ICML 2025 poster (2025)
Keywords: Universal approximation
Floating-Point Arithmetic
Primary area: deep_learning->theory

Abstract: Existing works on the expressive power of neural networks typically assume real-valued parameters and exact mathematical operations during the evaluation of networks. However, neural networks run on actual computers can take parameters only from a small subset of the reals and perform inexact mathematical operations with round-off errors and overflows. In this work, we study the expressive power of floating-point neural networks, i.e., networks with floating-point parameters and operations. We first observe that for floating-point neural networks to represent all functions from floating-point vectors to floating-point vectors, it is necessary to distinguish different inputs: the first layer of a network should be able to generate different outputs for different inputs. We also prove that such distinguishability is sufficient, along with mild conditions on activation functions. Our result shows that with practical activation functions, floating-point neural networks can represent floating-point functions from a wide domain to all finite or infinite floats. For example, the domain is all finite floats for Sigmoid and tanh, and it is all finite floats of magnitude less than 1/8 times the largest float for ReLU, ELU, SeLU, GELU, Swish, Mish, and sin.

Floating-Point Neural Networks Can Represent Almost All Floating-Point Functions
Geonho Hwang' Yeachan Park
Wonyeol Lee Sejun Park
Abstract
neural networks can approximate a large class of target fune- tions. Classical results show that two-layer cully-coneected networks using a non-polynomial activation function can approximate any continuous function on a compact domain within an arbitrary accuracy (Cybenko, 1989; Hornik et al., 1989; Pinkus, 1999; Leshno al., 1993). Such results have been extended width-bounded setups (Lu et al., 2017; Kidger & Lyons, 2020; Park al., 2021) and more practical networks (Zhou, 2020; Tabuada & Gharesifard, 2021; Yun et al., 2020; Ramanujan et al., 2020; Yuan & Agaian, 2023),
Existing works on the expressive power of neural networks typically assume real-valued parameters and exact mathematical operations during the eval- uation of networks. However, neural networks run on actual computers can take parameters only from small subset of the reals and perform inex- act mathematical operations with round-off errors and overflows. In this work, we study the expres- sive power of floating- point neural networks, i.e., networks with floating-point parameters and op- erations. We first observe that for floating-poit neural networks to represent all functions from floating-poing point vectors to floating point vectors, it is necessary to distinguish different inputs: the first layer of a network should be able to gen- erate different outputs for different inputs. We also prove that such distinguishability is sufficient, along with mild conditions activation func- tions. Our result shows that with practical acti- vation functions, floating-poin neural networks can represent noating-point functions from a wide domain to all finite or infinite floats. For exam- ple, the domain is all finite floats for Sigmoid and tanh, and it all finite floats of magnitude less than 1/8 times the largest float for ReLU, ELU, SeLU, GELU, Swish, Mish and sin.
These universal approximation results assume real-valued network parameters and/or exact mathematical operations (e.g., addition and multiplication) during the evaluation of networks. In practice, however, neural networks exe- cuted on actual computers in which network parameters can take values only from finite subset of the reals and math- ematical operations can be inexact due to round-off errors. This presents a clear gap between theoretical assumptions and practical environments for neural networks, which be- comes more significant under low-precision setups: network parameters can take only tiny number of values, and math- ematical operations can have huge round-off errors.
Several works have studied the expressive power of neural networks that take parameter values from fixed finite set. For example, Ding et al. (2019) investigate the universal approximation of ReLU networks with quantized parame- ters, and show that such "quantized" ReLU networks with at least distinct parameters can approximate Sobolev functions W7 (n 2 1). Similarly, Gonon et al. (2023) ex- amine the impact of quantization on the approximation abil- ity of ReLU networks, and show that a uniformly quantized network can approximate a network with real parameters. However, all these works assume exact mathematical oper- ations, which limits our understanding of neural networks under finite-precisiee operations with round-off errors.
1. Introduction
Deep neural networks have achieved remarkable success in various fields of science and engineering (LeCun et al., 2015). One of the theoretical foundations of neural networks is the universal approximation theorem, which states that
'Department of Mathematical Sciences, GIST DDepartment of Mathematics and Statistics, Sejong University 'Department of Computer Science and Engineering, POSTECH 'Department of Artificial Intelligence, Korea University. Correspondence to: Sejun Park essjjun.park000 gmail.com>.
Some recent studies have investigated more realistic setups, where parameters can take values from finite set and mathe- matical operations can be inexact. For example, Hwang et al. (2024) study the expressive power of neural networks under fixed-point arithmetic, and show that such "fixed-point" net- works with finite-rrecision weights and infinite- infinite-precision biases can approximate any continuous function when they
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Floating-Point Neural Networks Can Represent Almost All Floating-Poin Functions
use popular activation functions such as Sigmoid, ReLU, E.LUU SoftPlus, SIL,U, Mish, and GELU. Likewise, Park et al. (2024) analyze the expressive power of networks under peaating-point arithmetic. They show that such "floating- point" networks can represent all floating-poim functions over a unit cube when they use ReLU or Step (i.e., the step function) as activation functions.
that floating-point neural networks can represent a large class of floating-poin functions for various activation func- tions (e.g.. Identity, Sigmoid, tanh, ReLU, ELU, SeLU, GELU, Swish), under various floating- point formats from low-precision ones (e.g.. float8, 1 to high-precision ones (e.g., float32, float64) (Corollaries 3.8 and 3.11).
We note that a concurrent work by Hwang et al. (2025) studies universal approximation under floating-poin poatinngpoiittoo arith- metic. Their work considers interval universal approxima- tion, which generalizes the classical pointwise approxima- tion. However, a special case of their result for pointwise approximation is subsumed by our result (Theorem 3.4): their condition is strictly stronger, and their network con- struction requires more depth than ours. In addition, they do not study the necessity of their condition unlike our work.
Despite all these advances, no existing work has studied the expressive power of neural networks under the following arguably most practical setup:
Parameters are represented as floating- point numbers. Operations are performed using floating- point arithmetic.
A broad class of activation functions supported.
For instance, Hwang et al. (2024) considers fixed-point arith- metic, not flotting-poin arithmetic; and Park et al. (2024) considers only two activation functions (ReLU and Step), leaving out many widely-used activation functions such as GELU (Hendrycks & Gimpel, 2016), SeLU (Klambauer et al., 2017). Swish (Ramachandran et al., 2017), Mish (Bochkovskiy et al., 2020), and sin (Sitzmann et al., 2020). We remark that the proof technique of Park et al. (2024) relies heavily on the piecewise linearity of activation func-
2. Preliminaries
In this section, we introduce the basic notations and con- cepts used throughout the paper, including floating- point arithmetic and floating-point neural networks.
We begin with notations. We use N, Z, and I denote the sets of natural numbers, integers, and real numbers, re- spectively. For a, b E E. we define [a, b] == {x E R a S x S b} and (a.b) = fo E R a < x < b}; we define (a, b) and (a, b) analogously. For S C R, we de- fine [a, bls := [a, n, with (a, b). [a, b)s. and (a, b|s defined analogously. For n E N, we write [n] = [1, For d E N, set S. and E Sd, we define 2, as the i-th coordinate of x. In this paper, all fractional numbers with a radix point are assumed to be in binary representation: e.g.. 1.101 = 29 + 2- + 2-3 13/8.
tions, so extending their proof to non non-piecewise-Iie linear acti- vation functions is highly non-trivial, if possible at all. To summarize, the following question has remained open:
What is the expressivity of ffoattig-ppoin neural net- works using general practical activation functions?
1.1. Contribution
In this work, we investigate the class of floating point func- tions that can be represented by feed-forward neural net- works using general activation functions under floating- point arithmetic. To this end, we first observe that to rep- resent all functions from some domain X' of pooating-poin numbers to Dooting-poiin numbers, it is necessary for net- works to distinguish any pair of distinct inputs in the domain in the first layer (Lemma 3.2). Namely, for any x' e X with x + x', there should exist a floating-point affine trans- formation such that (((x)) * a(o(z')), where o de- notes the activation function. Using this property, we show that networks using the correctly rounded version of the cos activation function cannot represent some doating-poin functions (Lemma 3.3): e.g., some functions over [-25, 2$] for the 16-bit half- precisreciioon moating-point format.
2.1. Floating- PPint Arithmetic
Floating-pou point numbers. For p,g E N, we define IPp.q as the set of finite floating-poing numbers:
F Pp.g- = {s x (1.m, ... mmp) x 2 sE mi....,m, E {0, 1), E [emins emax] ]
EE
u ufs x (0.m ... m,)) x 25min
8E 11, m ..., mp E {(.11)}
(1)
where Emin and emax are defined as Cmin == -201 + 2 and Cmax == =2-11 1. Here, 8, m_ m1...m,, and E are called the vign, mantissa, and exponent of a floating-poin number, respectively. Note that p + q + 1 bits suffice to represent all numbers in Fp.g p bits and q bits for representing the mantissa and exponent, respectively, and one additional bit for the sign. For simplicity, we omit the subscript and write F for Fp.g when p and q are clear from the context.
We then show that such distinguishability sufficient for networks to represent all floating-por -oint functions under mild conditions activation functions (Theorem 3.4): can output zero and two moderate values (e.g.. values around 1) (Condition 1). We also provide easily verifiable conditions on activation functions that ensure distinguishability (Lem- mas 3.6, 3.7, and 3.). Using these conditions, we show
We use co and - oo to denote positive and negative infinities, and assume the usual order: -00 < x < oC for any x E R.
2

---

Floating-Point Neural Networks Can Represent Almost All Floating-Poinl Functions
Format
(p, ④)
where lim -- p(x) and lim p(r). Here, the existence of l,r includes the case 1 E (-00, 00).
16-bit half precision (IEEE, 2019)
(10,5) 5)
32-bit single precision (IEEE, 2019)
(23, 8)
For x, V E F, we define the floating-point operations (D, O and @ as I Gy = x + yl. x B = [x yl. and x @ y == I x yj. Note that the addition and multiplication are not associative: egg., (ar @ y) @ = + z @ (y + z) in general. Therefore, we must be very careful about the ordering of the operations. To concisely represent the addition of floating- point numbers T1,... Zn. we use ① as follows:
64-bit double precision (EEEE, 2019)
(52,
8-bit E5M2 (Micikevicius et al., 2022)
(2,5)
8-bit E4M3 (Micikevicius et al., 2022)
(3, 4)
6 (Google; Abadi et al., 2016)
(7,8)
Table I: List of frequently-used floating caait formats.
We use NaN to denote not-a-number, which can be pro- duced, e.g., when oo is added to -00 under floating-point arithmetic. We assume any operation including NaN pro- duces NaN; this does hold for Koating-point addition, sub- traction, and multiplication. We use F to denote the set of all floating-poie numbers (or floats) F == ( oc, NaN). We note that p.q can also be represented using ++ bits since we are not using the whole 2 representations for the exponent of floats in IF p.q. For x E IF, at and x- denote the smallest and largest floats greater than and less than Ir re- spectively. For x E IF, we use &₂ == max [log, |x]] ]. The smallest and largest finite positive floats are denoted by
I == I1
4
In.
where addition is performed from left to right. Likewise, we use + and + in a left-associative manner. For example,
+ ... 2 I1.m 2 22,1 @ ..... .... 22. Ci 21,1
4 Fn,1 ① Fn,m.
where addition performed from left to right. These defini- tions ensure that the addition is always performed sequen- tially, starting from the first element. When performing summation on the product of finite sets of integers, the sum- mation is always carried out in lexicographic order. E.g.
w := 2mmm and Ω == (2 - P-P) x 26max, respectively.
The IEEE-754 standard (IEEE, 2019) defines (p. q) for widely used floating-poin -pont formats: e.g.. (10,5) for the 16-bit half precision (6) (23, 8) for the 32-bit single precision (float32), and (52, 11) for the 64-bit double preci- sion (float64). In this paper, we assume that (p, q) satisfies 2 S p < 2 - 3. Many popular floating-poin formats satisfy this condition, as illustrated in Table 1.
+ (x = (1 () e (1 + ".NE (3.4)
① (2 2 (2@4).
For X' C Fd, we use 1x Fd + F to denote the indicator function of X: 1x (x) is one if z E X' and zero otherwise. If = {mo] is a singleton set, we use 1zo to denote 1 xxx
Floating-point operations. The rounding function [: RU (0000, oo, NaN] + is defined as
2.2. Ploating-Poin Neural Networks
To define floating- point neural networks, we first define a floating-point affine transformation. For di, d2 E N, let
arg min Der x vl
if [f < 0 < if = (1+2-=))) if (0(1+2--1)
8
wy = (Wi,1. Wt,d,) E Fd, and b, E F for all E [dg]. Then, for (wi...., , W'g bi...., "ab...,,,,,),, bdz), we define the (flotting-poin) affine transformation aff, Fdi, - Fdz
[x]s
00 NaN
if = NaN.
There can be two floats equidistant from real number. In such a case, we break the tie using the tie-to-even rule: [xrr is defined by the (unique) float whose last mantissa bit m, (see Eq. (1)) is zero. If F is clear from the text, we omit the subscript F in [-]r-
di
aff , (2......................... Id.):
A
(wis & 2;))
+ bi
for all i E [dz]. For a Roating-point activation function F + F, we will slightly abuse notation so that o also performs a vectorized operation: for d E N and Fd.
For p IR R. we define the correctly rounded function
Tpl F = F of p follows:
a(x) = (a(x)), (xa))
[p(x)]
x E F if x 0o ^ lo p(z). if 0o ^ 3im p(r), otherwise,
[
(x) =
Based on the above definitions, we define a a neural network as an alternating composition of floating-por point affine trans- formations and activation functions. Concretely, let E N,
[r]
NaN
3

---

Floating-Point Neural Networks Can Represent Almost All Floating-Poinl Functions
do....,,d de N, and let a, Fdi- + Frl. be affine trans- formations for all [7]. Then, for I (T1,..., It), function Nz= Ipd + [F defined by
Definition 3.1 (Distinguishability)) Let F F, d E N. X c FF4, and y C F. We say that "X' is a-distinguishable with range )" if for every x' E X with & x', there exists an affine transformation Φ F + F such that
N2 = aff ................................ ... aff o o a
(2)
a (((()) * (b(x')) and a(o(X)) c V.
(3)
is called a (floating-poinn number of layers of a network N as the number of affine transformations in N: e.g., N/ has layers. For techni- cal purposes, we also define a neural network ending with activation functions: for T' (J ................................ 1), the function
neural network. We define the
To represent all functions from X' c Fd to F one can observe that X' should be o-distinguishable with range FF U {-o0, jo} as stated in the following lemma. See Appendix D.1 for the formal proof.
Mr aff o a a aff.,'
Lemma 3.2. Let d E N. X C Fd, andd a : - not o-distinguishuble with range F (- oo), then there exists f : X' - ( 0o} such that J + g on X for all a networks g-
is
which omits the last affine transformation, is called neural network ending with activation functions. Here, M2 has
I 1-1 1 layers as it contains - affine transformations.
In this paper, we investigate the representation power of floating- poatting-pointan neural networks. For d1, d2 N, X C Fd, and / : X + (F (-00, )))2 we say that "f can be represented by (floating-pon pointt) neural network" if there exists floating-poin point neural network N such that
We note that if a one-dimensional subset X C F is a- distinguishable with some range, then for any d N, X is also o-distinguishable with the same range.
Using Lemma 3.2, show in Lemma 3.3 that networks using the [cos] activation function cannot represent all func- tions from [-21(0+7/21,, 21(++7/211] IF to F U {-20, oo} (e.g.. = 10 for float16 and = 23 for float32). The proof of Lemma 3.3 is in Appendix D.2.
f(x) = (xx for all r E X.
Note that f can output oo and -00.
Lemma 3.3. Any f [-216++7/2), 2 + F with f(0) + f(w) cannot be represented by a [cos] network.
3. Main Results
In this section, we formally present our main results. We first introduce necessary condition activation functions for floating-poin networks to represent floating-point func- tions (Section 3.1). We then introduce a sufficient condition and compare it with our necessary condition (Section 3.2). We lastly provide easily verifiable conditions on activation functions that imply the sufficient condition, and show that networks using practical activation functions can represent all floating- -pintt functions on wide domain (Section 3.3).
Nevertheless, for most practical activation functions, floating-poin neural networks can represent all functions over a wide domain (-2 -2 26max - )). We will see this in the next subsection.
3.2. Sufficient Condition on Activation Functions
While the distinguishability of X c Td with range {-20, oo} is necessary for representing all floating- point functions from X' to F U (-oc, oo), the distinguishability is also sufficient under mild assumptions on its range and the activation function (Theorem 3.4).
3.1. Necessary Condition on Activation Functions
Given Moating-point activation function a F - F and domain X' Fdd we are interested in identifying whether networks can represent all functions from X' to F F (- oc). A natural observation is that such universal representation is impossible if there exist v, x. E X such that ((())) = or (ø(x)) for all floating point affine trans- formations Φb. That is, for any a network, the outputs of the first layer and x' are identical. By the definition of neural networks (Eq. (2)), this implies that the final out- puts of the network at x and x' must also be identical for all networks; and thus, networks cannot represent any function f : X + F U )o} such that f(x) # f(x').
We first introduce the following condition to explain our sufficient condition.
Condition 1. For an activation function a : F - F. there
exist Co. C1, C2 E F such that [C,].ICa - C3l S 20mas for
all 0 < i,j S 2, and
ar (Co) = 0, 2 S ||0(Ci) < 4. Io(C2) > (2 2)+
Condition I requires the existence of three points Co, C1, C2 for an activation function a : FF + F. This condition can be easily satisfied for the correctly rounded versions of popular activation functions. For example, (C,) = can be satisfied for Co = 0 (e.g., ReLU, GELU, sin,
To formally describe the above idea, we define the distin- guishability of an input domain.
4

---

Floating-Poinl Neural Networks Can Represent Almost All Floating- Functions
Lanh) for Co of large magnitude such as -2 (e-g., Sigmoid). Furthermore, the conditions on C1 C2 can be easily satisfied by choosing some C₁ C2 so that (C = C2) E 1e.pp-))/////
(2,z(1) 22,x' (or "aee"" and dz E af e Wa (or (Maa.' "e, }). If we can find such an affine transforma- tion and a separating point for all distinct pairs in the domain, then the domain is distinguishable since a ( (x)) +
a (b (z')) (see Eq. (3)).
Under Condition 1, we show that the distinguishability of X with range |-2 2mmax] suffices for representing all functions from X to (FU (-00, )o
Based on the above idea, we propose sufficient condition for distinguishability in the following lemma. The proof of Lemma 3.6 is in Appendix D.3.
Theorem 3.4. Let a= : F + F, din damt E N, X C Fdo,, and f X + (I ( 003) mm.) Suppose that sat- isfies Condition 1 and X is o-distinguishable with range [-25max 22 Jr. Then, there exists four-layer a network 9 such that f = on X.
Lemma 3.6. Let a : F - F,, n E N. and mh Th E F be separating points of a with |m S... S |Ma]. Suppose that integers e1, e2 E [emin + 1, Cmax] satisfy
[Amin: E2|z C
Ulen. Ism. -1, Cy, Cmax 212 i=1
There are two notable differences between our necessary condition (Lemma 3.2) and sufficient condition (Theo- rem 3.4): Theorem 3.4 requires Condition 1, and considers a smaller range | 22mmax _2mmax] lg for distinguishability. First, we use Condition 1 for networks generate all possible val- ues in P -o00 oo} } (see Lemma 4.2). If an activation func- tion a can only output too large values (e.g.. [26max, ool) or too small values (e.g., in [-w,w] then a a network may not be able to generate some values in F U {-00, }. Second, the smaller range is due to technical reasons in our proof, which is used avoid overflow during the evaluation of net- works. If network can generate values with large absolute values (e.g., close to ②2) while distinguishing inputs, then multiplying/adding constants to those values may incur over- flow and the network may output NaN. However, if a has a well-bounded range (i.e., (I U (-oo, )) C [2-fmxx 25mmx) }) as in the correctly rounded versions of Sigmoid and tanh, then this range condition is automatically satisfied.
Then, (-202+1 20 )) is o d-disinnnuisabble with range
Res,es == =0(1(22ntertt @ |mml*)), (",),,21tee 2er-eat1 + Innlt)))
In Lemma 3.6, one can observe that having two separating points- one with small magnitude and one with moderate- to-large magnitude-suffiee to distinguish a large domain. In particular, if a has two separating points 1/1, 7/ with
|m| 2fmin and 4 S |122l < 2 PP-1
then (-2e2t1, 222+1)z is o-distinguishable with range Ro for all e2 E [Emin + 1, fmax]z. By choosing the largest ež such that Ro.e; C [-25mmx 25mmox] and by using Theorem 3.4, we can show that a networks can represent all functions from (-202+1, 2 )) to F U (-00, For example, ej = mmax - when o = ReL.U.
We note that having a separating point with small absolute value (e.g.. = 25min) critical for distinguishing a large domain using Lemma 3.6, while avoiding overflow. This is because, to distinguish two small numbers (e.g., 0 and w), we find w.b E F and a separating point 17 E F such that w & 0 + by (or 17*) and w @ w + b E (n (or {n n }). If is large in magnitude, then w and b must also be large. As a result, overflow may occur when the domain contains large numbers. We also note that a separating point with moderate-to-large absolute value (e.g.. 2 1) is necessary to distinguish large domain using Lemma 3.6.
3.3. Sufficient Conditions for Distinguishability
In this subsection, we provide casily verifiable conditions on floating-poii soating-poitttaoo activation functions (Lemmas 3.6 and 3.7) and real activation functions (Lemma 3.10) that imply distin- guishability. Using these conditions, we show that networks with popular activation functions are universal function rep- resenters (Corollaries 3.8 and 3.11).
To describe our conditions, we first define the separating points of a floating- point activation function a.
Definition 3.5 (Separating Point). For a : F - F, we say that 7 E F is a "separating point of o if
Using Lemma 3.6, we provide a sufficient condition for the distinguishability of floating footing-poiint point activation functions that have finite values at all inputs including to,, such as [Sigmoid]. The proof of Lemma 3.7
a ) $ (())),((1)))
}}} or a(n)) & ((()))((1")) }}-
Appendix D.4.
Here, 7/ (or v)+) denotes the largest (or smallest) float that is smaller (or larger) than 17 (see Section 1...
Lemma 3.7. Let a F - R such that a (F {-o0, 20)) C [-2emax _2mmaxx Jz. Suppose that a has two separating points
bm < 2 and [m2) 2 4. Then, F is o-distinguishable with range 1- 20max]
We design our sufficient conditions using separating points. Specifically, for each distinct pair x') of inputs in a domain X. we aim to find a floating-pont point affine transfor- mation dr and a separating point "e of a such that
Lemma 3.7 states that if (I (o0, -oo]]] is bounded by 26max and there exist two separating points of moderate size,
5

---

Ploating-Point Neural Networks Can Represent Almost All Floating-Poing Functions
tween L and Lyæ, and p' is lower bounded by L1 for all inputs between 0 and 2", then the domain (-2 2 )r is [p] -ol-iistinnuiiha with range [-2 225max Jr. where e' = Cmax + - 2. By Theorem 3.4, this implies that [e] networks can represent all functions from (-2e", 22') "rr to F U {-xc, oo). Since Lemma 3.10 covers various practical activation functions p with p(0) = O, the following corol- lary holds for the correctly rounded version of such p. The proof of Corollary 3.11 is in Appendix D.7.
1 S p(x)) S L
p(x;)
('(x)= 1
-- L1 S L2 S 2D
L
Corollary 3.11. Let a be one of [Identity]. [RRLU], [ELU], [SeLUJ, [GELUJ, [Swish], [Mish]] and [sin]. Then, for any d E N, a networks can represent all functions from (-2 -2 2 1amm--21 2 to F (-00,
0
2
Figure 1: Visualization of the conditions in Lemma 3.10.
4. Proof of Theorem 3.4
then F is o-distinguishabee with range [-26max 2 Jr. By Theorem 3.4, this implies that a networks can represent all functions from Idd to [ oo), leading the following corollary for [Sigmoid] and [tanh].
We prove Theorem 3.4 by explicitly constructing the tar- get four-layer network g under Condition 1. Specifically, we construct g as linear combination of indicator func- tions of points (Lemma 4.1). Given in the domain, our construction of the indicator function of & consists of three parts. In the first part, we create an injective layer consist- ing of an affine transformation followed by an activation function based on the distinguishability; Due to the injec- tivity, all inputs in the domain have distinct output vectors after passing this part. Here, we use y (y1..... : yk) to denote the output of the first part of In the second part, we construct 2n binary step functions f1,1, Jn.2....-JJ1,,
Corollary 3.8. Let be one of [Sigmoid] and [tanh]]. Then, for any d E N, networks can represent all functions from Ed to F U {-00,00)
We now consider a more realistic scenario: a is the correctly rounded version of a real activation function p. For this case, we first introduce a sufficient condition for having a separat- ing point. The proof of Lemma 3.9 is in Appendix D.5.
Lemma 3.9. Let P : RR + R. a. E F, and e E [emin= Emax]z lz. with [[]] (4)] S 2c+1 (1 P) Suppose that there
where ft,1 = (C1 Co) 2 yil + Co and f-2(-1)= (G1 - Go)1 S y/] + Cu where Co, C1 are from Condi- tion 1. Namely, all fa, (or f1.2) is C1 if and only if the input to the network is (or -2). To implement such fi., we exploit the round-off error in floating-point operations and show that can increase the gap between arbitrary two numbers by sequentially adding the same floats to those two numbers (see Lemma 4.6). In the third part, we construct a function that outputs C₁ or C2 if all ft,j are C1 and outputs Co otherwise. We then apply the activation function after this function so that our indicator function is either zero (i.e., a (C))) if the input is some non-zero value (i.e., (Ci) o (C2)) otherwise.
exists L > 0 such that L(b a) 2 and either
[p] (a) 2 0 and p¹ (x) 2 L for all E [a., b). or
[p] (a) < 0 and p'(x)= S for all x E [a, b).
Then, there exists a separating point r) E [a, blg of [p].
Lemma 3.9 states that if p is sufficiently increasing or de- creasing on a long enough interval, then that interval con- tains separating point of [p]. Using this lemma, we now real activation function o that guarantees the distinguishability of its correctly rounded version [p]. The proof of Lemma 3.10 is in Appendix D.6.
present sufficient condition
Lemma 3.10. Let P. R- R with ((x) = -r. Suppose that there exist أ E
We now formally prove Theorem 3.4.
C, o o <). Ia, T.2 > 0, and e E Z such that the following hold:
Lemma 4.1. Let F F,, d E N, and X Fd, Suppose that sutisfies Condition and X is o-distinguishable with range 1-25max 25mox] Jz. Then, for any E ld and c E (C1,C2h there exists a three-layer a network f X - F ending with the activation function such that
L S <(xx) S L2 all x E (0,22).
p'(x) 2 for all x: E (0, 25)
- - S < 1 + p. where 4 [log2 (L12( and
12 = (2[a)
f(x) - 0 (c)
Then, (-2 2 )z is [ijiiistinuuishable with range [-26mus 2mmux where e' Cmax + min(e-2) 11)
We present the proof of Lemma 4.1 in Section 4.2. Lemma 4.1 states that if an activation function a satis- fies Condition I and the domain X is o-distinguishable,
Lemma 3.10 roughly states that for some moderate-size T, Ta > 0, if a real activation function p is bounded be-
6

---

Floating-Point Neural Networks Can Represent Almost All Floating-Poiie Point Functions
then we can construct an indicator function of F with coefficient of (CC) a (C2). With Condition 1, this implies that we can construct indicator functions with co- efficients (CC) [2tmi 5/4] and (CC2) > (2-
fi o f2(x) = fz) @ 21 @ ... + Zz
= (x) @ (wi (e ))
@ (w, 2 (c,))
The following lemma states that applying affine transforma- tions to these coefficients suffice for generating all floats in PU{- (-00,,,))) x.).
111
1/2
1/3
1/
f2(1
(c1)
(
a (c
Lemma 4.2. Suppose that a : IF F satisfies Condi- tion 1. Then, for any E F {-oo, )0), there exist N, wi..... Wn F, and 21...., Zn E {C1, C2) such that
x (w) 2 (21)) @
(in (zn)).
To prove Lemma 4.2, we show that for each a E IF, there exist and E {C1, C2] such that x + (w (2)) = at Here, we note that w 8 (2) may not be exactly equal to x r due to the round-off error. For a more formal argument, see the full proof in Appendix D.8.
Figure 2: A composition of a network f2 ending with an activation funetion (blue) and a sequential addition fi (red). Each 2 in the sequential addition can be represented by wy @ (c)) and fi f2 can be represented by network.
We now prove Theorem 3.4 using Lemmas 4.1 and 4.2. Without loss of generality, we assume d2 = 1, ie.. the target function f is scalar-valued. To represent f' X + (F U {-oc, )o with d2 > 1, we can construct d2 networks that represent coordinatewise functions + fz), and concatenate them.
Definition 4.3 (Sequential addition). Let a - FF and M, == (w @ a(e) w. e E F with w & (c) E F). We say function f F - F is "sequential addition using o" if f(F) F and there exist n E N and , 2n Eg such that for each E I.
For any x follows:
X + F FU 10,000, we can represent f as
f(x) = Σ fz)
f(x) = x e 21 ①
1 2n.
(4)
yEX
Let = f(v). Then, by Condition and Lemma 4.2, for any E FU[- )o}, there exist n, E N, 2e,1,...,, Ze,n. E
We often drop a and use Σ to denote Eg if it is clear from the context.
{C1, C2), and we.l,..., We, FF such that
We often compose sequential addition with a network ending with an activation function: for a sequential addition fi using a and a network f2 F + F. fi o fo is also a a network (see Fig. 2 for an illustration). Here, we use additional activation functions and biases to represent z; in Eq. (4). However, since the floating peattng--poitt addition not associative, the sequential addition in Eq. (4) can be different from adding single float (e.g., bias) to x, i.e.,
De
θ
(Wot @ (Zcs)))
By Lemma 4.1, for each f(v) c E FF and E [ne]. there exists three-layer o-network he X F ending with the activation function such that he, = (zc,)) (x). We construct the target four-layer network g as follows:
x + x @ (#1 @ ... b 2n). We note that for sequential additions I1. 92 using a,, their composition 91 o g2 is also a sequential addition.
g
W & h
yE
By Definition 4.3, for any sequential addition f. we have f(F) C F, i.c., overflow does not occur for finite inputs. Fur- thermore, by the monotonicity of the floating-point addition, we have f(x1) I f(2) for all finite floats X1 S Ø2. i.e., sequential addition preserves the order of inputs. These properties also imply the following lemma. The proof of Lemma 4.4 is presented in Appendix D.9.
Since 1 (1) = 0 if y F it holds that for each x E X,
n)((a)
g(z) nI ( @ (2)())))) = f(x).
This completes the proof of Theorem 3.4.
Lemma 4.4. For any o .F. + F and a sequential addition
4.1. Sequential Addition and Transferability
I usinn a, [[[-2mmax 26max }R) c 20max 2maax F.
To describe our proof of Lemma 4.1, we introduce the se- quential addition defined as follows.
To concisely describe what the sequential addition can do,
7

---

Floating-Poini Neural Networks Can Represent Almost All Floating-Point Functions
define the transferability as follows.
Then, one can observe that 9j.k is a two-layer network ending with the activation function. Furthermore, for any E X. Ii. = (Ci) for all j E [n] and & E {1, 2) if and only if x = 2; otherwise, there exists some j. k such that (j.((z) = (Co) = 0.
Definition 4.5 (Transferability). Let n E N and F". We say "(Z1,-..;3f) is transferable to (y y.) using o" or write "(I1,...,ZT)) _ (yI,..., (y))"...",)" if there exists sequen- tial addition f:F. FF - FF using such that f(x;) = y: for all i E [n].
(#1:--..,3Pn Fn), (W1,-..,Bh)
We now construct the target three-layer network / ending with the activation function or as
By Definitions 4.3 and 4.5, one can observe that
f(x) = (1).) In. 2n,
h(g V/2)) h2--1 (..-hg h2 (h1 (81 1 12) @ ya) ... @ y2n)
(21,12) - (I1 @ L ... @ 2n, T2
1 21
1 ... 1 2n)
for some sequential additions ,y,..., "hy...,,nn-- h2n-11 Here, we choose h1, le,..., h22-1 such that for i E [2n 2].
for all 21....., Zn E Eg. We now describe what sequential addition can do in the following lemma.
Lemma 4.6. Let a F- F and suppose that a satis- fies Condition I. Then, for any y E [-2mmar 22 I and T1 22 [-26umxs, 25mna |F such that Z2 I1 E (0, 2 Fr.
1y (0, 0 (C) h2a-1: 12n-1:(0,0(() (0, (o) (001)) 2 (Co, Co.
(0,),
it holds that
Note that such sequential additions exist by Lemma 4.6. Under these choices of h=,,--. 1,2n, one can observe that for any E X. f(x) = 0 if + and f(z) (()) if I = 2. This completes the proof of Lemma 4.1 which leads to the proof of Theorem 3,4
(-2emax V/ 4² 2mmax) و (x1, I1, I1, x2,
Due to the order-preserving property of sequential additions, for the sequential addition (say f') in Lemma 4.6, it holds that f([-2cmmm Smox y/z) = { and fly 2 }F) = {a2)- Namely, we can split the set [-2 2 |z into two parts with respect the threshold y using S. We note that such sequential addition f cannot be constructed by adding single bias term; we use the round-off error and non- associativity of floating praating-pointtae addition to prove Lemma 4.6. For more details, see the proof of Lemma 4.6 in Section 4.3.
4.3. Proof of Lemma 4.6
To describe our proof of Section 4.3, we introduce the fol- lowing lemmas. The proofs of Lemmas 4.7-4.9 are pre- sented in Appendices D. 1.10-D.12, respectively.
Lemma 4.7. Let o : F + F and suppose that a satisfies Condition 1. Then, for any 21, T2 E 1-2emax 20ma with #1 < T2 there exists a constant E (0, 20mm) l such that
4.2. Proof of Lemma 4.1
We are now ready to prove Lemma 4.1. By the defi- nition of the distinguishability, there exist E N and affine transformations $1,...,,, F + satisfying the following two properties: (i) for any y E X, there ex- ists jy E [n] such that (fo), * a (v)) and (ii) (2)) C c 2-2mmax 26max] g for all j E [m].
(21, S (0, v).
Lemma 4.8. Let a : F F and suppose that satisfies
Condition 1, Then, for any 21, 22 E 21-25mms 2 } with
Z1 > 0,
(0, E x2)
Consider Co Ci E F in Condition 1. Without loss of gen- erality, we assume Co < C1. (C1) > 0, and Co < c; the proof for the remaining cases can be done similarly.
Lemma 4.9. Let a F + F and suppose that a satisfies
Condition 1. Then, for any æ T2 E 11-2emax 2 3 with
22 22-xi 21 (0, 25max] there exists x E (0, 25max Jp such that
By Lemma 4.6, there exist sequential additions fn,1, for all j e [n] such that
(0,
22).
Ci
if (d)())) < S 2fmux
We now prove Lemma 4.6. By Lemma 4.7, there exists c E (0, 20max]p and a sequential addition f1 : (y. y²) e (0,c). Then, by the order-preserving property of sequential additions, it holds that
fz,(())
Co
if 2mmax Sx x < (x),
Co
if (0)
x < 2 mmx Sx x < (z)
f)2(-x)==
Ci if 2 Emax
fI E E Cmav .Op. fi( E (c, 25mAr)
Define g5,k X + F as
for all x E [-20msx y/z and & E [yt 20mm= Further- more, by Lemma 4.8, there exists a sequential addition
(x) = (Ji, ((-1)^-1 x x x (o)()))))
8

---

Floating-Point Neural Networks Can Represent Almost All Floating-Poing Functions
f2 : (0,c) E (0, 2m Again, by the order-preserving property of sequential additions, we have
Nevertheless, our results do not establish (i) the minimum depth and width or (ii) depth-width trade-offs for floating- point universal approximation. This contrasts with existing results for real-valued universal approximation:
fi)) @ e
f2 o2 fi = 2 mmax
for all x E [-2 y/v and = E "ut=25maxxj Jr. Here, note that fa = f2 @ w is also a sequential addition. We also choose a sequential addition fa : (-w, 0) e (-26max which exists by Lemma 4.8. Then, it holds that
It is well-known that depth 2 is necessary and sufficient for universal approximation with non-polynomial acti- vation functions (Pinkus, 1999). Recently, has been shown that width max din dout. 2} is necessary and sufficient to approximate any continuous function from [0, 1]4h to Rede in the L distance (1 I p < oo), for activation functions that can approximate the identity function and the binary step function (Shin et al., 2025). The minimum width in the L distance is known only for specific pairs of dm and dout (Kim et al., 2024). iii For ReLU networks, the number of parameters re- quired for universal approximation decreases from O(g- to O(s-da/3) as the network depth increases (Yarotsky, 2018), This benefit of depth has also been observed in the approximation of specific function classes (Telgarsky, 2016; Safran & Shamir, 2017; Chatziafratis et al., 2020).
(i)
f4 fa(x) = 2 fao = (2 ) E [0, 2max)
for all x E [--2emax f)r and 2 E 22mmxx
By Lemma 4.9, there exist x² E [-2 1mmam 26max F and a sequential addition fs : (0,z*) S (x1, In addition, by Lemmas 4.7 and 4.8, there exists a sequential addition Ss (-26mms fa((emmax) _ (0,x*). Then, we have
Is o So Sa 12(x) 31, Is so ʃ4 fz(2) 22 for all x. E [-28mmax,] f)F and E 25men. This im- plies that (-2eaux, V. y² ,,,,,2m0 2mmax) ) A (1, 22) and completes the proof of Lemma 4.6.
We believe identifying the minimum depth/width and under- standing depth-width trade-offs for floating-poin universal approximation are important directions for future research.
5. Quantitative Analysis
Our network construction uses width ((ddin (dx+))) where d is the input dimension, p and q are the numbers of bits for the mantissa and exponent, respectively, and n is constant depending on the activation function. Specifically, we use width O(ndm for each indicator function, and the entire network consists of (2((+t))))) such indicator functions-one for each input. This results in a total width of O(ndin ((++))d++)))
6. Conclusion
In this work, we propose necessary and sufficient conditions on activation functions for floating-poin point neural networks to represent large class of floating-por point functions. Specifi- cally, we demonstrate that the distinguishability of an activa- tion function is crucial for determining the representability of neural networks. Furthermore, our results show that net- works using correctly rounded practical activation functions can represent all floating-poit functions on a wide domain. We believe that our research contributes to the theoretical understanding of practical floating-poin neural networks and will provide solid foundation for future research.
We note that width of 2(((++))d)) is necessary for floating- point networks of constant depth to represent all fune- tions from dloating-point domain (e.g.. [|,, to F U (f-00,00) Consider a floating-pont width W and constant depth. Such a network has at most ( parameters. Since each parameter can take at most 0(2+t9) distinct values, the network can represent at most 20((+t))W3 distinct functions. In contrast, the number of all functions from [] to U { (000,,0) is 2(p+g))*((+tt))))) as there are 20((x+t))du) inputs and ((2++)) outputs. Therefore, to represent all such functions, the network must have width at least ((P--))d...
network with
Acknowledgements
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government MSIT (RS-2019- 11190079, Artificial Intelligence Graduate School Program, Korea University); the IITP-ITRC (Information Technology Research Center) grant funded by MSIT (IITP-202-RS) 2024-00436857); the Culture, Sports, and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2024 NSS-2024-0334500 and 25% by RS RRR-2222-00030 2034-4034:; the KIAS Individual Grant by the Center for AI and Natural Sciences at Korea Institute for Advanced Study (AP092801
Such exponential growth in dn also arises in the real-valued setting: under constant depth, networks constructed in prior works (e.g., Yarotsky (2017; 2018); Park et al. (2021); Zhang et al. (2024)) require width e @(d), where denotes the tar- get approximation error. Moreover, this exponential growth has been shown to be necessary for RcLU networks (Yarot- sky, 2018). Our results in the seating--pim setting closely mirror these findings in the real- valued setting.
9

---

Mleating-Point Neural Networks Can Represent Almost All Floating-Point Functions
and AP090301); National Research Foundation of Korea grant funded by MSIT (RS-2025-005155222 and RS-2024- 00406127) the Global University Project grant funded by GIST in 2025; and the Sejong University faculty research
Hwang, G., Lee, W., Park, Y., Park, S., and Saad, F. Floating point neural networks are provably robust universal ap- proximators. In International Conference on Computer Aided Verification (CAV), 2025.
fund 2025.
IEEE. IEEE Standard for Floating- Point Arithmetic (IEEE Sid 754-2019). IEEE, Piscataway, NJ, USA, 2019, doi: 10. TA110///EESSTTD221119...
Impact Statement
This paper investigates the representability of floating-poin point neural networks. We could not find notable potential societal consequences of our work.
Kidger, P. and Lyons, T. Universal approximation with deep narrow networks. In Conference on Learning Theory (COLT), 2020.
Kim, N., Min, C., and Park, Minimum width for universal approximation using ReLU networks on compact domain. In International Conference Learning Representations (ICLR), 2024.
References
Abadi, M., Agarwal, A., Barham, P., Brevdo, E.,, Chen, Z Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.044407 2016.
Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S. Self-normalizing neural networks. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2017.
Bochkovskiy, A., Wang, C.-Y., and Liao, H.-Y. M. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004. 10934, 2020.
LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. nature, 521(7553):436 444, 2015.
Chatziafratis, V., Nagarajan, S. G., Panageas, I., and Wang, X. Depth-width trade-offs for ReLU networks via Sharkovsky's 's theorem. In International Conference Learning Representations (ICLR), 2020.
Leshno, M., Lin, V. Y., Pinkus, A., and Schocken, S. Mul- tilayer feedforward networks with nonpolynomial ac- tivation function can approximate any function. Neural Networks, 1993.
Cybenko, G. Approximation by superpositions of a sig- moidal function. Mathematics of Control, Signals and Systems, 2(4):303-314, 1989.
Lu, Z., Pu, H., Wang, F., Hu, Z., and Wang, L. The expres- sive power of neural networks: A view from the width. In Annual Conference Neural Information Processing Systems (NeurlPS), 2017.
Ding, Y., Liu, J., Xiong, J., and Shi, Y. On the univer- sal approximability and complexity bounds of quantized ReLU neural networks. In International Conference on Learning Representations (ICLR), 2019.
Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., et al. FP8 formats for deep learning. arXiv preprint arXiv:2209.05333 2022.
Gonon, A., Brisebarre, N., Gribonval, R., and Riccietti, E. Approximation speed of quantized versus unquantized ReLU neural networks and beyond. IEEE Transactions on Information Theory, (9(6):3900-997, 2023.
Park, S., Yun, C., Lee, J., and Shin, J. Minimum width for universal approximation. In International Conference on Learning Representations (ICLR), 2021.
Park, Y.. Hwang. G., Lee, W., and Park, Expressive power of ReL.U and step networks under floating-point operations. Neural Networks, 175:106297, 2024.
Google.
Improve your model's performance with blloat16. https://clooe http://ooooo..oggle..eo com/tpu/ Mocs/bfloatl6
Pinkus, A. Approximation theory of the MLP model in neural networks. Acta Numerica, 8:143 195, 1999.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (GELUs), arXiv preprint arXiv:1606.088: 160..6066.8. 2016.
Ramachandran, P., Zoph. B., and Le, Q. V. Searching for activation functions. arXiv preprint 17X 2017.
Hornik, K., Stinchcombe, M., and White. H. Multilayer feedforward networks universal approximators. Neu- ral Networks, 2(5):359-366 1989.
Ramanujan, V., Wortsman, M., Kembhavi, A., Farhadi, A., and Rastegari, M. What's hidden in a randomly weighted neural network? In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Hwang, G., Park, Y., and Park, S. On expressive power of quantized neural networks under fixed-point arithmetic. arXiv preprint 0rXiv:2409.0000 002977 2024.
10

---

Floating-Point Neural Networks Can Represent Almost All Floating-Poin Point Functions
Safran, I. and Shamir, O. Depth-width tradeoff's in ap- proximating natural functions with neural networks. In International Conference Machine Learning (ICML), 2017.
Shin, J., Kim, N., Hwang, G., and Park, S. Minimum width for universal approximation using squashable acti- vation functions. In International Conference on Machine Learning (ICML), 2025.
Sitzmann, V., Martel, J., Bergman, A., Lindell, D., and Wet- zstein, G. Implicit neural representations with periodic activation functions. In Annual Conference Neural Information Processing Systems (NeurIPS), 2020.
Tabuada, P. and Gharesifard, Universal approximation power of deep residual neural networks via nonlinear control theory. In International Conference on Learning Representations (ICIR), 2021.
Telgarsky, M. Benefits of depth neural networks. In Conference on Learning Theory (COLT), 2016.
Yarotsky, D. Error bounds for approximations with deep ReLU networks. Neural Networks, 94: 103-114, 2017.
Yarotsky, D. Optimal approximation of continuous func- tions by very deep ReLU networks. In Conference on Learning Theory (COLT), 2018.
Yuan, C. and Agaian, S. S. A comprehensive review of binary neural network. Artificial Intelligence Review, 56 (11):12949-13013, 2023.
Yun, C., Bhojanapalli, S., Rawat, A. S., Reddi, S., and Kumar, S. Are transformers universal approximators of sequence-to-sequence functions? In International Conference on Learning Representations (ICLR), 2020.
Zhang, S., Lu, J., and Zhao, H. Deep network approxi- mation: Beyond ReLU to diverse activation functions. Journal of Machine Learning Research (JMLR), 25(35):
1-39, 2024.
Zhou, D.-X. Universality of deep convolutional neural net- works. Applied and Computational Harmonic Analysis, 4872),787-794, 2020.

---

Floating-Poinl Neural Networks Can Represent Almost All Floating-Poing Point Functions
A. Notation Table
Table 2: Notation table
Symbol
Description
Definition
Reference
F p-g F
set of all finite floating Noating-pponnt numbers. of all floating-pois numbers bit for exponent. bit for mantissa.
Section 2.1
F
F
BUf-00, NaN
Section 2.1
p.g
Section 2.1
Section 2.1
Emin
the smallest exponent of F the largest exponent of F. the smallest positive float. the largest positive float. positive, negative infinity.
29 229 + 2. 2
Section 2.1
Cmax
Section 2.1
26min -P
Section 2.1
max (2 P-
Section 2.1
-00
Section 2.1
NaN
Not number. the exponent of x E F. the significand of æ E F. normalized exponent x F. normalized mantissa of E the predecessor of x E F. the successor of I.
Section 2.1
Section 2.1
my
Section 2.1
Eq. (7)
Appendix B
m,
Eq. (7)
Appendix B
Section 2.1
at
Section 2.1
round to nearest (tie-to-even rule)
T:J
rounding operation.
Section 2.
real-valued activation function. corrected rounded function p.
Section 2.1
[p]
Section 2.1
floating activation function.
Section 2.
@ e, @
floating-poit operations. addition of multiple floats. indicator function.
Section 2.1
+
Section 2.1
1
Section 2.1
o-distinguishable with range R
Definition 3.1
Section 3.1
Co, C1,Cr C2
constants for Condition 1, separating points of constants in Lemma 3.6.
Condition Definition 3.5
Section 3.2
171 172)...... Th
Section 3.3
C, Σ, = sequential addition using a.
Section 3.3
set of output after activation.
F a(c) w. F}.
Section 4.1
Definition 4.3
Section 4.1
transferrable using ceiling function floor function
Definition 4.5
Section 4.1
min(m E Z 2 x}
Appendix B
max E Z S x)
Appendix B
B. Additional notations
The set of floating-point numbers consists of two disjoint subsets,
{0} fs x (1.m ... ...m,) x 2 : E mi...., m, E [0, e [emin Smax]z
(5)
and
{s (0.m1 ... m,) x 2 : (-1,1), mi...., m, {0,
emin {0).
(6)
element of subset (5) is called a normal floating- poiiin number. And an element of subset (6) denoted subnormal floating-pott number. For each x E I,, can represented by products of three parts, the sign s, significand mo-m ... my. and the exponent
T. = 8 mg-m7 .p x
12

---

Floating-Poini Neural Networks Can Represent Almost All Floating-Poing Functions
To clearly denote the components of a floating-point number with each binary digit m, is denoted mz In other words, the significand can be expressed m, 02,10- ... m..p- Note that for normal floating-poinl number æ, mz,0 1 whereas for subnormal floating-poing number, my,0 = 0. The By the definition, we have &x x E IP and & Cmin for subnormal floating-pont number x. In short, x can be represented as
denote its entire significand m ... my of m.
sign of z is denoted as the exponent e of z is denoted
[Cmin- Emax]2 for any
X m. X n
As the significant of normal floating-point number is always in [1, it is often convenient to express subnormal Hoating-poin in the same form. For F, m, and i. are defined the unique numbers satisfying $r E {-1,1), f. 21, e 2 and
I = 8r x my x 2
(7)
Note that if æ normal, we have mn = my and E, cy. If is subnormal, we have Emin - p S cn S Cmin - 1.
Additionally, we define the ceiling function [x]2 and the floor function [a| follows.
[ l = min (m E Z: m 2 x}. [ = max Z m x).
C. Technical lemmas
Lemma C.1. Assume that satisfies Condition 1. Consider 21,22 E IF such that cx2 S ez2 Then,
(21, x22)
- (0, 2:2)
This lemma implies that if we have two floating-poii point numbers whose exponents differ by two more, performing sequential addition can effectively eliminate the smaller number. The lemma is used to simplify calculations in the proofs of other lemmas.
Proof technique: If there is a discrepancy between the exponents of two foating-poon numbers, one can select a floating-point number such that its addition is ignored by the larger number and modifies the smaller number.
Proof of Lemma C.I. Fix 22 and mathematical induction on the absolute value of æ to prove the statement. if there is nothing to prove. Assume that for x' F such that [x'] < [x1]. the induction hypothesis is satisfied. By Lemma C.6, there exists y such that y E (2
n E. Then, 22 @ (1t)) = T2. For I > 0, J $ (->) < Iı, and for I1 < 0, [2] ① y < 21. Therefore, there exists x' such that [x'| < [11l. and
(x), (a', 32 (0, 22).
where the last relation is by the induction hypothesis. This completes the proof.
The following lemma states that if the exponent of the floating point number x is not too close emin- then, (0, x)) is transferable to (0, x0, and (.)
Lemma C.2. Assume that a satisfies Condition Consider a floating point number < J. such that emin + 2 S e, < Cmax- Then,
(0,x) - (0
+2 And if +2 S x S 26max
(0,x) 2 ))
Proof of Lemma C.2. By Lemma C.6, there exists y such that y -2---111, -p-1 1-20. N. Therefore,
(0,z) f (7, (,, و (0,x
13

---

Floating-Point Neural Networks Can Represent Almost All Floating-Poing PPint Functions
where the last relation is by Lemma C.I. Similarly, if + 25=
(0, x) / (-7, x @ (-1)) = (-7, S (0,
If x = 2 there exists y such that E (26=-p-2, 20= pp- 2-= 1 n E, and similar arguments hold. This completes the proof.
The following lemma states that if the exponent of a floating-poin number z is close to emin, then, (0, x) is transferable to (0, (. Together with Lemma C.2, for any I such that O < x S 2 (0, 2)) can be transferable to (0, ).
Lemma C.3. Assume that satisfies Condition I. For E IF such that 0 < < 26min the following relation holds:
(0, 2) - 20,
Proof of Lemma C.3. By Lemma C.6, w, 2w E Σ. If 2 < x < 20min +2 and mr.p 0, then,
(0, x) - (w, x @ w) (w,x) - (2w, x) - (0, x @ (-2w)) (0,
If c Cmin 1 and ma. L, then, fz = Cmin + Cxl
Cmin + If ef Cmin 1, then, my = 1 and thus,
(0, - (-2w, ( (-2w)) (-2w, x - (-w, x (ww = (-w, x - (0,x*)
Symmetric arguments hold for exl Cmin 1 case.
If f < 2 mmin then Nw where N E [2
Nw) - (w, (N + 1)W) -
((20+1 - N) w, 20+
((2011- N) w B 2 w w) ((20+1 N + 1) w, 2
((2011 - N + 1) 1 (-w), 2p+1 (-w)) = ((22t1 .) w, (20-1 - 1)w)
((2011 N)) w 1 (-w), (2 1) w @ (-w)) = ((2011 - -1) -)) w. (2 -2)
2) w)
-
(0.z-)
The following lemma is the corresponding version of Lemma C.3 for x that is, if the exponent of a positive floating-poit number æ is not too big, then, (0, .F) transferable (0, y) for some larger floating point numbers y. Note that, unlike the previous lemma, only claims the existence of larger y. not xt. However, together with the combination of Lemmas C.2 and C.3, one can easily check that (0, is transferable (0, x.)
Lemma C.4. Assume that a satisfies Condition 1. Consider # such that e, S Cinxx 2p Then, there exists |z], and and
y E If such that xy > 0, 2maax |y
(0, (0.
Proof of Lemma C.4. As the floating-point number is symmetric with respect to zero, we only need to consider the case of x > 0. By Condition 1, there exists K E Σ such that еœ = t + + 1. If m = or > 2c.
mm 2 (mx x 2 2(xt)t1)*) 2 mm 26 (x)==
(0, + mx
- 0, (mx x 20=+p+1 e m 20= (0,2 (0.2
If fM, = 0, o. = 2d and x is normal, by Lemma C.6, there exists y E E such that 2c, - <n S 2c- Then,
(0,z) 3 (7,00) = (x) 1 ( [m x 25+tp+1 @ 7, mx 2 @ x²)
14

---

Floating-Poinl Neural Networks Can Represent Almost All Floating-Point Functions
mx x 22ttpt1 (mx x 2 )) | (o.(me (mr x 22it @ mk x 2
0, (0,2 or (0,2
If fK 0, 2 and is subnormal, by Lemma C.6, w E. Therefore,
(0,z) (w,w = (w,x+) 1 [k x 26attt, mm x 2irtpt))") ') i (0, or (0,25.t2)
This completes the proof.
Lemma C.S. Consider flloating-poin 1) E F. Then, for any 32, E B such that F x2, |011 S [22). and
mmin +1 Ey Cxy
Cmaxx
there exist w. b such that
w
2 x1
22 1 bb = {n or {n
Furthermore,
|w| S (1+2
2 222 |6| n
and
Proof of Lemma C.5.
Case 1: 0 S T1 < 22 Consider the case
0 200 x2
define w 25 -82-
and b as b = Then,
2 =1=1
and as w 00 S 2P--P11
7 10 8 2 n
Consider the case
0 01 20 x22
my.p 1, define w as w = 2cn 2x2 -p-- and as = 1). Then, similar to 22 2 case, w x x2 + b ") and
w Z 1 @ 17. m, 0, define w
2c, - p-11 and b n Then, @@ @b 7) and 2 2 @ ry
Consider the T2 subnormal
2º2 $ d2
Then, there exists E No such that
Majj Mzzaj for j E [i],
and
0 M2y Mrg.it1 1.
If my.p define F as
2in PP
Define b
b n 1.m Mzg, O...0 02..0 x2 p-i
15

---

Floating-Poinl Neural Networks Can Represent Almost All Ploating-Poin Functions
As
1. .... Maz,i O.... x
1m ... ma2.3 2
p-i
the operation is exact: 7) 1. L.m aazd O... 0 x2 pti Then,
p-i
10 ② 21 6 b = 1. 1. mzg. Om .42 ....i...m.... mzy x 2 ) b [1] + 0. mz1. x 22m-P-11 7)
and
X 22 1 b 1.m Mxy, 1m 222 ... mxz 2 @
n + 1. L.m 22. ... mxz.p x 2- =n)
If my.
0, define
1. 1. 1.. ... May,s O...0 x
1j
p-i
and
w
-2. -P Then, similarly, -2
w 2 1 + L. my m Om mzy 2 + = [1) -0.m ... mzyp 2 = =n]"
and
22 3 b L.m.g. Maga Imggi422 ... Mz2.p x 2 ptii + +6 [1] -12 marg X 2 17.
Case 2: 21 < 0 < 22 Define b as b = 17. If Mig 1.1, define
w = -2 -2-2 PP
Then.
42 = M,y x P
and
32 b
And as
w 8 32
2 m11
7 S w 82 1 n
may 2 1.1, define as
w = 2 Eog p-1
and we get the same conclusion. This completes the proof.
Lemma C.6. Suppose that a F FF satisfies Condition with constants C₁ and C2 and let [emin p. emax plz. Then, there exists IF and [2] such that
2--1 <7 s 2
Proof of Lemma C.6. Let C and C2 from Condition be represented as
a (C1) m,(Ci) 2 and (C2) ma(Cs) x (0))))
Consider the case Cmin - - 2 < e e (C,) E [1, 5 g. define y as
1.1 -11c
16

---

Floating-Point Neural Networks Can Represent Almost All Floating-Point Functions
As Co(C;) 2 min -29-1 + 2, Eo(C)) 1 29-1- mmax and 7 E R. Then,
3
15
x 2 57x0(C) m)(())) x0.1 2 <
x 2",
16
and thus
3
x 2" 7 70 (C,) x 1.1 2c 2",
4
where the first inequality is satisfied as E 2 Cmin
p If ()()) E 2 p. define 7 as
y= (-(1)-)))
Similar to the above case, 7 E F. Then,
1 2
2 2 (C1) [mmoc.)) x x20-1/p S 2",
where the first inequality is satisfied as 1 2 Cmin - Consider the case 2 << Cmax p. If (C2)))
[1, 4 define 7 as
? = 1.1 x 20 (0(02)-
As Ca(C2) - 1 S mmax p - 1 cr((2)
1 S mmax 7 E E. Then, similar to the above case,
3
x 2 < ((xa((Ca) 2
4
If (C,) E 2) g. define y as
7= (0
and similar arguments holds.
Consider the case e = mmax p. If (C2) 2 , ddefinee 7 as
: 2-- x 2 mma --2-
Then, 7 E F, and we have
2 mma p-1 <7 (mo @ (2-2 x 2mmx- < 20max
(C2)) S 1t, then, (o()2) 2 -p-11 1. (Cs) E [1, 01 define y as
? = 1.1 2 Emax p co(0s)
and if if m) E (3. define y as
= 25max --1-p------ co
Then, F, and similar arguments hold as in 2 < < 6max P) case.
Consider the case e Cmin + 1. 1 < (Ci)
< 5 1 define y as
7 (x(0))
Then,
780(C1) = (C,) 22 = -Ima(C,) x 2w|p 2w 2".
If 1 < m)(C)) < 3. define
? 1.1 x 20-00111-1
17

---

Floating-Poinl Neural Networks Can Represent Almost All Ploating-Poinl Functions
As (oCo)) S -1,7 F. Then,
r 8 oCi) (mo))) (m(()))
1.1 2- JF = - x 1.1 x whp 2n 2
If S mo())) < 2, define y as
7 (((1))-
Then,
00 [C1] [mmo(C)) x x 1 [mo(Ci) x w| 22 2
Consider the case e = Cmin
If & m,(C)) < 1.1, define as
7 (C(C)
Then,
( = [ma(Ci) x 2c = -(mo(Cs) x w/p = w = 29.
1.1 S ()(C,) < 2, define y as
7 = (0(0)))-
As Co(C) S - E.F.E... F.F.F. Then,
00 (C1) = [mo(c.) x 2" - mo(C,) x 1 = 2 2
This completes the proof.
D. Proof of lemmas
D.1. Proof of Lemma 3.2
Proof. Suppose that X' is not o-distinguishable with range I.. Then, there exist 21, X such that for any d2 N and affine transformations Φ1, Φ2 F + F²
= =
This implies that for any a network g : X + F, g(x1) = g(x2). other words, a networks cannot represent a function f: X FF with f(x) # (x2). This completes the proof.
D.2. Proof of Lemma 3.3
Proof. By Lemma 3.2, is sufficient to prove that [cos| can not distinguish the domain with range F f-o, 20). Let (xx) == [cos(r)]. As (0) and cos( - for IF such that [x|] < 2 m-1
x²a > 2-22-22
cos(x²)
2
Therefore, for | 2 1
o (x) = [cos(z)] = 1.
Consider w, E IF such that o(w @ b) (w x e b). Then, w should satisfy
w 2 P-1
Thus,
|w| >> 2 Smin
p+1 1z
And one of ww @ @ b), / @ (-x:) @ b) is greater than or equal to x. Therefore, for |x| > 23+1222111
X 2 + co or oc,
which means that
w b) NaN.
This completes the proof.
18

---

Floating-Point Neural Networks Can Represent Almost All Floating-Point Functions
D.3. Proof of Lemma 3.6
Proof. Consider arbitrary æ1, 22 E (-20+t1, 2 such that # # æ2 and [x S [22). By the condition of the lemma, there exists E [n] such that
Cx2 E [ey,
c1, Cy,
Cmax 21z-
By Lemma C.5, there exist wo and such that
a (w 2 x1 @ b) # ow @ x2 + b),
w 2 (1+2 2 and |6| s Imdl*.
Then, ((-2est), ) @ w b) c ((-2 20ttert [[=2atautt,eaattt]]]] 1esten+ ] @b) c a ([- (2 + |malt) 20stert1 @ |mmll (n]lt
This completes the proof.
D.4. Proof of Lemma 3.7
Proof. This direct consequence of Lemma 3.6. By defining ei = E2 Cmax, en1 S 1 and 6/2 2 have
[mmin: mmax|z c U
[cn. - mmaxi Cy,, + Cmax - 2]2
i=1
Thus, F (-26maxt1 26mxr +2) is o-distinguishable with range
([ (2 @ (/lllt)) tertet11 @ we2 +1p) co (FU C 26max 20max] B
This completes the proof.
D.5. Proof of Lemma 3.9
Proof. Without loss of generality, consider the case: // 2 L for all æ E [a, and [e] (a) 2 0. As
(b)) (a) L(b a) 2
(b) (a). Therefore, there exists 7) E [a, blg such that (n) (n) s (1)*). This completes the proof.
D.6. Proof of Lemma 3.10
Proof. Define a as a := [ā]. We will prove that for any e E [cmin; e - 1|z, there exists distinguishing point v such that
e, e. For any floating-pol taating-point number if (z)) is normal, as
[a( [[A(,)] S L2x x (1+2-P).
Ca(x) S ez + [logz (L2 x (1+2 Note that for any x 2 20min
a(x) 2 (2im))) > 20mm 2 w,
Consider Lemma 3.9 with 2 (00=((2)))) (2e1+1)= - L = L1, and Co(z). As
Li (2 - 2n1) = Ln (1 - P) (P??" 2 > L1 (1- - 2 20e(s) [1oRo [1xx((xx(14227)) -
there exists distinguishing point 7) E [2 (2er+1) ) such that en = E1. Then, by Lemma 3.6 with distinguishing points with exponents Emin, Cmin l....,e e 1, eι and e2 = min (emax - l2 1, Cmax + + e-3)), the interval (-22++1 1) e is o-distinguishable with range
a ([- (2-22t @ (mmltt) 2estest1 @ [nml+]) L2 (1+2 [- (2esteat1 + |nnllt) tex + Im |+]
2/x [- -2 (21max112) 12) 2 2mmax C --20mxx , 25max)
This completes the proof.
19

---

Ploating-Point Neural Networks Can Represent Almost All Ploating-Point Functions
D.7. Proof of Corollary 3.11
Proof. Let f(x)) p(z). The result is straightforward by Table 3.
p(x)
L1
L2
h
l
e
ReLU
maax
1
-1
mmax
ELU
Emax
1
-1
Cmax
GELU
Cmax
0.5
1
-2
Cmax
SeLU
Cmax
1.05
2
-1
Smax
Swish
Emax
0.5
1
-2
Cmax
Mish
mmax
0.6
1
-2
Cmin 1
sin
0
0.540
1
-2
Cmin 2
Table 3: Properties flotting-point format for verifying the conditions. Numbers the table are rounded the second decimal place.
D.8. Proof of Lemma 4.2
Proof. By Lemma C.6, for any E [emin p, Cmax
pl. there exists y and i [2] such that
2 < 2 (C;) 2
Therefore, for any F, there exists y and i [2] such that @ 2 o(C) xt. This completes the proof.
D.9. Proof of Lemma 4.4
Proof. Represent f as
f(x)
Vi.
where n E N and yi E 2) for E [n]]. We first note that for any if y/ < 1-max-p-1 then, Dtw = 92 and -Q0 = -
And if lysl 2 2mmox -p-1 then f2 t yi = oc or -Q y/s 00. As is a function from IF to F, lya < 2 pp-1 for any i [n]. and thus, Ω @ O V:- = Ω and -Ω + +i_ V: S for any j E [n].
Now, we will use proof by contradiction. Without loss of generality, assume that f(2 > 2fmux Then, there exists
d E [n] such that 2 @ + < 2 Fmax and 2mmax + + 40 contradiction. This completes the proof.
== Di > 2mmxx Then, Bio > 2mmax -p-1 and it gives the
D.10. Proof of Lemma 4.7
Proof. Without loss of generality, assume that æ1 < x2. Use mathematical induction on |21/. If I1 = 0, there is nothing to prove. Assume that if |x'| < æ1 for any x/2 + x', then there exists y/ such that (x' (x',x%) - (0,
0 < < <2, by Lemma C.6, there exists y E Σ such that < and æ @ (-7) < and 22 @ (-7) * 4 (-7).
Therefore,
(11, f (xn @ (-7),
(-7)) (0.
by the induction hypothesis.
m < < Ø2, by Lemma C.6, there exists 7 E Σ such that .i ④ > Z1. Therefore,
(x) 22) + (x) 0y.22 @7) - (0. (0.3))
by the induction hypothesis. This completes the proof.
20

---

Floating-Point Neural Networks Can Represent Almost All Floating-Poinl Functions
D.11. Proof of Lemma 4.8
We only prove the case 21, > 0 and the remaining case is by symmetry. We first prove that for any FF such that (0, fmmxx )
(0, x - (0,x*))
If c. emin + 2, is by Lemma C.2. cz S Cmin + 1, then by Lemma C.4, there exists y E such that (0, - (0, v). Use mathematical induction on decreasing order to prove that for any < x' I (0,x) e (0,x²). Assume that if r' x". then x) > (0, x.). By Lemmas C.2 and C.3, (0, (0,z't) = (0,z"). Therefore,
(0, x)) _ (0, x (0, x²).
Therefore, the induction hypothesis holds for any x', which leads to (0, (0,x*).
Use mathematical induction on the difference
122 x1l. #1 = 22, there is nothing to prove. Assume that the lemma holds x If Ø < 22 by Lemmas C.2 and C.4 and the induction hypothesis (note that
for any s11, 2 such that ( æil 1m2 | T2 222 211.)
(0, m (0, + (0.
Similarly, if æ1 >> 22, by Lemmas C.2 and C.3 and the induction hypothesis.
(0,z1) - (0, x - 22,)
This completes the proof.
D.12. Proof of Lemma 4.9
Proof. By the symmetry, we only need to consider the case [2n2]] of 01. Assume that there exists x E F such that
[x1]- Use mathematical induction on absolute value
(0, x f (α x2);
for any |a'|| < [x1]. First consider the case 21 2 0. Φ O, there is nothing to prove.
Ji S 2emin+1, by Lemma C.6, we have w E Since there exists x', x²2 such that of @ w #1, ale w = #2, and
ii < z/2, we have
(x/, 2/2) E (o1 o (ww 2 @w) = (II, x2).
As |z1| < [xr]. the induction hypothesis is satisfied.
2ı > +1 define e as é == Cx2 By Lemma C.6, there exists E Σ such that
2--p-1 <9 2E
Then, there exists 2/2 I2 or z. such that
1/2 + 7 = Z2
Then,
(5T x2)
p e , .7) (2),
As | < xi and xı < 26max, the induction hypothesis is satisfidd
Similarly, if 2, < 0, there exists such that
(2),
S (w1 θ 7, θ7) (01,
(8)
Therefore, the induction hypothesis is satisfied. This completes the proof.
21


Reviews:
[{"review_id": "zUMbxi7Oty", "reviewer": "Reviewer_8ACf", "summary": "This article explores the representation of arbitrary floating-point functions using neural networks under floating-point arithmetic. It proves that, given certain conditions on the activation function, such representation is feasible for many floating-point functions. Throughout the proof, the article emphasizes the importance of the distinguishability of the activation function.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for their valuable comments. Our response is below.\n\n> I don't see how the generalizations of this paper are helpful for understanding neural networks in real-world applications.\n\nClassical universal approximation theorems establish that sufficiently wide neural networks can approximate any continuous function. Although achieving universal approximation requires an impractically large number of parameters, these results remain important: as Reviewer ZVKU points out, they help researchers understand the theoretical limits and fundamental properties of the tools they use.\n\nOur work extends these results by considering neural networks with *general activation functions* operating under *floating-point arithmetic*. In contrast, recent related works consider narrower or less practical settings: [Park+ 2024] focuses only on ReLU and Step activations, while [Hwang+ 2024] analyzes networks under fixed-point arithmetic. Given that modern networks use diverse activation functions (beyond ReLU and Step) and rely on floating-point arithmetic (as opposed to exact-real or fixed-point arithmetic), our results provide important new insights into the fundamental limits and capabilities of modern networks.\n\n> The paper does not provide an estimate for network width, which is crucial for practical applications. \n\nAs mentioned above, we focus on showing the *existence* of a floating-point network that can exactly represent a target floating-point function, as in existing universal approximation results (under real-valued parameters and operations). In this regard, the networks constructed in our paper are not for practical use.\nThat said, we clarify the network size as follows. In the paper, we construct a $\\sigma$-network of width $O(nd2^{(p+q)(d+1)})$, where $d$ is the dimension of an input and $n$ is a constant depending on the activation function $\\sigma$ (e.g., $n \\leq 10$ for ReLU, ELU, etc.). However, this size is to cover a general class of activations, and we believe a smaller width may suffice to obtain the same or similar results. Analyzing the minimum required width would be an important future direction.\n\n> On mixed-precision.\n\nOur construction can be extended to mixed-precision setups. E.g., if the precision in the last layer can represent the target outputs and the domain is distinguishable under the precision of the first layer, then our construction can still represent any floating-point function even with low-precision intermediate layers, as long as they satisfy Condition 1 under the low precision. \n\n> In Theorem 3.4, the activation function and the domain of the floating-point function are coupled, making the conclusions somewhat ambiguous. What happens if the domain is the entire F?\n\nThe coupling between the activation function $\\sigma$ and the domain is necessary, as shown in Lemma 3.2: if the domain (e.g., $\\mathbb F$) is not $\\sigma$-distinguishable with range $\\mathbb F$ (Definition 3.1), then universal approximation using $\\sigma$ networks is *provably impossible*. If the domain is the entire $\\mathbb F$, Theorem 3.4 states that universal approximation is possible whenever $\\mathbb F$ is $\\sigma$-distinguishable with range $[-2^{e_{max}},2^{e_{max}}]$. This condition is satisfied, e.g., for $\\sigma = \\lceil\\text{Sigmoid}\\rfloor, \\lceil\\tanh\\rfloor$ (Corollary 3.11). \n\n> On the depth of universal approximators.\n\nOur current construction can be extended to five or more layers, but not to two or three layers. This is because our indicator function uses more than two layers and we use an additional layer to represent a linear combination of indicator functions. Investigating the minimum number of layers for universal approximation under floating-point arithmetic is an interesting research direction. \n\n> On multithreaded environment.\n\nIn our construction, we control round-off errors from floating-point operations by considering a single, fixed order of these operations (Sections 2.2–2.3). We expect that our results can be extended to multithread setups if the order of the operations in each layer is fixed (i.e., it does not depend on input values, memory size, etc.). However, if the order is not fixed, extending our results might be non-trivial due to the non-associativity of floating-point arithmetic.\n\n> The cos⁡ function is not periodic in computer arithmetic. Could this impact the conclusion of Lemma 3.3?  \n\nThe $\\lceil\\cos\\rfloor$ function in Lemma 3.3 is the rounded version of $\\cos$. Thus, $\\lceil\\cos\\rfloor$ is already aperiodic and the conclusion of Lemma 3.3 is valid for $\\lceil\\cos\\rfloor$.\n\n> On “almost all” in the abstract.\n\nDue to the 5000-character limit, we refer to our response to the same question from Reviewer EifD.\n\n> On related works.\n\nFollowing the reviewer’s comment, we will include suggested related works on binary networks and sparse networks.\n\nWe would be happy to clarify any concerns or answer any questions, that may come up during the discussion period.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods make sense for the problem at hand.", "theoretical_claims": "I only checked the statements but not the proofs. I believe the claims are correct.", "experimental_designs_or_analyses": "Null", "relation_to_broader_scientific_literature": "This paper extends the results of Hwang et al. (2024) and Park et al. (2024). However, I don't see how the generalizations of this paper are helpful for understanding neural networks in real-world applications. Especially, the discussion of network width is lacking in this paper.", "essential_references_not_discussed": "From a broader perspective, there has been extensive research on floating-point weights, including studies on binary networks and related topics.\n\ne.g. Yuan, C., & Agaian, S. S. (2023). A Comprehensive Review of Binary Neural Network. Artificial Intelligence Review, 56(11), 12949-13013.\n\nFrom a proof-based standpoint, the networks considered in this paper need to be extremely wide, which is closely related to the concept of \"network in network.\" When the network is sufficiently wide, a subnetwork that achieves universal approximation can be selected.\n\ne.g. Ramanujan et al. (2020). What’s Hidden in a Randomly Weighted Neural Network? CVPR 2020.", "other_strengths_and_weaknesses": "1. The floating-point functions and floating-point weights considered in this paper use the same precision, which differs from real-world applications. In practical neural networks, some layers use 8-bit weights, while others use 16-bit. Whether this study can be extended to mixed-precision scenarios?  \n\n2. In Theorem 3.4, the activation function and the domain $M$ of the floating-point function are coupled, making the conclusions somewhat ambiguous. What happens if the domain is the entire $F$? Additionally, the abstract mentions the concept of “almost all,” which is confusing—under what mathematical sense is this term being used?  \n\n3. The main result of this paper proves the existence of a four-layer network. Is this conclusion too specific? Can it be extended to networks with arbitrary depth, such as two-layer or five-layer networks?  \n\n4. The paper does not provide an estimate for network width, which is crucial for practical applications. If the required width is extremely large, then even if the network has function representation capabilities, it may not be practically useful.", "other_comments_or_suggestions": "To achieve a given level of precision, is it better to use a wide network with low precision or a narrow network with high precision? It would be valuable for the authors to explore this question in the paper.", "questions_for_authors": "1. In real-world applications, computations may be performed in a multithreaded environment. How does the non-associativity of floating-point arithmetic affect the results in such cases?\n2. In practice, the cos⁡ function is not periodic in computer arithmetic. Could this impact the conclusion of Lemma 3.3?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yxsonmhePa", "reviewer": "Reviewer_ZVKU", "summary": "The paper describes a condition, under which the discrete neural networks (all NNs running on digital computers with finite precision of real numbers) are universal approximators. I think this is important line of work, because the proof from 90s assumes real numbers, but, computers cannot store real numbers. With respect to the latest popularity of low-bits representations (16bit Floats are common, we even see 4bit Float), the importance is even more pronounced. While these proofs do not have an immediate impact on the practical applications, I believe it is important to know the limits and properties of tools we use, to know if we do the right thing.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive evaluation and thoughtful feedback. We address all comments of the reviewer below.\n\n> It is very hard to read. Sometimes, I have a feeling symbols are not well defined and explained. For example in Definition 3.5, I do not know what $\\eta^+$ and $\\eta^-$ is and where it come from.\n\nFor better readability, we will add a table describing notations and definitions in the Appendix of the final version. $\\eta^+$ and $\\eta^-$ denote the smallest float that is larger than $\\eta$ and the largest float that is smaller than $\\eta$, respectively. Although we defined these notations in Section 2.1, we will recall their definitions after Definition 3.5. We will also recall seldom-used notations in the final draft. \n\n> The supplementary material is absolutely necessary for understanding. In my opinion, the authors could help the reader by repeating the relevant definitions from the main text, such that the reader is spared from going back and forth.\n\nFollowing the reviewer’s comment, we will make the Appendix self-contained in the final version, by redefining notations and definitions at the beginning of the Appendix along with a table summarizing these notations and definitions.\n\nWe would be happy to clarify any concerns or answer any questions that may come up during the discussion period.", "claims_and_evidence": "The paper claims that in order the neural network to approximate all functions, it needs to be able to separate all point. This condition is very similar to the condition used in the proof of universal approximation theorem based on Stone-Weirstrass theorem, and therefore expectable and understandable.\n\nThe main claim is Theorem 3.4, which states that if conditions are satisfied, then any function can be approximated with four layer neural network. The proof is based on expressing the function as an affine combination of indicator functions, which is plausible.", "methods_and_evaluation_criteria": "No experiments.", "theoretical_claims": "I confess that I did not have enough time to go through the proofs. All I can say that based on my knowledge of match, they looks very plausible. The tools seems appropriate.", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "While the universal approximation theorems do not have immediate impact on practice, I personally consider them important. The importance for me is that the problem I am solving is within the solution space, although the precise number of units is not known.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "It is very hard to read. Sometimes, I have a feeling symbols are not well defined and explained. For example in Definition 3.5, I do not know what $\\eta^+$ and $\\eta^-$ is and where it come from.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "n/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BcEj0eqCRb", "reviewer": "Reviewer_pLPG", "summary": "This paper investigates the expressive power of floating-point neural networks, focusing on their ability to represent floating-point functions under practical constraints such as finite precision and inexact arithmetic operations. The authors establish necessary and sufficient conditions for activation functions to ensure that floating-point neural networks can represent almost all floating-point functions. The main algorithmic contribution lies in the construction of a four-layer network that can represent any floating-point function under the given conditions, leveraging properties of floating-point arithmetic and activation function.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive evaluation and thoughtful feedback. We address all comments of the reviewer below.\n\n> Lack of empirical validation. While the theoretical results are robust, the absence of experimental results or case studies makes it difficult to assess the practical implications and real-world performance of the proposed methods.\n\nSince our universal approximator construction consists of an affine combination of indicator functions (Section 4), following the reviewer’s comment, we have empirically verified our indicator function constructions (Lemma 4.1) using GELU networks and 16-bit floating-point arithmetic (float16). Specifically, we constructed indicator functions $f_z(x) := 1[x=z]$ of points $z \\in [-1,1]\\_{\\mathbb F}$ (whose domain is $[-1,1]\\_{\\mathbb F}$), where $\\mathbb F$ denotes the set of 16-bit floats. We sampled $120$ random $z$ from $[-1,1]\\_{\\mathbb F}$ (i.e., $120$ indicator functions), and constructed the indicator functions $f_z$ using GELU networks. For each $z$, we checked whether our construction generates the same output as in $1[x=z]$ for $1000$ random inputs $x$ drawn from $[-1,1]\\_{\\mathbb F}$. According to our experimental results, the outputs of our constructions coincided with those of the true indicator functions, confirming that our construction is correct.\n\nThe code is available at the following anonymous link: \n\nhttps://anonymous.4open.science/r/floating-points-indicators-A112/\n\nTo run the code, use the following command (trials: the number of indicator constructions, verification_steps: the number of verification steps for each constructed indicator) with NumPy version 1.24.0: \n\n`python indicator_test.py --activation gelu --trials 120 --verification_steps 1000`\n\n> While the paper covers a wide range of activation functions, it does not explore the potential trade-offs or practical challenges associated with implementing these functions in real-world systems, such as computational efficiency or hardware constraints.\n\nTheorem 3.4 (our main result) holds for all floating-point activation functions that satisfy Condition 1 and a distinguishability condition. Corollaries 3.9 and 3.11 show that the following are examples of such activation functions: the correctly-rounded versions of many popular real-valued activation functions, such as id (the identity function), sin, tanh, ReLU, ELU, SeLU, GELU, Swish, Mish, and Sigmoid. We believe there would be no practical challenges in implementing these functions in real-world systems:\n\n- The following functions are already implemented in real-world systems and widely used: $\\lceil \\text{id} \\rfloor$ and $\\lceil \\text{ReLU} \\rfloor$ are available in deep learning frameworks (e.g., PyTorch, TensorFlow, JAX) as `torch.nn.Identity()(x)`, `torch.nn.ReLU()(x)`, etc; $\\lceil \\text{sin} \\rfloor$ and $\\lceil \\text{tanh} \\rfloor$ are available in correctly-rounded math libraries (e.g., RLibm, CORE-MATH, CRLibm) as `sin(x)` and `tanh(x)`.\n\n- To our knowledge, the other functions listed above (i.e., the correctly-rounded versions of ELU, SeLU, etc.) are not yet implemented in real-world systems. Nevertheless, we believe these functions will also be implemented in the near future, especially given recent active research on correctly-rounded math libraries in various areas such as computer arithmetic (e.g., [Sibidanov+ 2022]) and programming languages (e.g., [Lim+ 2021]).\n\n[Sibidanov+ 2022] The CORE-MATH Project, ARITH 2022.  \n[Lim+ 2021] High performance correctly rounded math libraries for 32-bit floating point representations, PLDI 2021.\n\nWe would be happy to clarify any concerns or answer any questions that may come up during the discussion period.", "claims_and_evidence": "The claims made in the submission are generally well-supported by clear and convincing evidence. The authors provide rigorous mathematical proofs and detailed analyses to substantiate their claims regarding the necessary and sufficient conditions for activation functions in floating-point neural networks.  Overall, the evidence is strong, but empirical validation would enhance the paper's impact.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-suited for the problem at hand, focusing on the theoretical underpinnings of floating-point neural networks and their ability to represent floating-point functions. The authors employ a rigorous mathematical approach, leveraging properties of floating-point arithmetic and activation functions to establish necessary and sufficient conditions for universal representation.", "theoretical_claims": "The authors provide detailed and rigorous arguments to support their claims, particularly in establishing the necessary and sufficient conditions for activation functions in floating-point neural networks.", "experimental_designs_or_analyses": "The paper primarily focuses on theoretical contributions and does not include experimental designs or empirical analyses.", "relation_to_broader_scientific_literature": "The key contributions of this paper are closely related to and build upon several important strands of research in the broader scientific literature on neural networks and their expressive power. The authors extend the classical universal approximation theorems, which traditionally assume real-valued parameters and exact mathematical operations, to the more practical context of floating-point arithmetic with finite precision and inexact operations.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "This paper demonstrates significant strengths in its originality and theoretical depth. However, the paper also has some weaknesses that could be addressed to further strengthen its contribution.\n\n-Lack of empirical validation. While the theoretical results are robust, the absence of experimental results or case studies makes it difficult to assess the practical implications and real-world performance of the proposed methods. \n\n-While the paper covers a wide range of activation functions, it does not explore the potential trade-offs or practical challenges associated with implementing these functions in real-world systems, such as computational efficiency or hardware constraints.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VrACBgJdqf", "reviewer": "Reviewer_EifD", "summary": "This paper explores the classes of functions over various floating-point arithmetics\nthat can be represented using feedforward neural networks (FNNs) employing different activation functions, constrained by the corresponding\nfloating-point arithmetic.\n\nIn particular, the authors present a necessary condition, referred to as distinguishability (Lemma 3.2), and a sufficient condition (Condition 1) for an activation function, under which there exists an FNN representing the same function (Theorem 3.4). Furthermore, they provide sufficient conditions for activation functions to meet distinguishability (Section 3.3) and demonstrate that, for several common floating-point arithmetics and activation functions constrained by said arithmetics, they satisfy distinguishability and Condition 1 over the entire range of floating-point numbers (Corollaries 3.9 and 3.11). To provide a comprehensive view, they include an example of an activation function, the rounded cosine, which is not generally expressive when limited by floating-point arithmetic (Lemma 3.3).\n\n## update after rebuttal\nNo changes, I stand with my initial review.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive evaluation and thoughtful feedback. We address all comments of the reviewer below.\n\n> The paper might benefit from offering more intuitions or informal explanations for its results. Overall, it is somewhat challenging to read for those not deeply engaged with the formal or theoretical aspects of machine learning.\n\nFollowing the reviewer’s comment, we will include the following informal explanations describing our intuition behind our proof at the beginning of Section 4.\n\n“We construct the target four-layer network as a linear combination of indicator functions of points (Lemma 4.1). Given $z$ in the domain, our construction of the indicator function of $z$ consists of three parts. In the first part, we create an injective layer consisting of an affine transformation followed by an activation function based on the distinguishability of the activation function. Due to the injectivity, all inputs in the domain have distinct output vectors after passing this part. Here, we use $y=(y_1,\\dots,y_k)$ to denote the output of the first part of $z$. In the second part, we construct $2n$ binary step functions $f_{1,1},f_{1,2}, \\dots,f_{n,1}, f_{n,2}$ where $f_{i,1}(x)=(C_1-C_0)1[x\\ge y_i]+C_0$ and $f_{i, 2}(x)=(C_1-C_0)1[x\\le y_i]+C_0$. Namely, all $f_{i,j}$ are $C_1$ if and only if the input to the network is $z$. To implement such $f_{i,j}$, we exploit the round-off error in floating-point operations and show that we can increase the gap between arbitrary two numbers by sequentially adding the same floating-point numbers to those two numbers (see Lemma 4.6). In the third part, we construct a function that outputs $C_1$ or $C_2$ if all $f_{i,j}$ are $C_1$ and outputs $C_0$ otherwise. We then apply the activation function after this function so that our indicator function is either zero (i.e., $\\sigma(C_0)$) if the input is $z$ or some non-zero value (i.e., $\\sigma(C_1)$ or $\\sigma(C_2)$) otherwise.”\n\n> The abstract is slightly unclear because it does not explicitly specify what \"almost all\" pertains to. While this level of ambiguity might be acceptable for the title, I believe that the abstract should offer more clarification in this regard for better comprehension.\n\nWe thank the reviewer for pointing out a confusing expression. Following the reviewer’s comment, we will change the last sentence in the abstract from\n\n“Our result shows that with practical activation functions, floating-point neural networks can represent almost all floating-point functions.”\n\nto\n\n“Our result shows that with practical activation functions, floating-point neural networks can represent floating-point functions from a wide domain to all finite or infinite floats, where the domain is all finite floats for the activations sigmoid and tanh, and it is all floats of magnitude less than $1/8$ times the largest float for the activations ReLU, ELU, SeLU, GELU, Swish, Mish, and sin.”\n\n> As I am not familiar with the work directly related, can you make clear how the arguments of Park et al. differ from yours? I assume they rely on similar \"constructive arguments\"?\n\nThe network construction of Park et al. (2024) and that of our work are both based on a linear combination of indicator functions, as in our Lemma 4.1. However, Park et al.’s construction of indicator functions is specialized to ReLU and binary step activation functions (since they consider only these two activations), whereas our construction covers general activation functions satisfying Condition 1.\n- For example, in (Park et al., 2024), they use the vanilla binary step function as an indicator function; for ReLU, they construct an indicator function using the following intuition: $1[x\\ge z] \\approx \\text{ReLU}\\left(\\frac1{z-z^-}(x-z^-)\\right) - \\text{ReLU}\\left(\\frac1{z-z^-}(x-z)\\right)$.\n\n- On the other hand, our construction of indicator functions is completely different: we mainly use the properties of a floating-point summation and the round-off errors from floating-point additions (Lemma 4.6). \n\nWe would be happy to clarify any concerns or answer any questions that may come up during the discussion period.", "claims_and_evidence": "All claims are provided with rigorous definitions and proofs.", "methods_and_evaluation_criteria": "As the contributions are purely theoretical, the methods by which they are proven make sense.", "theoretical_claims": "While I did not examine the core proofs in granular detail, there is no reason to doubt their correctness given the precise definitions and claims the paper provides.", "experimental_designs_or_analyses": "Not applicable.", "relation_to_broader_scientific_literature": "As stated by the authors themselves (p. 1, right column, last paragraph), there are two works that are very similar to this one, specifically Hwang et al. (2024) and Park et al. (2024). However, I believe the contribution of this submission is substantial as it represents a natural continuation and generalisation.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "strengths:\n\n- The paper presents clear and precise arguments, making its case convincingly.\n- It is very rigorous in defining the floating-point arithmetic considered, making \nthe results well-defined.\n- It provides a relatively comprehensive picture of the expressive capabilities of FNNs within floating-point frameworks.\n\nweaknesses:\n\n- The paper might benefit from offering more intuitions or informal explanations\n  for its results. Overall, it is somewhat challenging to read for those not\n  deeply engaged with the formal or theoretical aspects of machine learning.", "other_comments_or_suggestions": "The abstract is slightly unclear because it does not explicitly specify what \"almost all\" pertains to. While this level of ambiguity might be acceptable for the title, I believe that the abstract should offer more clarification in this regard for better comprehension.", "questions_for_authors": "- As I am not familiar with the work directly related, can you make clear how the arguments of Park et al. differ from yours? I assume they rely on similar \"constructive arguments\"?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper contributes to the understanding of universal approximation properties of neural networks with floating point arithmetics. The reviewers largely agreed that the paper is mathematically sound, well-written, and on a timely topic - namely bringing theoretical properties (universal approximation) together with practical restrictions (floating point arithmetics). However, there was some discussion about the novelty and significance of the results, with mainly one reviewer objecting against acceptance because they see the contribution as too narrow. In the end, however, we could reach consensus that this work constitutes a meaningful contribution to ICML, by proving fundamental results which hopefully will, on the long run, trigger more research on the topic. I urge the authors to take the comments by the reviewers seriously. Among others, they should incorporate a decent overview of quantitative UA theorems in the real-valued case and mention analoguous results for the floating point case as an open question.
Decision: Accept (poster)