SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhun Deng
zhun.d@columbia.edu
Thomas P. Zollo
tpz2105@columbia.edu
Jake C. Snell
js2523@princeton.edu
Toniann Pitassi
toni@cs.columbia.edu
Richard Zemel
zemel@cs.columbia.edu
Explicit finite-sample statistical guarantees on model performance are an important ingredient in responsible machine learning. Previous work has focused mainly on bounding either the expected loss of a predictor or the probability that an individual prediction will incur a loss value in a specified range. However, for many high-stakes applications it is crucial to understand and control the dispersion of a loss distribution, or the extent to which different members of a population experience unequal effects of algorithmic decisions. We initiate the study of distribution-free control of statistical dispersion measures with societal implications and propose a simple yet flexible framework that allows us to handle a much richer class of statistical functionals beyond previous work. Our methods are verified through experiments in toxic comment detection, medical imaging, and film recommendation.
Learning-based predictive algorithms are widely used in real-world systems and have significantly impacted our daily lives. However, many algorithms are deployed without sufficient testing or a thorough understanding of likely failure modes. This is especially worrisome in high-stakes application areas such as healthcare, finance, and autonomous transportation. In order to address this critical challenge and provide tools for rigorous system evaluation prior to deployment, there has been a rise in techniques offering explicit and finite-sample statistical guarantees that hold for any unknown data distribution and black-box algorithm, a paradigm known as distribution-free uncertainty quantification (DFUQ). In [1], a framework is proposed for selecting a model based on bounds on expected loss produced using validation data. Subsequent work [30] goes beyond expected loss to provide distribution-free control for a class of risk measures known as quantile-based risk measures (QBRMs) [8]. This includes (in addition to expected loss): median, value-at-risk (VaR), and conditional value-at-risk (CVaR) [23]. For example, such a framework can be used to get bounds on the 80th percentile loss or the average loss of the  $10\%$  worst cases.
While this is important progress towards the sort of robust system verification necessary to ensure the responsible use of machine learning algorithms, in some scenarios measuring the expected loss or value-at-risk is not enough. As models are increasingly deployed in areas with long-lasting societal consequences, we should also be concerned with the dispersion of error across the population, or the extent to which different members of a population experience unequal effects of decisions made based on a model's prediction. For example, a system for promoting content on a social platform may offer less appropriate recommendations for the long tail of niche users in service of a small set
of users with high and typical engagement, as shown in [19]. This may be undesirable from both a business and societal point of view, and thus it is crucial to rigorously validate such properties in an algorithm prior to deployment and understand how the outcomes disperse. To this end, we offer a novel study providing rigorous distribution-free guarantees for a broad class of functionals including key measures of statistical dispersion in society. We consider both differences in performance that arise between different demographic groups as well as disparities that can be identified even if one does not have reliable demographic data or chooses not to collect them due to privacy or security concerns. Well-studied risk measures that fit into our framework include the Gini coefficient [33] and other functions of the Lorenz curve as well as differences in group measures such as the median [5]. See Figure 4 for a further illustration of loss dispersion.
Figure 1: Example illustrating how two predictors (here  $h_1$  and  $h_2$ ) with the same expected loss can induce very different loss dispersion across the population. Left: The loss CDF produced by each predictor is bounded from below and above. Middle: The Lorenz curve is a popular graphical representation of inequality in some quantity across a population, in our case expressing the cumulative share of the loss experienced by the best-off  $\beta$  proportion of the population. CDF upper and lower bounds can be used to bound the Lorenz curve (and thus Gini coefficient, a function of the shape of the Lorenz curve). Under  $h_2$  the worst-off population members experience most of the loss. Right: Predictors with the same expected loss may induce different median loss for (possibly protected) subgroups in the data, and thus we may wish to bound these differences.
In order to provide rigorous guarantees for socially important measures that go beyond expected loss or other QBRMs, we provide two-sided bounds for quantiles and nonlinear functionals of quantiles. Our framework is simple yet flexible and widely applicable to a rich class of nonlinear functionals of quantiles, including Gini coefficient, Atkinson index, and group-based measures of inequality, among many others. Beyond our method for controlling this richer class of functionals, we propose a novel numerical optimization method that significantly tightens the bounds when data is scarce, extending earlier techniques [21, 30]. We conduct experiments on toxic comment moderation, detecting genetic mutations in cell images, and online content recommendation, to study the impact of our approach to model selection and tailored bounds.
To summarize our contributions, we: (1) initiate the study of distribution-free control of societal dispersion measures; (2) generalize the framework of [30] to provide bounds for nonlinear functionals of quantiles; (3) develop a novel optimization method that substantially tightens the bounds when data is scarce; (4) apply our framework to high-impact NLP, medical, and recommendation applications.
We consider a black-box model that produces an output  $Z$  on every example. Our algorithm selects a predictor  $h$ , which maps an input  $Z \in \mathcal{Z}$  to a prediction  $h(Z) \in \hat{\mathcal{V}}$ . A loss function  $\ell : \hat{\mathcal{V}} \times \mathcal{V} \to \mathbb{R}$  quantifies the quality of a prediction  $\hat{Y} \in \hat{\mathcal{V}}$  with respect to the target output  $y \in \mathcal{V}$ . Let  $(Z, Y)$  be drawn from an unknown joint distribution  $\mathcal{P}$  over  $\mathcal{Z} \times \mathcal{V}$ . We define the random variable  $X^h := \ell(h(Z), Y)$  as the loss induced by  $h$  on  $\mathcal{P}$ . The cumulative distribution function (CDF) of the random variable  $X^h$  is  $F^h(x) := \mathbb{P}(X^h \leq x)$ . For brevity, we sometimes use  $X$  and  $F$  when we do not need to explicitly consider  $h$ . We define the inverse of a CDF (also called inverse CDF)  $F$  as  $F^{-}(p) = \inf\{x : F(x) \geq p\}$  for any  $p \in \mathbb{R}$ . Finally, we assume access to a set of validation samples  $(Z, Y)_{1:n} = \{(Z_1, Y_1), \ldots, (Z_n, Y_n)\}$  for the purpose of achieving distribution-free CDF control with mild assumptions on the loss samples  $X_{1:n}$ . We emphasize that the "distribution-free"
requirement is on  $(Z,Y)_{1:n}$  instead of  $X_{1:n}$ , because the loss studied on the validation dataset is known to us and we can take advantage of properties of the loss such as boundedness.
3 Statistical dispersion measures for societal applications
In this section, we motivate our method by studying some widely-used measures of societal statistical dispersion. There are key gaps between the existing techniques for bounding QBRMs and those needed to bound many important measures of statistical dispersion. We first define a QBRM:
Definition 1 (Quantile-based Risk Measure). Let  $\psi(p)$  be a weighting function such that  $\psi(p) \geq 0$  and  $\int_0^1 \psi(p) dp = 1$ . The quantile-based risk measure defined by  $\psi$  is
$$
R _ {\psi} (F) := \int_ {0} ^ {1} \psi (p) F ^ {-} (p) d p.
$$
A QBRM is a linear functional of  $F^{-}$ , but quantifying many common group-based risk dispersion measures (e.g. Atkinson index) also involves forms like nonlinear functions of the (inverse) CDF or nonlinear functionals of the (inverse) CDF, and some (like maximum group differences) further involve nonlinear functions of functionals of the loss CDF. Thus a much richer framework for achieving bounds is needed here.
For clarity, we use  $J$  as a generic term to denote either the CDF  $F$  or its inverse  $F^{-}$  depending on the context, and summarize the building blocks as below: (i) nonlinear functions of  $J$ , i.e.  $\xi(J)$ ; (ii) functionals in the form of integral of nonlinear functions of  $J$ , i.e.  $\int \psi(p) \xi(J(p)) dp$  for a weight function  $\psi$ ; (iii) composed functionals as nonlinear functions of functionals for the functional  $T(J)$  with forms in (ii), i.e.  $\zeta(T(J))$  for a non-linear function  $\zeta$ .
3.1 Standard measures of dispersion
We start by introducing some classic non-group-based measures of dispersion. Those measures usually quantify wealth or consumption inequality within a social group (or a population) instead of quantifying differences among groups. Note that for all of these measures we only consider non-negative losses  $X$ , and assume that  $\int_0^1 F^{-}(p) dp > 0^1$ .
Gini family of measures. Gini coefficient [33, 34] is a canonical measure of statistical dispersion, used for quantifying the uneven distribution of resources or losses. It summarizes the Lorenz curve introduced in Figure 4. From the definition of Lorenz curve, the greater its curvature is, the greater inequality there exists; the Gini coefficient is measuring the ratio of the area that lies between the line of equality (the  $45^{\circ}$  line) and the Lorenz curve to the total area under the line of equality.
Definition 2 (Gini coefficient). For a non-negative random variable  $X$ , the Gini coefficient is
$$
\mathcal {G} (X) := \frac {\mathbb {E} | X - X ^ {\prime} |}{2 \mathbb {E} X} = \frac {\int_ {0} ^ {1} (2 p - 1) F ^ {-} (p) d p}{\int_ {0} ^ {1} F ^ {-} (p) d p},
$$
where  $X^{\prime}$  is an independent copy of  $X$ .  $\mathcal{G}(X)\in [0,1]$ , with 0 indicating perfect equality.
Because of the existence of the denominator in the Gini coefficient calculation, unlike in QBRM we need both an upper and a lower bound for  $F^{-}$  (see Section 4.1.1). In the appendix, we also introduce the extended Gini family.
Atkinson index. The Atkinson index [2, 19] is another renowned dispersion measure defined on the non-negative random variable  $X$  (e.g., income, loss), and improves over the Gini coefficient in that it is useful in determining which end of the distribution contributes most to the observed inequality by choosing an appropriate inequality-aversion parameter  $\varepsilon \geq 0$ . For instance, the Atkinson index becomes more sensitive to changes at the lower end of the income distribution as  $\varepsilon$  increases.
Definition 3 (Atkinson index). For a non-negative random variable  $X$ , for any  $\varepsilon \geq 0$ , the Atkinson index is defined as the following if  $\varepsilon \neq 1$ :
$$
\mathcal {A} (\varepsilon , X) := 1 - \frac {(\mathbb {E} [ X ^ {1 - \varepsilon} ]) ^ {\frac {1}{1 - \varepsilon}}}{\mathbb {E} [ X ]} = 1 - \frac {\left(\int_ {0} ^ {1} (F ^ {-} (p)) ^ {1 - \varepsilon} d p\right) ^ {\frac {1}{1 - \varepsilon}}}{\int_ {0} ^ {1} F ^ {-} (p) d p}.
$$
<sup>1</sup>This assumption is included for concise description, but not necessary. We also set  $0 / 0 = 0$ .
And for  $\varepsilon = 1$ ,  $\mathcal{A}(1, X) \coloneqq \lim_{\varepsilon \to 1} \mathcal{A}(\varepsilon, X)$ , which will converge to a form involving the geometric mean of  $X$ .  $\mathcal{A}(\varepsilon, X) \in [0, 1]$ , and 0 indicates perfect equality (see appendix for details).
The form of Atkinson index includes a nonlinear function of  $F^{-}$ , i.e.  $(F^{-})^{1 - \varepsilon}$ , but this type of nonlinearity is easy to tackle since the function is monotonic w.r.t. the range of  $F^{-}$  (see Section 4.2.1).
Remark 1. The reason we study the CDF of  $X$  and not  $X^{1 - \varepsilon}$  is that it allows us to simultaneously control the Atkinson index for all  $\varepsilon$ 's.
In addition, there are many other important measures of dispersion involving more complicated types of nonlinearity such as the quantile of extreme observations and mean of range. Those measures are widely used in forecasting weather events or food supply. We discuss and formulate these dispersion measures in the appendix.
3.2 Group-based measures of dispersion
Another family of dispersion measures refer to minimizing differences in performance across possibly overlapping groups in the data defined by (protected) attributes like race and gender. Under equal opportunity [11], false positive rates are made commensurate, while equalized odds [11] aims to equalize false positive rates and false negative rates among groups. More general attempts to induce fairly-dispersed outcomes include CVaR fairness [32] and multi-calibration [14, 15]. Our framework offers the flexibility to control a wide range of measures of a group CDF  $F_{g}$ , i.e.  $T(F_{g})$ , as well as the variation of  $T(F_{g})$  between groups. As an illustration of the importance of such bounds, [5] finds that the median white family in the United States has eight times as much wealth as the median black family; this motivates a dispersion measure based on the difference in group medians.
Absolute/quadratic difference of risks and beyond. The simplest way to measure the dispersion of a risk measure (like median) between two groups are quantities such as  $|T(F_g) - T(F_{g'})|$  or  $[T(F_g) - T(F_{g'})]^2$ . Moreover, one can study  $\xi(T(F_g) - T(F_{g'}))$  for some general nonlinear functions. These types of dispersion measures are widely used in algorithmic fairness [11, 20].
CVaR-fairness risk measure and its extensions. In [32], the authors further consider a distribution for each group,  $\mathcal{P}_g$ , and a distribution over group indices,  $\mathcal{P}_{\mathrm{Idx}}$ . Letting  $CVaR_{\alpha,\mathcal{P}_Z}(Z) \coloneqq \mathbb{E}_{Z \sim \mathcal{P}_Z}[Z|Z > \alpha]$  for any distribution  $\mathcal{P}_Z$ , they define the following dispersion for the expected loss of group  $g$  (i.e.  $\mu_g \coloneqq \mathbb{E}_{X \sim \mathcal{P}_g}[X]$ ) for  $\alpha \in (0,1)$ :
$$
\mathcal {D} _ {C V, \alpha} (\mu_ {g}) := C V a R _ {\alpha , \mathcal {P} _ {\mathrm {l d x}}} \left(\mu_ {g} - \mathbb {E} _ {g \sim \mathcal {P} _ {\mathrm {l d x}}} [ \mu_ {g} ]\right).
$$
A natural extension would be  $\mathcal{D}_{CV,\alpha}(T(F_g))$  for general functional  $T(F_{g})$ , which we can write in a more explicit way [23]:
$$
\mathcal {D} _ {C V, \alpha} (T (F _ {g})) = \min  _ {\rho \in \mathbb {R}} \left\{\rho + \frac {1}{1 - \alpha} \cdot \mathbb {E} _ {g \sim \mathcal {P} _ {\mathrm {I d x}}} [ T (F _ {g}) - \rho ] _ {+} \right\} - \mathbb {E} _ {g \sim \mathcal {P} _ {\mathrm {I d x}}} [ T (F _ {g}) ].
$$
The function  $[T(F_g) - \rho]_+$  is a nonlinear function of  $T(F_g)$ , but it is a monotonic function when  $\rho$  is fixed and its further composition with the expectation operation is still monotonic, which can be easily dealt with.
Uncertainty quantification of risk measures. In [4], the authors study the problem of uncertainty of risk assessment, which has important consequences for societal measures. They formulate a deviation-based approach to quantify uncertainty for risks, which includes forms like:  $\rho_{\xi}(\mathcal{P}_{\mathrm{Idx}}) \coloneqq \mathbb{E}_{g \sim \mathcal{P}_{\mathrm{Idx}}}[\xi(T(F_g))]$  for different types of nonlinear functions  $\xi$ . Examples include variance uncertainty quantification, where  $\mathbb{E}_{g \sim \mathcal{P}_{\mathrm{Idx}}}\left(T(F_g) - \mathbb{E}_{g \sim \mathcal{P}_{\mathrm{Idx}}}T(F_g)\right)^2$ ; and  $\mathbb{E}_{\psi}[\xi(F^{-}(\alpha))] \coloneqq \int_0^1 \xi(F^{-}(\alpha))\psi(\alpha)d\alpha$  to quantify how sensitive the  $\alpha$ -VaR value is w.r.t its parameter  $\alpha$  for some non-negative weight function  $\psi$ .
4 Distribution-free control of societal dispersion measures
In this section, we introduce a simple yet general framework to obtain rigorous upper bounds on the statistical dispersion measures discussed in the previous section. We will provide a high-level summary of our framework in this section, and leave detailed derivations and most examples to the
appendix. Our discussion will focus on quantities related to the inverse of CDFs, but similar results could be obtained for CDFs.
In short, our framework involves two steps: produce upper and lower bounds on the CDF (and thus inverse CDF) of the loss distribution, and use these to calculate bounds on a chosen target risk measure. First, we will describe our extension of the one-sided bounds in [30] to the two-sided bounds necessary to control many societal dispersion measures of interest. Then we will describe how these CDF bounds can be post-processed to provide control on risk measures defined by nonlinear functions and functionals of the CDF. Finally, we will offer a novel optimization method for tightening the bounds for a chosen, possibly complex, risk measure.
4.1 Methods to obtain confidence two-sided bounds for CDFs
For loss values  $\{X_i\}_{i=1}^n$ , let  $X_{(1)} \leq \ldots \leq X_{(n)}$  denote the corresponding order statistics. For the uniform distribution over  $[0,1]$ , i.e.  $\mathcal{U}(0,1)$ , let  $U_1, \ldots, U_n \sim^{iid} \mathcal{U}(0,1)$  denote the corresponding order statistics  $U_{(1)} \leq \ldots \leq U_{(n)}$ . We will also make use of the following:
Proposition 1. For the CDF  $F$  of  $X$ , if there exists two CDFs  $F_U, F_L$  such that  $F_U \succeq F \succeq F_L^2$ , then we have  $F_L^- \succeq F^- \succeq F_U^-$ .
We use  $(\hat{F}_{n,L}^{\delta},\hat{F}_{n,U}^{\delta})$  to denote a  $(1 - \delta)$ -confidence bound pair  $((1 - \delta)$ -CBP), which satisfies  $\mathbb{P}(\hat{F}_{n,U}^{\delta}\succeq F\succeq \hat{F}_{n,L}^{\delta})\geq 1 - \delta$ .
We extend the techniques developed in [30], wherein one-sided (lower) confidence bounds on the uniform order statistics are used to bound  $F$ . This is done by considering a one-sided minimum goodness-of-fit (GoF) statistic of the following form:  $S \coloneqq \min_{1 \leq i \leq n} s_i(U_{(i)})$ , where  $s_1, \ldots, s_n : [0,1] \to \mathbb{R}$  are right continuous monotone nondecreasing functions. Thus,  $\mathbb{P}(\forall i : F(X_{(i)}) \geq s_i^- (s_\delta)) \geq 1 - \delta$ , for  $s_\delta = \inf_r \{r : \mathbb{P}(S \geq r) \geq 1 - \delta\}$ . Given this step function defined by  $s_1, \ldots, s_n$ , it is easy to construct  $\hat{F}_{n,L}^\delta$  via conservative completion of the CDF. [30] found that a Berk-Jones bound could be used to choose appropriate  $s_i$ 's, and is typically much tighter than using the Dvoretzky-Kiefer-Wolffowitz (DKW) inequality to construct a bound.
4.1.1 A reduction approach to constructing upper bounds of CDFs
Now we show how we can leverage this approach to produce two-sided bounds. In the following lemma we show how a CDF upper bound can be reduced to constructing lower bounds.
Lemma 1. For  $0 \leq L_{1} \leq L_{2} \cdots \leq L_{n} \leq 1$ , if  $\mathbb{P}(\forall i: F(X_{(i)}) \geq L_{i}) \geq 1 - \delta$ , then, we have
$$
\mathbb {P} (\forall i: \lim  _ {\epsilon \rightarrow 0 ^ {+}} F (X _ {(i)} - \epsilon) \leq 1 - L _ {n - i + 1}) \geq 1 - \delta .
$$
Furthermore, let  $R(x)$  be defined as  $1 - L_{n}$  if  $x < X_{(1)}$ ;  $1 - L_{n - i + 1}$  if  $X_{(i)} \leq x < X_{(i + 1)}$  for  $i \in \{1,2,\dots ,n - 1\}$ ;  $1$  if  $X_{(n)} \leq x$ . Then,  $F \preceq R$ .
Thus, we can simultaneously obtain  $(\hat{F}_{n,L}^{\delta},\hat{F}_{n,U}^{\delta})$  by setting  $L_{i} = s_{i}^{-}(s_{\delta})$  and applying (different) CDF conservative completions. In practice, the CDF upper bound can be produced via post-processing of the lower bound. One clear advantage of this approach is that it avoids the need to independently produce a pair of bounds where each bound must hold with probability  $1 - \delta /2$ .
4.2 Controlling statistical dispersion measures
Having described how to obtain the CDF upper and lower bounds  $(\hat{F}_{n,L}^{\delta},\hat{F}_{n,U}^{\delta})$  , we next turn to using these to control various important risk measures such as the Gini coefficient and group differences. We will only provide high-level descriptions here and leave details to the appendix.
4.2.1 Control of nonlinear functions of CDFs
First we consider bounding  $\xi(F^{-})$ , which maps  $F^{-}$  to another function of  $\mathbb{R}$ .
2We also loosely consider a non-negative increasing function  $\tilde{F}$  such that  $\lim_{x\to \infty}\tilde{F} (x) = 1$  , but  $\tilde{F} (x) <   1$  for any real value  $x$  , as a CDF.
Control for a monotonic function. We start with the simplest case, where  $\xi$  is a monotonic function in the range of  $X$ . For example, if  $\xi$  is an increasing function, and with probability at least  $1 - \delta$ ,  $\hat{F}_{n,U}^{\delta} \succeq F \succeq \hat{F}_{n,L}^{\delta}$ ; then further by Proposition 1, we have that  $\xi(\hat{F}_{n,L}^{\delta,-}) \succeq \xi(\hat{F}^{-}) \succeq \xi(\hat{F}_{n,U}^{\delta,-})$  holds with probability at least  $1 - \delta$ . This property could be utilized to provide bounds for the Gini coefficient or Atkinson index by controlling the numerator and denominator separately as integrals of monotonic functions of  $F^{-}$ .
Example 1 (Gini coefficient). If given a  $(1 - \delta)$ -CBP  $(\hat{F}_{n,L}^{\delta},\hat{F}_{n,U}^{\delta})$  and  $\hat{F}_{n,L}^{\delta}\succeq 0^{3}$ , we can provide the following bound for the Gini coefficient. Notice that
$$
\mathcal {G} (X) = \frac {\int_ {0} ^ {1} (2 p - 1) F ^ {-} (p) d p}{\int_ {0} ^ {1} F ^ {-} (p) d p} = \frac {\int_ {0} ^ {1} 2 p F ^ {-} (p) d p}{\int_ {0} ^ {1} F ^ {-} (p) d p} - 1.
$$
Given  $F^{-}(p) \geq 0$  (since we only consider non-negative losses, i.e.  $X$  is always non-negative), we know
$$
\mathcal {G} (X) \leq \frac {\int_ {0} ^ {1} 2 p \hat {F} _ {n , L} ^ {\delta , -} (p) d p}{\int_ {0} ^ {1} \hat {F} _ {n , U} ^ {\delta , -} (p) d p} - 1,
$$
Control for absolute and polynomial functions. Many societal dispersion measures involve absolute-value functions, e.g., the Hoover index or maximum group differences. We must also control polynomial functions of inverse CDFs, such as in the CDFs of extreme observations. For any polynomial function  $\phi(s) = \sum_{k=0} \alpha_k s^k$ , if  $k$  is odd,  $s^k$  is monotonic w.r.t.  $s$ ; if  $k$  is even,  $s^k = |s|^k$ . Thus, we can group  $\alpha_k s^k$  according to the sign of  $\alpha_k$  and whether  $k$  is even or odd, and flexibly use the upper and lower bounds already established for the absolute value function and monotonic functions to obtain an overall upper bound.
Example 2. If we have  $(T_L^\delta (F_g),T_U^\delta (F_g))$  such that  $T_{L}^{\delta}(F_{g})\leq T(F_{g})\leq T_{U}^{\delta}(F_{g})$  holds for all  $g$  we consider, then we can provide high probability upper bounds for
$$
\xi \left(T \left(F _ {g _ {1}}\right) - T \left(F _ {g _ {2}}\right)\right)
$$
for any polynomial functions or the absolute function  $\xi$ . For example, with probability at least  $1 - \delta$
$$
\left| T \left(F _ {g _ {1}}\right) - T \left(F _ {g _ {2}}\right) \right| \leq \max  \left\{\left| T _ {U} ^ {\delta} \left(F _ {g _ {1}}\right) - T _ {L} ^ {\delta} \left(F _ {g _ {2}}\right) \right|, \left| T _ {L} ^ {\delta} \left(F _ {g _ {1}}\right) - T _ {U} ^ {\delta} \left(F _ {g _ {2}}\right) \right| \right\}.
$$
Control for a general function. To handle general nonlinearities, we introduce the class of functions of bounded total variation. Roughly speaking, if a function is of bounded total variation on an interval, it means that its range is bounded on that interval. This is a very rich class including all continuously differentiable or Lipschitz continuous functions. The following theorem shows that such functions can always be decomposed into two monotonic functions.