SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Weiwei Kong
weiweikong@google.com
Andres Muñoz Medina
ammedina@google.com
A well-known numerical bottleneck in the differentially-private stochastic gradient descent (DP-SGD) algorithm is the computation of the gradient norm for each example in a large input batch. When the loss function in DP-SGD consists of an intermediate linear operation, existing methods in the literature have proposed decompositions of gradients that are amenable to fast norm computations. In this paper, we present a framework that generalizes the above approach to arbitrary (possibly nonlinear) intermediate operations. Moreover, we show that for certain operations, such as fully-connected and embedding layer computations, further improvements to the runtime and storage costs of existing decompositions can be deduced using certain components of our framework. Finally, preliminary numerical experiments are given to demonstrate the substantial effects of the aforementioned improvements.
Machine learning models — more specifically, neural network-based models — are becoming more popular in industrial applications, in user-facing products, and different scientific fields. The popularity of these models lies on their flexibility and ability to be trained on ever bigger datasets, which may contain personal information about individuals. As these models become bigger and more descriptive, ML practitioners need to ensure that the models, and their black-box interactions, do not reveal information about the data used to train the model. In fact, it has been shown repeatedly [18] that large neural-network models can be used to know if a particular example was used in the training data. Another line of attacks has demonstrated that one can actually reconstruct some training instances with simple interactions with a trained model [3].
The only known robust way of protecting against these attacks is to train models using differential privacy [10, 11]. Using this approach, ML practitioners provide an information theoretic guarantee that ensures that the final model does not depend on any individual example<sup>1</sup>. To date, the most popular method for training models with differential privacy is the differentially private stochastic gradient descent (DP-SGD) [1] method. In theory, the DP-SGD algorithm requires only minimal changes with respect to its non-private counterpart; one only requires to clip gradients observed in the training process and add some noise proportional to the clipping value. However, in practice, the (naive) gradient clipping step has been shown to increase memory and computational costs in all popular learning platforms (JAX, Tensorflow, and Pytorch). More precisely, for a batch of size  $n$  a naive implementation of DP-SGD requires calculating  $n$  gradients (one for each example in the batch) so they can be clipped. This is in stark contrast with most back-propagation-based training algorithms which calculate a single gradient. This implies a dependency on the runtime and memory in proportion to the batch size used to train the model, and for large models, this cost makes the
<sup>1</sup>In particular, it is impossible to distinguish whether or not an individual example is in the dataset, let alone reconstruct it.
prospect of training with differential privacy simply not viable. Several papers [8, 17, 21] have also observed that large models need to be trained with large batch sizes to obtain competitive levels of utility, under reasonable amounts of privacy.
In [12], it was recognized that one could clip gradients without actually materializing every example gradient. This technique is referred to as ghost clipping. The initial proposal from [12] was specialized to fully connected, feed-forward neural networks that consist of only dense layers. The ghost clipping algorithm was later extended to handle convolution layers [13, 15] and attention layers [14].
However, each of the above solutions required an ad-hoc analysis on the correctness of their implementation. In this work we present a general analysis of the ghost clipping algorithm. To produce our results, we study this algorithm from the lens of linear operator theory. Our paper has three main contributions:
1. We unify years of ad-hoc analysis and interpretations of the ghost clipping algorithm under a single framework.
2. We provide a future-proof way of expanding the ghost-clipping technique to new layers with (possibly) nonlinear dependencies on their weights.
3. We demonstrate that by framing the ghost clipping problem in the language of linear operators, we can obtain a better performance of DP-SGD on models with embedding layers (crucial for personalization models consisting of embeddings of tens of millions of parameters) and fully connected layers with linear bias broadcasting operators.
Additionally, we show in our Appendix how to apply our framework to more complex tranforms/layers such as layer normalization and multi-head attention.
To complement the results of this paper, we open-sourced the general interface of the code using the TensorFlow Keras API<sup>2</sup>. By introducing an abstract interface, we also expect practitioners to easily extend the ghost clipping algorithm to any type of layer.
2 Notation and preliminaries
Throughout the paper,  $(\mathcal{W},\langle \cdot ,\cdot \rangle)$  denotes a Hilbert space and  $\| \cdot \|$  denotes its induced norm. Examples of  $\mathcal{W}$  are  $\mathbb{R}^d$  with the standard dot product and the space of matrices  $\mathcal{W} = \mathbb{R}^{p\times q}$  with the Euclidean (Frobenius) inner product between  $A,B\in \mathcal{W}$  given by  $\langle A,B\rangle = \mathrm{tr}(A^\top B)$ . For two matrices  $A,B$  we let  $\| A\|$  denote the Frobenius norm of  $A$ .
Given two Hilbert spaces  $(\mathcal{W},\langle \cdot ,\cdot \rangle_{\mathcal{W}})$ $(\mathcal{V},\langle \cdot ,\cdot \rangle_{\mathcal{V}})$ , we denote linear operators between them by italicized letters  $(A\colon \mathcal{W}\to \mathcal{V})$  and  $A^{*}:\mathcal{V}\rightarrow \mathcal{W}$  to be the adjoint of  $\mathcal{A}$ . That is,  $A^{*}$  is the unique linear operator that satisfies
$$
\langle y, \mathcal {A} w \rangle_ {\mathcal {Y}} = \left\langle \mathcal {A} ^ {*} y, w \right\rangle_ {\mathcal {W}} \quad \forall w \in \mathcal {W}, \quad \forall y \in \mathcal {Y}. \tag {1}
$$
Let  $(\mathcal{W},\langle \cdot ,\cdot \rangle_{\mathcal{W}})$  and  $(\mathcal{V},\langle \cdot ,\cdot \rangle_{\mathcal{V}})$  denote two Hilbert spaces with respective induced norms  $\| \cdot \|_{\mathcal{W}}$  and  $\| \cdot \|_{\mathcal{Y}}$ . Moreover, let  $\psi :\mathcal{W}\to \mathcal{V}$  be an arbitrary function. The Fréchet derivative of  $\psi$  at  $w_0\in \mathcal{W}$  is given by the unique bounded linear operator  $D\psi (w_0)\colon \mathcal{W}\to \mathcal{V}$  satisfying
$$
\lim _ {\delta \to 0} \frac {\| \psi (w _ {0} + \delta) - \psi (w _ {0}) - D \psi (w _ {0}) \delta \| _ {\mathcal {Y}}}{\| \delta \| _ {\mathcal {W}}} = 0.
$$
We say  $\psi$  is differentiable if its Fréchet derivative exists for all  $w_0\in \mathcal{W}$ . Throughout this paper we will use two special properties of the Fréchet derivative: the chain rule and the existence of gradients. Let  $(\mathcal{Z},\langle \cdot ,\cdot \rangle_{\mathcal{Z}})$  be another Hilbert space and  $\phi \colon \mathcal{Y}\to \mathcal{Z}$  be given. The chain rule provides us with a simple way to calculate the derivative of the function  $\phi \circ \psi \colon \mathcal{W}\rightarrow \mathcal{Z}$ , namely,
$$
D (\phi \circ \psi) (w _ {0}) = D \phi (\psi (w _ {0})) D \psi (w _ {0}).
$$
The Fréchet derivative of  $\psi$  at  $w_0$  with respect to a subset of variables  $u$  is denoted by  $D_u\phi(w_0)$ . Finally,  $\nabla \psi(w_0) \in \mathcal{W}$  denotes the (unique) gradient of a function  $\psi$  at  $w_0$ , which satisfies
$$
D \psi \left(w _ {0}\right) \delta = \left\langle \nabla \psi \left(w _ {0}\right), \delta \right\rangle_ {\mathcal {W}} \quad \forall \delta \in \mathcal {W} \tag {2}
$$
The existence of the gradient is guaranteed by the well-known Riesz-Fréchet Representation Theorem [16]. The gradient of  $\psi$  at  $w_{0}$  with respect to a set of variables  $u$  is denoted by  $\nabla_u\psi (w_0)$ .
See the repo in https://github.com/google-research/google-research/tree/master/fast-gradient_clipping
Algorithm 1 DP-SGD algorithm
Input: Data sample  $S$ , parameter initialization  $w_0$ , number of rounds  $T$ , learning rate  $\eta$ , noise multiplier  $\sigma$ , clipping norm  $C$ .  
Set  $w = w_0$   
for  $t = 1, \ldots, T$  do  
    Get a batch of inputs  $B \subset S$   
Calculate per example gradients  $g_x = \nabla_w \ell(x, w)$  for every  $x \in B$   
Clip gradients  $\widetilde{g}_x = \min \left\{\frac{C}{\|g_x\|}, 1\right\} g_x$   
Calculate private batch gradient  $\widetilde{g} = \frac{1}{|B|} \sum_{x \in B} \widetilde{g}_x + N\left(0, \frac{C^2 \sigma^2}{|B|^2} I\right)$   
Update model  $w = w - \eta \widetilde{g}$   
end for  
return  $w$
Private Stochastic Gradient Descent We now turn our attention to the DP-SGD algorithm and specifically, to its instantiation for neural network-based models. Let  $\mathcal{X}$  be an arbitrary space and  $\mathcal{S} = \{x_1,\ldots ,x_n\} \subset \mathcal{X}$  be a sample of examples. Let  $\mathcal{W}$  denote a parameter space and  $h\colon \mathcal{X}\times$ $\mathcal{W}\to \mathbb{R}$  denote an arbitrary function. The stochastic gradient descent (SGD) algorithm solves the optimization problem
$$
\min  _ {w \in \mathcal {W}} \frac {1}{n} \sum_ {i = 1} ^ {n} h (x _ {i}, w)
$$
by iteratively updating the model parameters using gradients of the loss over a batch  $B \subset S$  of data.
The celebrated DP-SGD algorithm was introduced by [1] as a simple modification on the SGD algorithm to make it private. Specifically, the DP-SGD algorithm, given in Algorithm 1, is identical to the SGD algorithm except in two steps:
1. The DP-SGD algorithm needs to calculate  $|B|$  (so-called) per-example gradients in order to clip them to have a bounded norm.
2. Adding Gaussian noise  $N(0,C^2\sigma^2 /|B|^2)$  for some noise multiplier  $\sigma >0$
Notice that the first step has a prohibitively large cost for large networks. Specifically, in the per example gradient calculation, notice that the computational and memory usage of a naive implementation<sup>4</sup> of DP-SGD algorithm increases as  $\Theta (n_w|B|)$ , where  $n_w$  is the number of parameters in the network. This increase in resources effectively negates the advantages of the backpropagation algorithm. In this paper we show how to run the DP-SGD algorithm with only a small constant increment in both the memory footprint and runtime of traditional SGD.
We will focus on the scenario where the learner is trying to minimize the loss across the space of neural networks. That is, the parameter vector  $w$  is a concatenation of  $k$  parameters  $(w_{1},\ldots ,w_{k})$  and there exists  $k$  functions  $\phi_1,\dots ,\phi_k$ , and a loss function  $\ell$  such that
$$
h (x, w) = \ell \left(x, \phi_ {k} \left(w _ {k}, \phi_ {k - 1} \left(w _ {k - 1}, \dots , \phi_ {1} \left(w _ {1}, x\right)\right)\right). \right.
$$
Note that  $\{\phi_j\}$  correspond to the layers of the network and  $\{w_j\}$  are the vectors parameterizing these layers.
3 Previous work
The DP-SGD algorithm was first introduced by [1]. Due to its simplicity to adapt to standard machine learning frameworks it is now probably the most popular method for training private
<sup>3</sup>When talking about privacy we are referring to the concept of differential privacy. We refer the reader to [11] for a comprehensive guide on differential privacy. In this work we focus on the computational aspects and not the privacy properties of the DP-SGD algorithm
4 Alternatively, one could consider sequentially computing the gradient norms to remove the linear memory scaling in  $|B|$ . However, this approach may prohibitively increase the runtime cost in practice.
machine learning models. In the past few years, a large number of papers have been devoted to improving the privacy-utility trade-offs of DP-SGD [4] as well as demonstrating that DP-SGD can be used on multiple tasks [7, 9, 2]. While the main focus on this line of research has been in understanding the privacy-utility trade-offs, some authors did touch on the performance issues of calculating per-example gradients. A simple solution to reduce the memory and computational blow-up of the DP-SGD algorithm is to calculate the gradient with respect to a micro-batch of examples instead. While this reduced the computational cost of running DP-SGD, it came at the cost of decreased quality of the model learned. Some authors [2] got around this performance issues by using some JIT compilation features of JAX. However these authors still observed an increase in the memory use of their private implementations.
In order to speed up the gradient clipping step without sacrificing quality, [12] proposed the ghost clipping technique (see Section 4). ITs general idea is that one can obtain a private gradient estimate  $\tilde{g}$  without materializing each per example gradient as long as one knows the norm of each per-example gradient. Moreover, the authors show that one can easily calculate these norms by using information already materialized in the forward and backward passes of the back propagation algorithm for training neural networks. The results however were limited to neural networks consisting of only dense layers. The ghost clipping trick was later on applied to train transformers [14] and image classification models which consist of networks with convolution layers [15].
The above works provide ad-hoc proofs that their implementation of the ghost clipping algorithm is correct for their particular layer implementation but do not attempt to extend their results to arbitrary layers. In this work we present a way to extend the ghost clipping technique to arbitrary layers and pinpoint what properties of each layer are crucial for enabling an efficient implementation of the DP-SGD algorithm.
Finally, it is also worth mentioning that paper [6] presents an efficient implementation of the ghost clipping technique in PyTorch for models with fully-connected, embedding, convolution, and normalization layers using variable caching to avoid recomputing certain intermediate parameter gradients. Our implementation also takes advantage of variable caching, but does this implicitly through TensorFlow's GradientTape API<sup>5</sup>.
4 The Ghost Clipping Algorithm
Algorithm 1 suggests that in order to implement DP-SGD one must calculate every per-example gradient, clip it to achieve a bounded norm and then aggregate it to obtain the private gradient estimate  $\widetilde{g}$ . The main observation from [12] was that, with knowledge of the norms  $\| g\| _x$  one could estimate  $\widetilde{g}$  without explicitly materializing each per example gradient. To do this, the authors define weights  $r_x = \min \left\{C / \| g_x\| ,1\right\}$  and a new weighted sum
$$
S (w) = \sum_ {x \in B} r _ {x} h (x, w).
$$
Treating the weights  $r_x$  as fixed (with respect to the parameters  $w$ ), one can use the linearity of the gradient operator to see that  $\nabla S(w) = \widetilde{g}$ . Crucially, for neural networks, the gradient of  $S$  can be efficiently calculated using a single back-propagation step. The second contribution of [12] was showing that, for neural networks consisting of dense layers, one can also efficiently calculate  $\| g_x \|$  for every  $x \in B$  using only a single forward and backward pass of the back propagation algorithm. That is, the cost of calculating a private gradient is simply twice that of calculating a non-private gradient step.
In the subsections below, we present a general formulation for calculating  $\| g_x\|$  in a single forward and backward pass for arbitrary feed-forward neural networks, under the assumption that:
$\triangleright \ell (x,\cdot),\phi_1(\cdot ,x),\ldots ,\phi_k(\cdot ,x)$  are Fréchet differentiable for every  $x\in \mathcal{X}$
$\triangleright$  each function  $\phi_i(\cdot, x)$  can be decomposed as the composition of at least two Fréchet differentiable subfunctions.
See https://www.tensorflow.org/api_docs/python/tf/GradientTape
Algorithm 2 General Gradient Norm Framework
Input: Data  $x\in \mathcal{X}$ , parameter  $w = (w_{1},\ldots ,w_{k})$ , and functions  $\{\psi_{x,i}\}_{i = 1}^{k}$ ,  $\{Z_{x,i}\}_{i = 1}^{k}$  satisfying
$$
h (x, \tilde {w}) = \ell_ {x} \circ \psi_ {x, i} \circ Z _ {x, i} (\tilde {w} _ {i}), \quad i = 1, \dots , k \tag {5}
$$
for any  $\tilde{w} = (\tilde{w}_1,\dots ,\tilde{w}_k)\in \mathcal{W}$ , where  $\ell_{x}(\cdot)\equiv \ell (x,\cdot)$ .
for  $i = 1,\dots ,k$  do
Compute  $\zeta_{x,i} \coloneqq Z_{x,i}(w_i)$
Store a "nice" representation  $\Omega_{x,i}(\cdot)$  of the squared-norm function  $g\mapsto \| [DZ_{x,i}(w_i)]^* (g)\| ^2$
for any gradient  $g$  of the function  $\ell_{x}\circ \psi_{x,i}(\cdot)$
Compute  $g_{x,i} \coloneqq \nabla_{\zeta_{i,x}}(\ell_x \circ \psi_{x,i})(\zeta_{x,i})$  and  $\tau_{x,i} \coloneqq \Omega_{x,i}(g_{x,i})$
return  $(\sum_{i = 1}^{k}\tau_{x,i})^{1 / 2}$
4.1 The General Norm Calculation Algorithm
Let  $w = (w_{1},\ldots ,w_{k})$  parameterize a neural network. We first note that  $\| \nabla_w h(x,w)\| ^2 = \sum_{j = 1}^k\| \nabla_{w_k}h(x,w)\| ^2$ . Hence, we can focus on efficiently calculating the norm of the gradient corresponding to the parameters of each layer. Let us then fix a layer and denote  $\bar{w}\in \bar{\mathcal{W}}$  to be its parameter vector for some restricted space  $\bar{\mathcal{W}}$ . Moreover, let the input  $x$  be fixed. We see that, as a function of the layer parameters  $\bar{w}$  only, the loss  $h$  has the form
$$
h (x, \bar {w}) = \ell_ {x} \circ \phi_ {x} (\bar {w}) \quad \forall x \in \mathcal {X}, \tag {3}
$$
where  $\phi_x$  corresponds to operations performed by the fixed layer and all subsequent layers in the network, and  $\ell_x(\cdot) \equiv \ell(x, \cdot)$  corresponds to the loss function, e.g., mean-squared error, for the input example  $x$ . The following result, whose proof can be found in the Appendix, gives an expression for the gradient of  $h$  under any decomposition of  $\phi_x$ .
Proposition 4.1. Let  $x \in \mathcal{X}$  be fixed and let  $(\ell_x, \phi_x)$  be a pair of Fréchet differentiable functions satisfying (3). Moreover, let  $(\psi_x, Z_x)$  be a pair of Fréchet differentiable functions satisfying  $\phi_x = \psi_x \circ Z_x$ , and denote
$$
\mathcal {A} = \mathcal {A} _ {x} (\bar {w}) := D Z _ {x} (\bar {w}), \quad g = g _ {x} (\bar {w}) := \nabla \left(\ell_ {x} \circ \psi_ {x}\right) \left(Z _ {x} (\bar {w})\right). \tag {4}
$$
Then,  $\nabla_{\bar{w}}h(x,\bar{w}) = A^{*}g$
The above result gives us an alternative way of computing  $\| \nabla_{\bar{w}}h(x,\bar{w})\| ^2$ , namely, (i) pick a decomposition  $(\psi_x,Z_x)$  of  $\phi_{x}$  and (ii) compute  $\| \mathcal{A}^* g\|^2$  in some efficient manner. In later subsections, we provide examples of layers and decompositions where this two-step approach is drastically more efficient than the naive approach of materializing  $\nabla_{\bar{w}}h(x,\bar{w})$  and computing its Euclidean norm.
In view of the decomposition in Proposition 4.1, we present a unified method for computing  $\| \nabla h(x,w)\|$  in Algorithm 2. Specifically, it consists of two loops over the parameters  $w_{i}$ . The initial loop is performed to obtain the intermediate outputs  $\zeta_{i,x}$  and some sort of "nice" or efficient representation of the squared-norm functions  $\Omega_{i,x}$ . The follow-up loop, also over the parameters  $w_{i}$ , computes some necessary intermediate gradients and combines them with the functions  $\Omega_{i,x}$  to obtain the gradient decomposition (and subsequent norm computation) given by Proposition 4.1.
4.2 Efficient Implementation of Algorithm 2
We now describe how Algorithm 2 can be implemented efficiently when  $h(\cdot, \cdot)$  is formed by a neural network  $\mathcal{N}$ . First, it is shown in Appendix B.1 that the intermediate quantities  $\{\zeta_{i,x}\}_{i=1}^{k}$  and  $\{g_{i,x}\}_{i=1}^{k}$  can be computed in a single forward and backward pass of  $\mathcal{N}$  (as opposed to naively traversing the network  $\Theta(k)$  times). Second, it is shown in Appendix B.2 that for any  $i$ , the batch gradients  $\{\nabla_{\zeta_{x,i}}(\ell_x \circ \psi_{x,i})(\zeta_{x,i})\}_{x \in B}$  can be obtained a single (batched) backward pass of  $\mathcal{N}$  if (i) condition (5) holds for every  $x \in B$  and (ii) the gradient  $g_{x,i}$  is replaced by  $\nabla_{\zeta_{x,i}}\{\sum_{x \in B} \ell_x \circ \psi_{x,i}(\zeta_{x,i})\}$ . Consequently, if we ignore the costs of forming and evaluating  $\Omega_{x,i}(\cdot)$  for each  $x \in B$
and  $i = 1,\ldots ,k$ , then the above facts imply that we can obtain the norms  $\{\| \nabla h(x,w)\| \}_{x\in B}$  using only one additional forward and (batched) backward pass of  $\mathcal{N}$ .
5 Efficient Squared-Norm Functions
This section presents efficient representations of the functions  $\Omega_i(\cdot)$  in Algorithm 2 for some basic layer functions  $\phi_x(w) \equiv \phi(w, x)$ . Examples involving more complicated layer functions (e.g., layer normalization and multi-head attention) can be found in the Appendix.
Each subsection begins with a precise description of  $\phi_x(\cdot)$ , presents a decomposition  $\phi(\cdot) = \psi_x \circ Z_x(\cdot)$ , and gives an analysis of the runtime and storage costs of an efficient representation of the function  $\Omega_x(\cdot) \equiv \|[DZ_x(w)]^*(\cdot)\|^2$ . For the sake of conciseness, the proofs of these decompositions are given in the Appendix.
In addition, we make comparisons of our approach with the naive approach (of computing gradients for each example) and other ghost clipping-like approaches. The results of these comparisons are summarized in Tables 1-2 (see the first paragraphs in Subsections 5.1-5.3 for descriptions of the variables/dimensions).
Naive	Ghost Clipping	Ours
Fully-Connected6	Θ(|B|{pq+m})	Θ(r2+|B|{rq}2)	Θ(|B|r2)
Embedding7	Θ(|B|qd)	O(|B|q2)	Θ(|B|q)
Rank-k Approx.	Θ(|B|nk)	-	O(1)
Table 1: Asymptotic storage costs for computing  $\{\|\nabla h(x,W)\|_{x\in B}\}$ . For ghost clipping and our approach, this is the storage cost of representing the squared-norm function  $\Omega_x(\cdot)$  on the entire batch  $B$ .
Naive	Ghost Clipping	Ours
Fully-Connected6	Θ(|B|{pq+m})	Θ(r2{p+|B|q2}+m{rq2})	Θ(r2{p+|B|q})
Embedding7	Θ(|B|qd)	O(|B|q2d)	Θ(|B|{q log q̂+ q̂d})
Rank-k Approx.	Θ(|B|n2k)	-	Θ(n2k+|B|)
Table 2: Asymptotic runtime costs for computing  $\{\| \nabla h(x,W)\| \}_{x\in B}$ . For ghost clipping and our approach, this includes the time used to generate, represent, and evaluate the squared-norm function  $\Omega_{x}(\cdot)$  on the entire batch  $B$ .
5.1 Fully-Connected Layers
Given variables  $V \in \mathbb{R}^{p \times q}$  and  $b \in \mathbb{R}^m$ , a layer input  $U_x \in \mathbb{R}^{r \times p}$ , an activation function  $\alpha: \mathbb{R}^{r \times q} \mapsto \mathbb{R}^{r \times q}$ , and a linear broadcasting operator  $\mathcal{Q}: \mathbb{R}^m \mapsto \mathbb{R}^{r \times q}$  satisfying  $m \mid rq$ , the standard fully-connected layer function  $\phi_x(\cdot)$  is given by
$$
\phi_ {x} (V, b) = \alpha \left(U _ {x} V + \mathcal {Q} b\right). \tag {6}
$$
Typically,  $p, q, r$ , and  $m$  are called the input dimension, output dimension, channel dimension, and bias dimension, respectively. Usually, it is the case that  $r \ll \min \{p, q\}$  and a common case is  $r = 1$ .
We now consider the squared-norm function  $\Omega_x: \mathbb{R}^{r \times q} \mapsto \mathbb{R}$  generated by the choice of  $Z_x(V, b) \coloneqq U_x V + \mathcal{Q}b$ . Denoting  $\mathcal{A}$  as in (4), for some  $\bar{w} = (V, b)$ , we have that
$$
\Omega_ {x} (g) = \left\| \mathcal {A} ^ {*} g \right\| ^ {2} = \left\| U _ {x} ^ {*} g \right\| ^ {2} + \left\| \mathcal {Q} ^ {*} g \right\| ^ {2} = \left\langle U _ {x} U _ {x} ^ {*}, g g ^ {*} \right\rangle + \left\| \mathcal {Q} ^ {*} g \right\| ^ {2}, \tag {7}
$$