SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We decided to interpolate between the "dense" and "2-pair" interaction structures using a two-stage procedure with real parameters  $\alpha$  and  $\lambda$  and the number of strongly interacting components  $K$ . While the precise description is in Appendix D.2, we can interpret  $\alpha$  as controlling the baseline strength of interaction between every pair of variables in  $\{X_1,\ldots ,X_{10},Y_1,\ldots Y_{10}\}$  and  $\lambda$  as the additional strength of interaction between pairs of variables  $(X_{1},Y_{1}),\dots,(X_{K},Y_{K})$ . Dense interaction structure has  $\lambda = 0$  and 2-pair interaction structure has  $\alpha = 0$  and  $K = 2$ . First, we set  $K = 10$  and  $\lambda \approx 0$  and decrease  $\alpha$  raising  $\lambda$  at the same time to maintain constant MI of 1 nat. When  $\alpha = 0$ , whole information is contained in the pairwise interactions  $(X_{1},Y_{1}),\ldots ,(X_{10},Y_{10})$ . We then decrease  $K$  and increase  $\lambda$ , still maintaining constant MI.
In Fig. 3 we observe that the performance of all estimators considered (apart from CCA, a model-based approach suitable for multivariate normal distributions) degrades when the interactions between
Figure 3: Two-stage interpolation between dense and sparse matrices. From left to right the sparsity of interactions increases. Shaded regions represent the sample standard deviation.
Figure 4: MI estimates as a function of the information contained in the tail of the distribution. Left: lengthening the tails via  $x \mapsto |x|^k$  sign  $x$  mapping with changing  $k$ . Right: increasing the degrees of freedom in multivariate Student distribution shortens the tails. Shaded regions represent the sample standard deviation.
every pair of variables become sparser (lower  $\alpha$ ). In particular, even neural estimators are not able to model the information in ten interacting pairs of variables. However, when we decrease the number of interacting pairs of variables, the performance of the neighborhood-based KSG estimator is qualitatively different from the neural estimators: performance of KSG steadily deteriorates, while neural estimators can find (some of the) relevant pairs of variables.
This motivates us to conclude that in the settings where considered variables are high-dimensional and the interaction structure is sparse, neural estimators can offer an advantage over neighborhood-based approaches.
4.2 Long-Tailed Distributions
In Fig. 4 we investigate two different ways to lengthen the tails. First, we lengthen the tails of a multivariate normal distribution using the mapping  $x \mapsto |x|^k$  sign  $x$  (with  $k \geq 1$ ) applied to each dimension separately. In this case, we see that the performance of CCA, KSG, and neural estimators is near-perfect for  $k$  close to 1 (when the distribution  $P_{XY}$  is close to multivariate normal), but the performance degrades as the exponent increases. Second, we study multivariate Student distributions varying the degrees of freedom: the larger the degrees of freedom, the lower the information content in the tails of the distribution. Again, we see that the performance of CCA, KSG, and neural estimators is near-perfect for high degrees of freedom for which the distribution is approximately normal. For low degrees of freedom, these estimators significantly underestimate MI, with the exception of CCA which gives estimates with high variance, likely due to the correlation matrix being hard to estimate.
Overall, we see that long tails complicate the process of estimating MI. The Student distribution is particularly interesting, since even after the tails are removed (with asinh or preprocessing strategies described in Appendix E) the task remains challenging. We suspect that the MI might be contained in regions which are rarely sampled, making the MI hard to estimate. When neural estimators are used, pointwise mutual information learned by the critic does not match the ground truth (see Appendix E).
Hence, we believe that accurate estimation of MI in long-tailed distributions is an open problem. Practitioners expecting long-tailed distributions should approach this task with caution.
Figure 5: Left: Spiral diffeomorphism with different speed parameter  $v$ . Right: Performance of considered estimators for increasing speed.
4.3 Invariance to Diffeomorphisms
The benchmark results (Fig. 2) as well as the experiments with function  $x \mapsto |x|^k$  sign  $x$  lengthening tails (Fig. 4) suggest that even if ground-truth MI is preserved by a transformation (see Theorem 2.1), the finite-sample estimates may not be invariant. This can be demonstrated using the spiral diffeomorphism.
We consider isotropic multivariate normal variables  $X = (X_{1},X_{2})\sim \mathcal{N}(0,I_{2})$  and  $Y\sim \mathcal{N}(0,1)$  with  $\operatorname {Cor}(X_1,Y) = \rho$ . Then, we apply a spiral diffeomorphism  $s_v\colon \mathbb{R}^2\to \mathbb{R}^2$  to  $X$ , where
$$
s _ {v} (x) = \left( \begin{array}{c c} \cos v | | x | | ^ {2} & - \sin v | | x | | ^ {2} \\ \sin v | | x | | ^ {2} & \cos v | | x | | ^ {2} \end{array} \right)
$$
is the rotation around the origin by the angle  $v||x||^2$  (see Fig. 5; for a theoretical study of spiral diffeomorphisms see Appendix B). In Fig. 5, we present the estimates for  $N = 10000$ ,  $\rho = 0.8$  and varying speed  $v$ .
In general, the performance of the considered estimators deteriorates with stronger deformations. This is especially visible for the CCA estimator, as MI cannot be reliably estimated in terms of correlation.
4.4 High Mutual Information
Gao et al. [2015] proves that estimation of MI between strongly interacting variables requires large sample sizes. Poole et al. [2019], Song and Ermon [2020] investigate estimation of high MI using multivariate normal distributions in the setting when the ground-truth MI is changing over the training time. We decided to use our family of expressive distributions to see the practical limitations of estimating MI. We chose three one-parameter distribution families: the  $3 \times 3$  multivariate normal distribution with correlation  $\rho = \operatorname{Cor}(X_1, Y_1) = \operatorname{Cor}(X_2, Y_2)$ , and its transformations by the half-cube mapping and the spiral diffeomorphism with  $v = 1/3$ . We changed the parameter to match the desired MI.
In Fig. 6 we see that for low enough MI neural estimators and KSG reliably estimate MI for all considered distributions, but the moment at which the estimation becomes inaccurate depends on the distribution. For example, for Gaussian distributions, neural estimators (and the simple CCA baseline) are accurate even up to 5 nats, while after the application of the spiral diffeomorphism, the estimates become inaccurate already at 2 nats.
We see that in general neural estimators perform well on distributions with high MI. The performance of CCA on the multivariate normal distribution suggests that well-specified model-based estimators may require lower number of samples to estimate high mutual information than model-free alternatives (cf. Gao et al. [2015]).
Existing Benchmarks Khan et al. [2007] used one-dimensional variables and an additive noise assumption to evaluate the estimators based on histograms, nearest neighbors, and kernels. However,
Figure 6: We select three one-parameter families of distributions varying the MI value ( $x$ -axis and dashed line). We use  $N = 10000$  data points and  $n = 5$  replicates of each experiment.
Table 1: Comparison between benchmarks.
Khan et al. [2007]	Poole et al. [2019]	Song and Ermon [2020]	Proposed Benchmark
Gaussian Distrib.	✓	✓	✓	✓
Long-Tailed Distrib.	(✓)	(✓)	✗	✓
Sparse/Dense Inter.	✗	✗	✗	✓
High MI	✗	(✓)	(✓)	✓
Invariance to Homeom.	✗	✗	✗	✓
Self-Consistency	✗	✗	✓	✗
Language Indep.	✗	✗	✗	✓
Code Available	✗	✗	✓	✓
the original code and samples are not available. Song and Ermon [2020] proposed to evaluate neural estimators based on variational lower bounds [Oord et al., 2018, Belghazi et al., 2018] on image data. However, besides the multivariate Gaussian which they also evaluate, the ground-truth MI is not available. Therefore, they proposed self-consistency checks, which include  $\mathbf{I}(X;Y) = 0$  for independent  $X$  and  $Y$ , and preservation of the data processing inequality. Poole et al. [2019] study neural estimators in settings employing Gaussian variables, affine transformation followed by axis-wise cubic transformation and investigate the high MI cases. In our benchmark, we focus on distributions with known MI and information-preserving continuous injective mappings such as homeomorphisms, evaluate long tail distributions, and compare representative estimators from different classes and platforms. We summarize the differences between the benchmarks in Table 1.
Causal Representation Learning and Nonlinear ICA Recent work on disentanglement and nonlinear independent component analysis (ICA) aims at identifying causal sources up to an ambiguous transformation [Xi and Bloem-Reddy, 2022]. Examples include orthogonal transformations [Zimmermann et al., 2021], permutations and diffeomorphisms applied along specific axes [Khemakhem et al., 2020], or the whole diffeomorphism group [Von Kugelgen et al., 2021, Daunhawer et al., 2023]. To check whether the learned representations indeed capture underlying causal variables, it is important to use estimators which are invariant to the ambiguous transformations specific to the model. Our results suggest that such requirements are not met by MI estimators and practitioners should carefully choose the used metrics, so they are not only theoretically invariant, but also can be robustly estimated in practice.
Venturing beyond the typical evaluation of mutual information estimators on multivariate normal distributions, we provided evidence that the evaluation on Gaussian distributions results in overly optimistic and biased results for existing estimators. In particular, a simple model-based approach as
$^{4}$ Khan et al. [2007] consider two one-dimensional r.v. related by additive noise, in which one of the r.v. is parametrized with a cubic function. This implicitly lengthens the tails, so we marked this with  $(\checkmark)$ ; similarly, Poole et al. [2019] consider cubic transformation of a multivariate normal variable. The experimental setup of estimation of MI considered by Poole et al. [2019] and Song and Ermon [2020] requires changing ground truth MI over training time of a neural estimator. This procedure differs from the usual approaches to estimating MI, so we marked it with  $(\checkmark)$ .
CCA is already sufficient if the distribution is (close to) multivariate normal, highlighting the effect of using available prior information about the problem to design efficient estimators. To provide a basis for more thorough evaluation, we designed a benchmark, compatible with any programming language, with forty general tasks focusing on selected problems, as estimation with sparse interactions, the effects of long tails, estimation in problems with high MI, and (only theoretical) invariance to diffeomorphisms.
Our results suggest that in low- and medium-dimensional problems the KSG estimator remains a solid baseline, requiring a lower number of samples than the alternatives and no parameter tuning. In high-dimensional settings where the interactions are sparse, neural estimators offer an advantage over classical estimators being able to more consistently pick up the signal hidden in the high-dimensional space. We further provide evidence that although MI is invariant to continuous injective transformations of variables, in practice the estimates can differ substantially. Appropriate data preprocessing may mitigate some of these issues, but no strategy clearly outperforms the others (see Appendix E).
We hope that the proposed benchmark encourages the development of new estimators, which address the challenges that we highlight in this paper. It can be used to assess invariance to homeomorphisms, understand an estimator's behavior in the situations involving distributions of different complexity, low-data behaviour, high mutual information or to diagnose problems with implementations. We believe that some parts of the benchmark could also be used to assess whether other estimators of other statistical quantities, such as kernel dependence tests, transfer entropy or representation similarity measures, can be reliably estimated on synthetic problems and if they are robust to selected classes of transformations.
Limitations and Open Problems Although the proposed benchmark covers a wide family of distributions, its main limitation is the possibility to only transform the distribution  $P_{XY}$  via transformations  $f \times g$  to  $P_{f(X)g(Y)}$  starting from multivariate normal and Student distributions. In particular, not every possible joint distribution is of this form and extending the family of distributions which have known MI and allow efficient sampling is the natural direction of continuation. We are, however, confident that due to the code design it will be easy to extend the benchmark with the new distributions (and their transformations) when they appear.
Estimation of information in multivariate Student distributions remains a challenge for most estimators. Although we provide the evidence that distributions with longer tails pose a harder challenge for considered estimators, shortening the tails with asinh transform did not alleviate all the issues. In neural estimators this can be partly explained by the fact that the critic may not learn to approximate pointwise mutual information (up to an additive constant) properly, as demonstrated in Appendix E, but the question of what makes the neighborhood-based estimator, KSG, fail, remains open.
As noted, unless strong inductive biases can be exploited (as illustrated for CCA), estimation of high MI is an important challenge. An interesting avenue towards developing new estimators can be by incorporating a stronger inductive bias, which could also take a form of development of better data normalization strategies and, in case of neural estimators, critic architectures. Better normalization strategies may make the estimators more invariant to considered transformations.
We would like to thank Craig Hamilton for the advice on scientific writing and help with the abstract. FG was supported by the Norwegian Financial Mechanism GRIEG-1 grant operated by Narodowe Centrum Nauki (National Science Centre, Poland) 2019/34/H/NZ6/00699. PC and AM were supported by a fellowship from the ETH AI Center.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pages 12979-12990. PMLR, 2021.
A Invariance to Continuous Injective Mappings 17
B Spiral Diffeomorphisms 18
C Overview of Estimators 20
D Additional Information on Benchmark Tasks 22
D.1 Task Descriptions 22
D.2 Covariance Matrix Family 23
D.3 Technical Results 24
E Additional Experiments 25
E.1 Pointwise Mutual Information 25
E.2 Sample Size Requirements 26
E.3 Preprocessing 27
E.4 Hyperparameter Selection 27
F Visualisation of Selected Tasks 29
G Author Contributions 30
A Invariance to Continuous Injective Mappings
In this section, we provide a proof for Theorem 2.1, formally proving the invariance of mutual information with respect to continuous injective mappings. First, we recall the standard definitions from Chapters 1 and 2 of Pinsker and Feinstein [1964].
Definition A.1. Let  $(\Omega, \mathcal{A})$  be a measurable space. A family of sets  $\mathcal{E} \subseteq \mathcal{A}$  is a partition of  $\Omega$  if for every two distinct  $E_1$  and  $E_2$  in  $\mathcal{E}$  it holds that  $E_1 \cap E_2 = \emptyset$  and that  $\bigcup \mathcal{E} = \Omega$ . If  $\mathcal{E}$  is finite, we call it a finite partition.
Definition A.2 (Mutual Information). Mutual information between r.v.  $X$  and  $Y$  is defined as
$$
\mathbf {I} (X; Y) = \sup _ {\mathcal {E}, \mathcal {F}} \sum_ {i, j} P _ {X Y} (E _ {i} \times F _ {j}) \log \frac {P _ {X Y} (E _ {i} \times F _ {j})}{P _ {X} (E _ {i}) P _ {Y} (F _ {j})},
$$
where the supremum is taken over all finite partitions  $\mathcal{E} = \{E_1, E_2, \ldots\}$  of  $\mathcal{X}$  and  $\mathcal{F} = \{F_1, F_2, \ldots\}$  of  $\mathcal{Y}$  with the usual conventions  $0 \log 0 = 0$  and  $0 \log 0 / 0 = 0$ .
Remark A.3. If  $P_{XY}$  is absolutely continuous with respect to  $P_X \otimes P_Y$  (e.g.,  $X$  and  $Y$  are finite r.v.), mutual information is equal to the following Kullback-Leibler divergence:
where  $f$  is the Radon-Nikodym derivative  $f = \mathrm{d}P_{XY} / \mathrm{d}(P_X\otimes P_Y)$ . If the absolute continuity does not hold, then  $\mathbf{I}(X;Y) = +\infty$  [Pinsker and Feinstein, 1964, Theorem 2.1.2].
Theorem 2.1. Let  $\mathcal{X},\mathcal{X}^{\prime},\mathcal{Y}$  and  $\mathcal{V}'$  be standard Borel spaces (e.g., smooth manifolds with their Borel  $\sigma$ -algebras) and  $f\colon \mathcal{X}\to \mathcal{X}'$  and  $g\colon \mathcal{Y}\to \mathcal{Y}'$  be continuous injective mappings. Then, for every  $\mathcal{X}$ -valued r.v.  $X$  and  $\mathcal{V}$ -valued r.v.  $Y$  it holds that  $\mathbf{I}(X;Y) = \mathbf{I}(f(X);g(Y))$ .
Proof. First, note that a continuous mapping between Borel spaces is measurable, so random variables  $f(X)$  and  $g(Y)$  are well-defined.
Then, observe that it suffices to prove that
$$
\mathbf {I} (f (X); Y) = \mathbf {I} (X; Y).
$$
It is well-known that we have  $\mathbf{I}(f(X);Y)\leq \mathbf{I}(X;Y)$  for every measurable mapping  $f$ , so we will need to prove the reverse inequality.
Recall form Definition A.2 that
$$
\mathbf {I} (X; Y) = \sup  _ {\mathcal {E}, \mathcal {F}} \sum_ {i, j} P _ {X Y} \left(E _ {i} \times F _ {j}\right) \log \frac {P _ {X Y} \left(E _ {i} \times F _ {j}\right)}{P _ {X} \left(E _ {i}\right) P _ {Y} \left(F _ {j}\right)},
$$
where the supremum is taken over all finite partitions  $\mathcal{E}$  of  $\mathcal{X}$  and  $\mathcal{F}$  of  $\mathcal{Y}$ .
The proof will go as follows: for every finite partition  $\mathcal{E}$  of  $\mathcal{X}$  we will construct a finite partition of  $\mathcal{X}'$ , which (together with the same arbitrary partition  $\mathcal{F}$  of  $\mathcal{Y}$ ) will give the same value of the sum. This will prove that  $\mathbf{I}(f(X); Y) \geq \mathbf{I}(X; Y)$ .
Take any finite partitions  $\{E_1,\ldots ,E_n\}$  of  $\mathcal{X}$  and  $\{F_1,\dots ,F_m\}$  of  $\mathcal{V}$ . First, we will show that sets
$$
G _ {0} = \mathcal {X} ^ {\prime} \backslash f (\mathcal {X})
$$
$$
G _ {1} = f (E _ {1})
$$
$$
G _ {2} = f (E _ {2})
$$
form a partition of  $\mathcal{X}'$ .
It is easy to see that they are pairwise disjoint as  $f$  is injective and it is obvious that their union is the whole  $\mathcal{X}'$ . Hence, we only need to show that they are all Borel.
For  $i \geq 1$  the Lusin-Suslin theorem [Kechris, 2012, Theorem 15.1] guarantees that  $G_{i}$  is Borel (continuous injective mappings between Polish spaces map Borel sets to Borel sets). Similarly,  $f(\mathcal{X})$  is Borel and hence  $G_{0} = \mathcal{X}' \setminus f(\mathcal{X})$  is Borel as well.
Next, we will prove that
$$
\sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} P _ {X Y} (E _ {i} \times F _ {j}) \log \frac {P _ {X Y} (E _ {i} \times F _ {j})}{P _ {X} (E _ {i}) P _ {Y} (F _ {j})} = \sum_ {i = 0} ^ {n} \sum_ {j = 1} ^ {m} P _ {f (X) Y} (G _ {i} \times F _ {j}) \log \frac {P _ {f (X) Y} (G _ {i} \times F _ {j})}{P _ {f (X)} (G _ {i}) P _ {Y} (F _ {j})}.
$$
Note that  $P_{f(X)}(G_0) = P_X(f^{-1}(G_0)) = P_X(\varnothing) = 0$ , so that we can ignore the terms with  $i = 0$ .
For  $i\geq 1$  we have
$$
P _ {f (X)} (G _ {i}) = P _ {X} \left(f ^ {- 1} \left(G _ {i}\right)\right) = P _ {X} \left(f ^ {- 1} \left(f \left(E _ {i}\right)\right)\right) = P _ {X} \left(E _ {i}\right)
$$
as  $f$  is injective.
Similarly,  $f \times 1_{\mathcal{Y}}$  is injective and maps  $E_i \times F_j$  onto  $G_i \times F_j$ , so
$$
P _ {f (X) Y} \left(G _ {i} \times F _ {j}\right) = P _ {X Y} \left(\left(f \times 1 _ {\mathcal {Y}}\right) ^ {- 1} \left(G _ {i} \times F _ {j}\right)\right) = P _ {X Y} \left(E _ {i} \times F _ {j}\right).
$$
This finishes the proof.
B Spiral Diffeomorphisms
In this section we will formally define the spiral diffeomorphisms and prove that they preserve the centred Gaussian distribution  $\mathcal{N}(0,\sigma^2 I_n)$ . They were inspired by the "swirling construction", discussed by Hauberg [2019].
Definition B.1. Let  $O(n)$  be the orthogonal group and  $\gamma \colon \mathbb{R} \to O(n)$  be any smooth function. The mapping
$$
f \colon \mathbb {R} ^ {n} \to \mathbb {R} ^ {n}
$$
given by  $f(x) = \gamma (||x||^2)x$  will be called the spiral.
Example B.2. Let  $\mathfrak{so}(n)$  be the vector space of all skew-symmetric matrices. For every two matrices  $R\in O(n)$  and  $A\in \mathfrak{so}(n)$  and any smooth mapping  $g\colon \mathbb{R}\to \mathbb{R}$  we can form a path
$$
\gamma (t) = R \exp (g (t) A).
$$
In particular,  $R = I_{n}$  and  $g(t) = vt$  yields the spirals defined earlier.
From the above definition it is easy to prove that spirals are diffeomorphisms. We formalize it by the following lemma.
Lemma B.3. Every spiral is a diffeomorphism.
Proof. Let  $\gamma$  be the path leading to the spiral  $f$ . As matrix inverse is a smooth function, we can define a smooth function  $h(x) = \gamma (||x||^2)^{-1}x$ .
Note that  $||f(x)||^2 = ||x||^2$ , as orthogonal transformation  $\gamma \left( ||x||^2 \right)$  preserves distances.
$$
\begin{array}{l} h (f (x)) = \gamma \left(\left\| f (x) \right\| ^ {2}\right) ^ {- 1} f (x) \\ = \gamma \left(\left\| x \right\| ^ {2}\right) ^ {- 1} f (x) \\ = \gamma \left(\left\| x \right\| ^ {2}\right) ^ {- 1} \gamma \left(\left\| x \right\| ^ {2}\right) x \\ = x. \\ \end{array}
$$
Analogously, we prove that  $f \circ h$  also is the identity mapping.
Finally, we will give the proof that spirals leave the centred normal distribution  $\mathcal{N}(0,\sigma^2 I_n)$  invariant. This will follow from a slightly stronger theorem:
Theorem B.4. Let  $P$  be a probability measure on  $\mathbb{R}^n$  with a smooth probability density function  $p$  which factorizes as:
$$
p (x) = u \left(\left\| x \right\| ^ {2}\right)
$$
for some smooth function  $u\colon \mathbb{R}\to \mathbb{R}$
Then, every spiral  $f: \mathbb{R}^n \to \mathbb{R}^n$  preserves the measure  $P$ :
$$
f _ {\#} P = P,
$$
where  $f_{\#}P = P \circ f^{-1}$  is the pushforward measure.
In particular, for  $P = \mathcal{N}(0,\sigma^2 I_n)$  (a centered isotropic Gaussian distribution), if  $X \sim \mathcal{N}(0,\sigma^2 I_n)$ , then  $f(X) \sim \mathcal{N}(0,\sigma^2 I_n)$ .
Proof. As we have already observed, for every  $x \in \mathbb{R}^n$  it holds that  $||f(x)||^2 = ||x||^2$ , so the PDF evaluated at both points is equal
$$
p (x) = p (f (x))
$$
Hence, we need to prove that for every point  $x \in \mathbb{R}^n$ ,  $|\det f'(x)| = 1$ , where  $f'(x)$  is the Jacobian matrix of  $f$  evaluated at the point  $x$ .
The proof of this identity was given by Carapetis [2022], but for the sake of completeness we supply a variant of it below.
First, we calculate the Jacobian matrix:
$$
\begin{array}{l} \frac {\partial f _ {i} (x)}{\partial x _ {j}} = \frac {\partial}{\partial x _ {j}} \sum_ {k} \gamma (| | x | | ^ {2}) _ {i k} x _ {k} \\ = \sum_ {k} \gamma (| | x | | ^ {2}) _ {i k} \frac {\partial x _ {k}}{\partial x _ {j}} + \sum_ {k} x _ {k} \frac {\partial \gamma (| | x | | ^ {2}) _ {i k}}{\partial x _ {j}} \\ = \gamma (| | x | | ^ {2}) _ {i j} + \sum_ {k} x _ {k} \cdot 2 x _ {j} \cdot \gamma^ {\prime} (| | x | | ^ {2}) _ {i k} \\ = \gamma (| | x | | ^ {2}) _ {i j} + 2 \left(\sum_ {k} \gamma^ {\prime} (| | x | | ^ {2}) _ {i k} x _ {k}\right) x _ {j} \\ \end{array}
$$
$$
f ^ {\prime} (x) = \gamma (\left\| x \right\| ^ {2}) + 2 \gamma^ {\prime} (\left\| x \right\| ^ {2}) x x ^ {T}.
$$