# Measuring Learned Optimization in Small Transformer Models

*This is original, independent research carried out in March and April of 2024.*

![A graphical abstract: A formula generates data which is used to pretrain models. These are then subject to three RL tasks of varying difficulty. The reward function of a model can be identified by looking at a task-specific prediction of optimization as comapared to measured optimization. ](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/66e64d42b1a3df9c4e9f6320320410fcd67b438ed27bafa5.png)

The degree to which a a policy optimizes the future can be quantified mathematically. A set of of very small transformer models were pretrained to predict the next token in a mathematical sequence, then subjected to reinforcement learning finetuning.

The optimizing power of each model can be predicted with high accuracy based on each model's score on its own RL task. By comparing predictions of optimization based on scores on each different RL task, a model's original reinforcement objective can be identified.

A related measure for impact can also be derived mathematically, and given a theoretical lower bound based on RL score. This gives further information about model behavior, and allows for the same analysis as the measure of optimization.

I also investigate the possibility of getting models to self-evaluate optimization and impact, with limited success.

Methods
=======

Pretraining on Sequence Prediction
----------------------------------

I defined a simple mathematical sequence defined by the following stochastic recurrence relation. This produces a pseudo-random but (to 98%) predictable sequence, alternating between elements of $\{0,...,7\}$ on even values of $t$ and $\{8,...,15\}$ on odd values of $t$.

$s_t = \big ( \cases{((\overset{16}{\underset{i=1}{\prod}}(s_{t-i}+1)\mod 17)\mod 8)  & with probability 98% \  \in\{0,...,7\} & with probabiltiy 2%} \big )  + 8 \times (t\mod2)$

I then trained a small encoder-only transformer model to predict the next element in the sequence given the previous 20 elements of the sequence.

This was followed by a reinforcement-learning phase in which the transformer was used to generate the next token on odd values of $n$ only, and the recurrence relation was used to generate the value of $s_{t+1}$. If $s_{t+1}$ was in $\{0,2,4,6\}$, this was used as a "successful" example to reinforce the model. I used a temperature of 1 when generating these sequences to introduce some randomness, but the temperature was reduced to 0 during evaluations and when calculating optimization.

A small amount of "maintenance" training (much lower learning rate) was used during this phase to ensure that model performance on the predictive tasks for even values of $t$ was maintained. Without this I saw rapid loss of performance on the "maintenance" dataset. I also found that I was unable to include "unsuccessful" examples (i.e. where $s_{t+1} \notin \{0,2,4,6\}$) with even a tiny negative learning rate, as this caused worsened performance at all tasks. Here is a typical set of results from training and evaluation:

I carried out this training on $N=5$ models per size for four model sizes between 18k and 402k parameters, giving the following plot:

![On the left is a plot showing pretraining and maintenance loss, on the right is a graph showing success rate on a task. Both are a function of parameters. Pretraining and maintenance loss show a U-shape, success generally increases with the number of parameters.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/9770dfb53023a4df3532db8ff7820124bbf9c86c95a0c251.png)

Pretraining loss increases over the last few model sizes, and the loss/time plots (some of which I have put in the Supplementary Information at the bottom of this post) showed signs of overfitting in the large models. Regularization was employed during training (0.01 weight decay in an AdamW optimizer, 10% dropout rate for neurons) so perhaps a larger dataset size is required to totally avoid this.

I then repeated the RL phase twice, once with $s_{t+1} \in \{0,4\}$ being reinforced,  ($n_{good}$ = 2) and once with $s_{t+1} \in \{0,1,2,4,5,6\}$ being reinforced ($n_{good}$ = 6). Here is a plot of success rate against model size across all three conditions.

![A plot showing success rate against model size on different tasks. Success rate is better than chance in all conditions, and increases with model size in all conditions.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/81cfc786c16024566a1f83de2f98d940103fb75ca2f2e8e4.png)

This plot shows mean $\pm$ standard error. In all cases model performance is a lot better than chance, and increases with model size.

Measuring Optimization
----------------------

I used a Monte Carlo simulation to measure the nats of optimization that are being applied to $s_{t+1}$ using the split-history method I've previously outlined. This involves taking the difference in entropy between two distributions:

![A diagram with the following equation at the top: Op(F, P; A) = H(F'|P'') - H(F). Below are two Bayes nets. The one on the left has nodes P, A,and F with nodes P to A, P to F and A to F. The network on the left has nodes P, P'', A', and F'. It has arrows P' to F', P'' to A', and A' to F'.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/2a190bce92510a6cf76f17a3da1be0adf232cf4609807524.png)

The algorithm in practice is this:

1.  Take a bunch of sequence examples from the testing data, ensuring that $t$ is odd.
2.  Feed them into the models to get a value for $s_t$, append to the sequence.
3.  Use the sequence-generator to get a set of values for $s_{t+1}$
4.  Look at the entropy of the resulting distribution over $s_{t+1}$, this is the optimized entropy.
5.  For each sequence from the training data, and get a value of $s_t'$ from this.
6.  Repeat Steps 3-4 with the entire data set as $...,s_{t-2},s_{t-1}$ prepended to $s_t'$, get one sample of "unoptimized" entropy
7.  Repeat Steps 5-6 with each sequence from the initial dataset, take the average unoptimized entropy
8.  Optimization = unoptimized entropy - optimized entropy

Here is a schematic illustration:

![A diagram showing the process outlined above. The final equation is Op(F, P; A) =  -H(F) + H(F'|P'')](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8e9b0cb324ec8657e00bbf7e96a849d51210e5668216a80d.png)

On the left, the model is able to respond to each sequence fed into it, so it can optimize the future creating a low-entropy distribution. On the right, the model is forced to give a fixed response to the external input, which is then appended to the sequences. This means that it cannot optimize the future and the entropy is higher.

I ran this calculation 10 times with 200 sequences in $P$ and took an average to get an idea of the model's optimizing capability. I also tested the sequence-generating function's self-optimization.

![A plot showing optimization against model size under three conditions. The fewer potential results are "good", the higher the optimization for a given model size. Optimization increases with model size in all cases.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b753ae9f718e9593157b0594a9827ae2fccda3e23a9c5c1d.png)

The fact that the sequence is optimizing itself mostly just amounts to saying that it is not a random walk, which we already knew. It is a good sanity check that all of the models get values either equal to or above this, and that optimization improves with model size.

Results
=======

Optimization vs RL Success Rate
-------------------------------

Optimization is calculated as the entropy difference of two distributions. Let us consider three parameters: $n$: the number of possible outcomes; $p$: the proportion of outcomes which are "successes"; and $s$:  the chance that the model achieves a successful outcome.

Assuming the model is "ambivalent" over successful outcomes, and likewise ambivalent over failed outcomes, then the value of $\mathrm H(F)$ should be equal to $-s \ln(\frac{s}{np}) -(1-s) \ln(\frac{1-s}{n(1-p)})$. If we then assume that all outcomes are equally likely when the model's outputs are "randomized", then $\mathrm H(F' |P'')$ is just $-\ln(\frac1n)$. If we take the difference we get the following expression:

$\mathrm{Op} \approx s \ln(\frac s {np}) + (1-s) \ln(\frac{1-s}{n(1-p)}) - \ln(\frac1n)$

$\mathrm{Op} \approx s \ln\frac s p + s \ln\frac1n + (1-s)\ln\frac{1-s}{1-p} + \ln \frac1n - s \ln\frac1n - \ln\frac1n$

$\mathrm{Op} \approx s \ln \frac sp + (1-s) \ln \frac {1-s}{1-p}$

Now I can plot this theoretical value for $\mathrm{Op}$ against $s$ for $p \in \{0.25, 0.5, 0.75\}$, and also plot all of the models on the same axes. Since we see a lot of run-to-run variation in model performance, I'll plot the raw data per model rather than statistics.

![A plot showing optimization plotted against success rate for three conditions. Three series of points are shown, each sitting on a similarly-colored curve.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8011aade0d0309cf7bd742dcb4c27525f488cdca4fa390ef.png)

I tried to find some galaxy-brained variable change which would make those three curves the same, but I couldn't. Instead I will just plot the predicted value of $\mathrm{Op}$ (based on their success rate) for each model against the actual value:

![The headline figure is repeated but with annotations. All the points lie very close to the theoretical line y=x.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/538357310086e320ccfc355ee445948d3761aaef7d9bf415.png)

In theory none of the models should fall below the dashed line representing the equation derived above. In practice they do. Some of this is error in measurements, but I'm sure some of it is also due to errors in my assumptions. In particular the assumption that $\mathrm H(F'|P'')$ is completely flat is unlikely to hold.

On the other hand, there there is no reason at all why the models shouldn't fall quite far above the theoretical line. Consider the $n_{good}=6$ case, so successful values are$s_{t+1} \in\{0,1,2,4,5,6\}$. If the model ends up with a 85:15 ratio of values 2 and 3 for $s_{t+1}$(and never gets any other number, which is impossible in this specific case but that's not important for my point) then it will have a success rate of 0.85, which implies an $\mathrm{Op}$ of 0.03, but its actual $\mathrm {Op}$ will be 1.66!

![A plot showing potential different outcomes to training. These outcomes are listed below.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5ff511265b29647adae65cf66d268420fb9cccb359a20441.png)

Sorry for the poor aesthetics of this diagram, it's very difficult to get the point across that the bottom left one has the lowest optimization.

For a given training performance (in this case I've tried to make them all ~75% correct) the model might do one of several things:

1.  Optimize for reinforced results and against non-reinforced results in an unbiased way
2.  Optimize against non-reinforced results correctly, but only optimize for some of the successful results
3.  Optimize for reinforced results correctly, but only optimize against some of the non-reinforced results
4.  Behave in such a way that certain reinforced results are optimized against and certain non-reinforced results are optimized for.

The last three are examples of one type of inner misalignment. All three are characterized by excess optimization compared to the "correct" answer.

Cross-Objective Evaluation
--------------------------

The fact that most of the models don't sit far above the line shows that the RL is doing its job! They are (mostly) pursuing the training objective faithfully, without picking up on other objectives. I simulate what might happen if a model picks up on an "incorrect" objective by cross-evaluating models on each others' tasks:

![A plot showing Measured Optimization against Score on n_good=4 task. The points corresponding to policies trained on the n_good=4 task lie on the theoretical curve, models trained on other tasks cluster to the left and above it.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/ddc7872b17137e6e7ae912087b3f4ee9851d6f08db2f00b8.png)

Now it becomes obvious that the $n_{good} = 4$-trained models are behaving correctly whereas the other models are not. This is important since a model trained on the $n_{good}=2$ task will typically have a good score on the $n_{good}=4$ task, sometimes even higher than a similarly-sized model trained on the $n_{good}=4$ task (!) but this plot clearly shows us that something is going wrong.

It becomes even clearer if we instead take the value of $\mathrm{Op}$ implied by the formula from above, and plot it against the actual value of $\mathrm{Op}$. Here are the results for all three tasks:

![A plot with predicted optimization on the x axis and measured on the y axis. The purple points for n_good=2 sit on a purple y=x line. The other points mainly sit above the line.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/4c58e378aae8de1b6481e3a60b137af67f25db4fd37ddc65.png)

![As above but with blue n_good=4 points sitting on a blue line, and the other clusters above.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/19c6b810d21c47feb38161b89d9ddaee2d1179f3c9dac7e3.png)

![As above but with yellow n_good = 2 points on the line and other clusters above it.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/942299d9b0ffa1a1bf2728f0fe597d600f3de9f5cb996299.png)

By comparing implied and measured $\mathrm{Op}$, we can separate the models trained on a given reward signal from ones trained on a different reward signal, even when one of the reward signals is a subset of the other.

Optimization vs Impact
----------------------

I will now take a second look at the measure for $\mathrm{Op}$. What I really wanted all along is to measure a policy's **impact** on the world, and optimizing the future is only one way in which this can happen. Another way is flattening the possibilities out! Consider the following informal diagram. "Original" distribution is not well-defined here, the point is just to give an intuitive explanation:

![A spiky original distribution is subject to two types of impact. With Optimizing impact, the distribution becomes spikier, and lower entropy. With Flattening impact, the distribution becomes flatter and higher entropy.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/908952891d9fd8aabd53c65df571edd8c53d32b7414e6479.png)

The motivation and derivations for this can be found in Appendix A, with longer proofs listed in Appendix B, but the upshot is that we can define a new function: $\mathrm{Imp}(F, P;A)$ using the KL divergence of $F$ and $F'$ like this:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/0a60047c8d65435fe6a2eea80538e6c0d42285786f2a6fec.png)

Which if we have a task success rate $s$ in $F$ and $s'$ in $F'$ must obey the following equation:

$\mathrm{Imp}(F,P;A) \equiv D_{KL}(F \Vert F'|P'') \ge s \ln \frac{s}{s'} + (1-s) \ln \frac{1-s}{1-s'} \equiv \mathrm{Imp}_{min}(F,P;A)$$$

This can be measured using the same split-history method used to measure $\mathrm{Op}$:

![A plot showing impact against parameters. Impact increases with parameter count and decreases with n_good](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8243070003ca5cdd20c3ca1aafe22d053746ffc489e6b269.png)

The lower bound $\mathrm{Imp}_{min}$ can be calculated from success rates and the two can be compared:

![A model showing Measured Impact plotted against Minimum Impact From Score. The points form a plume around y = 2.6x + 0.059](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/efde29f6640d0c45076f7b37c35331456faf7fdf3e5ee0a4.png)

Larger models have a higher impact, but they also tend to have a lower ratio of $\mathrm{Imp} / \mathrm{Imp}_{min}$than do the smaller ones. The single-line fit is also somewhat misleading, and actually the models appear to lie on three separate lines. I note that the values of  $\mathrm{Imp}_{min}$ are somewhat low in this case, rather than our measured impact being unexpectedly high. Our measure for $\mathrm{Imp}$ may not be perfect in this case.

I think the difference between $\mathrm{Imp}$ and $\mathrm{Imp}_{min}$ tells us something. In this case it tells us that the success rates in $F'$ are actually quite high:

![A plot showing success in F' against parameter count. Models for all values of n_good have better performance than chance, and it seems to slightly scale with model size.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/bb8187f1d13c92137aa43599f61f97e9494ab1168f7bbc62.png)

Success rate in $F'$ doesn't scale very much at all with model size, which indicates that models of all sizes exhibit roughly the same bias in outputs, based on statistical regularities in the data (e.g. a "13" is more commonly followed by a "2" than a "15" is). It might also be an artefact stemming from an uneven distribution of numbers in our sequence by default.

This highlights an important point about split-history measures: they only pick up on the effects of "active" inference in the policy, not on biases in the outputs which are produced during training.  We don't have to use a split-history method to generate $F'$, we can think up plenty of alternative ways to generate a comparison future, and therefore generate a family of measures for $\mathrm{Imp}$ and $\mathrm{Op}$:

*   Use a privileged "Do nothing" token as the model's output
*   Use a totally random output token
*   Use an untrained model, or a pretrained (but not finetuned) model

Despite these problems, if we repeat cross-objective evaluation using $\mathrm{Imp}$ we get these plots:

![A plot showing measured Impact against Imp_min based on the n_good=2 task. Points trained on the "correct" task form a plume above the line y=x, whereas points trained on the n_good=4 task are squished over to the left.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/83abac6c9af2f8391de6fdcc8051747c04ad85a7786b8395.png)

The $n_{good} = 6$ models don't appear because they do so badly on the task that $\mathrm{Imp}_{min}$ cannot be calculated. 

![As above but the Imp_min is based on score on the n_good=4 task. The correct policies lie above the line y=x, the n_good=2 models lie far above them, and the n_good=6 lines are squished over to the left](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/29ff70eb915d44886b0de5641b442369d9b1e0caeb1a89ae.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/103fab41e30dfa2196c3b12806c34f8bae57f94d9fbf23ce.png)

While the split is not as good as we saw with $\mathrm{Op}$, it seems that models not trained on the "correct" fall along a line of lower gradient, and models trained on "incorrect" tasks have higher gradients i.e. the latter diverge from the $\mathrm{Imp}_{min}$ line more rapidly.

There are many potential ways to visualize what's going on here, two more of which are in Appendix C at the end of this post.

Model Self-Evaluation
---------------------

A secondary goal of my work was to get models to evaluate their own optimizing power and impact. This is the reason for the maintenance training during finetuning. The self-evaluation scheme is the same as the optimization measuring scheme, except that instead of using the sequence-generating formula to get values for $s_{t+1}$, I used the model's own predictive power to get expected distributions over $s_{t+1}$ and $s_{t+1}'$:

A plot showing measured optimization plotted against self-estimated optimzation. Models from the $n_{good}=6$ task tend to fall on the line $y=x$, but as $n_{good}$ decreases, models fall further above the line.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f31b93afd07149d53e451c89e19faa9e96347d2cd058f27c.png)

I suspect what's happened here is bog standard model bias. The more heavily the model is optimizing, the less even the distribution of examples of $s_{t+1}$ it gets fed during the maintenance training. This might make it more biased towards predicting the rewarded variables after any sequence, which would cause it to underestimate $\mathrm{Op}$.

The same can be done for $\mathrm{Imp}$:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/81641dad85789c783d7ca580760c76c8df583c1c55ad2c00.png)

Which gives even worse results! I'm not sure whether or not this is also model bias. I don't have a tremendous amount of experience with ML, so I will not pursue this further in this post.

Conclusions
===========

The measure I've introduced for $\mathrm{Op}$ seems to be predicted very well by theory, and might have utility in detecting how well a policy's results line up with an RL task when it can be predicted accurately. The measure for $\mathrm{Imp}$ has some issues but shows some success towards the same goal. These can be considered complementary measures in many ways:

| **Optimization** | **Impact** |
| --- | --- |
| Can be positive or negative, the sign gives us information | Always $\\ge 0$ |
| Can be cancelled out by opposite-signed optimization elsewhere | Always a positive value, always at least adds up across different sources |
| No easy way to calculate theoretical values of Op in general (that I've found) | $\\mathrm{Imp}_{min}$ can be calculated for any success/failure RL task |
| Excellent at identifying model behaviour in this case | Less good at identifying model behaviour in this case |

I would be interested in applying both of these to more complex ML systems, but at the moment I don't any specific plans to do so, due to my very limited experience with ML. In Appendix A I also present equivalent measures which can be applied to the activations of a neural network.

A theoretical limitation of this work is that $\mathrm{Imp}_{min}$ can only be calculated when a task is binary success/failure. If the policy is trying to maximize some function world-state then there's no simple formula for $\mathrm{Imp}_{min}$ based on RL score. Work on these systems must either develop new maths or or find a way to break down utility maximization into a series of success/failure tasks, which can separately be evaluated for impact.

One practical limitation is in the self-evaluation stage. If the model's self-evaluation is biased, the measures of $\mathrm{Op}$ and $\mathrm{Imp}$ will be as well. This is especially problematic if $\mathrm{Op}$ or $\mathrm{Imp}$ is underestimated, allowing the policy to "lie" to us about how much optimization/impact it is doing. My wild guess is that AIs which include explicit world-modelling (such as DreamerV3) will be less biased by default, but that attempts at introducing impact regularization might reintroduce bias.

### Alex Turner's Existing Work

The expression for impact that I present here is totally independent with the AUP measure of impact as proposed by Alex Turner in [Towards a New Impact Measure](https://www.alignmentforum.org/posts/yEa7kwoMpsBgaBCgb/towards-a-new-impact-measure?_ga=2.10300598.204290679.1711976843-989286032.1698594240). There might be some subtle way in which they're related but I haven't thought about this enough to say more. I've also read [World State is the Wrong Abstraction for Impact](https://www.lesswrong.com/posts/pr3bLc2LtjARfK7nx/world-state-is-the-wrong-abstraction-for-impact) and agree with some of the points presented.

In response I would say that the metric I present here relies strongly on a model $F$ of the future world state, so only details captured in $F$ can affect the impact. In the limit where the future states only consist of $f \in \{\mathrm{success},\mathrm{failure}\}$, then impact is trivially equal to the lower bound and excess impact = 0.

Appendices
==========

Appendix A: Impact and Differential Impact
------------------------------------------

### Derivation of $\mathrm{Imp}$

I present the motivation for, and derivation of, the measure of $\mathrm{Imp}$ which I've used in this post. Returning to my estimator for $\mathrm{Op}$:

$\mathrm{Op} \approx s \ln \frac s p + (1-s) \ln \frac{1-s}{1-p}$

If we think these being two distributions over $\{\mathrm{success},\mathrm{failure}\}$ and think of $p$ as the probability of succeeding by chance, this becomes the formula for a Kullback-Liebler divergence. For a quick recap, the KL divergence of two distributions over a variable $x \in \chi$ follows this formula:

$D_{KL}(P \Vert Q) = \underset{x \in \chi}\sum P(x) \ln \frac{P(x)}{Q(x)}$

We could also estimate the probability of succeeding by chance using the split-history method. Let $s$ be the success rate in $F$, and $s'$ be the success rate in $F'$:

$s \ln \frac s {s'} + (1-s) \ln \frac{1-s}{1-s'}$

I will now present some results about KL divergences. I will start by defining $\chi_s \subset \chi$ as the successful outcomes. Let us define a "baseline" probability distribution $P(x)$ and from it a "baseline" success rate $s_0$:

$s_0 = \underset{x_s \in \chi_s} \sum P(x_s)$ 

Now imagine a policy acts on this distribution $P(x)$ and changes it to a new distribution. If this policy has a success rate $s \in [s_0,1]$, I define $Q(x)$ as follows:

 $Q(x) = \cases{\frac s {s_0} P(x) & $x \in \chi_s$ \ \frac {1-s} {1-s_0} P(x) & $x \not\in \chi_s$}$

This is what we might expect to be the effects of a "minimum impact" policy: it makes successful outcomes more likely and unsuccesful outcomes less likely while leaving relative probabilities otherwise intact. The KL-divergence between $Q$ and $P$ can then be calculated, and it looks familiar:

$D_{KL} (Q \Vert P) = s \ln \frac{s}{s_0} + (1-s) \ln  \frac{1-s}{1-s_0}$

This is the minimum KL divergence possible in shifting a distribution to achieve a given success rate. If we change the variable names to $Q \to F, P \to F'|P'', s_0 \to s'$ this allows us to write down the original relation for a policy's impact:

$\mathrm{Imp}(F,P;A) \equiv D_{KL}(F \Vert F'|P'') \ge s \ln \frac{s}{s'} + (1-s) \ln \frac{1-s}{1-s'} \equiv \mathrm{Imp}_{min}(F,P;A)$

### Differential Impact

In this case "differential" means "to do with differentiation". I've studied a construct I call differential optimization in the past, as it pertains to functions of real-valued variables. In this case if we have the functions $A = f(P)$, $F = g(A, P) \equiv h(P)$ we can define the following value:

$\mathrm{C}(F,P;A) = \frac{\partial F}{\partial P}|_{A \mathrm{varies}} / \frac{\partial F}{\partial P}|_{A \mathrm{constant}} \equiv \frac{\mathrm d h}{\mathrm d P} / \frac{\partial g}{\partial P}$

Intuitively, if $A$ is "optimizing" $F$, then when it is allowed to vary, $C \le 1$ since $F$ will change less when we allow $A$ to change than when we fix $A$. This led to the derivation of the differential optimization $\mathrm{Op} = -\ln (C)$.

This can be extended to a differential impact metric:

 $\mathrm{Imp} \equiv - \ln(C) + C(C - 1)$

This has a minimum $\mathrm{Imp} = 0$ at $C = 1$, but it is speculative and has not been tested, so while I will present it here I can give no guarantees at all about its utility.

We can also extend this to vector-valued $\mathbf P, \mathbf F$. If we define $\mathbf J_g$ as the Jacobian when $A$ varies, and $\mathbf J_h$ as the Jacobian when $A$ is constant, then $\mathbf C = \mathbf J_g^{-1} \mathbf J_h$ we get the following values for $\mathrm{Op}$ and $\mathrm{Imp}$:

$\mathrm{Op} = -\ln|\mathbf C| \equiv \ln|\mathbf J_g|  - \ln|\mathbf J_h|$

$\mathrm {Imp} =-\ln|\mathbf C| + \mathrm{Tr}(\mathbf C \mathbf C^\mathrm T) - \mathrm{Tr}(\mathbf C)$

If $\mathbf P$ and $\mathbf F$ do not have the same dimension, then $\mathbf J_g^{-1}$ does not exist and instead the following construction must be used:

$\mathrm{Op} = \frac12\ln|\mathbf J_g\mathbf J_g^\mathrm T| - \frac12\ln |\mathbf J_h \mathbf J_h^\mathrm T|$

$\mathrm {Imp} = \frac12\ln|\mathbf J_g\mathbf J_g^\mathrm T| - \frac12\ln |\mathbf J_h \mathbf J_h^\mathrm T| + \mathrm{Tr}(\mathbf J_h \mathbf J_h^\mathrm T(\mathbf J_g \mathbf J_g^\mathrm T)^{-1}) - \mathrm{Tr}(\mathbf J_g \mathbf J_h^T(\mathbf J_g \mathbf J_g^\mathrm T)^{-1})$

The motivation for constructions like this is to apply them to the activations of neural networks. For a network with width $w$, and a backpropagation time of $t$, I believe the time-complexity of this contains a polynomial term in $w$ (possibly $\mathcal O(w^5(\log w)^2)$ if using Bareiss algorithm) for the matrix inverse, and a term in $w \times t$.

Appendix B: Proofs
------------------

### Derivation and proof of $\mathrm{Imp} \ge \mathrm{Imp}_{min}$

$D_{KL}(Q\Vert P) = \underset {x_s \in \chi_s} \sum\frac {s}{s_0}  P(x_s) \ln \frac {\frac {s}{s_0} P(x_s)}{P(x_s)} + \underset {x_u \not\in \chi_s} \sum \frac {1-s}{1-s_0} P(x_u) \ln \frac {\frac {1-s}{1-s_0} P(x_u)}{P(x_u)}$

$D_{KL}(Q \Vert P) = \frac{s}{s_0} \underset{x_s \in \chi_s} \sum P(x_s) \ln \frac{s}{s_0} + \frac{1-s}{1-s_0} \underset{x_u \not \in \chi_s} \sum P(x_u) \ln\frac{1-s}{1-s_0}$

$D_{KL}(Q \vert P) = \frac {s}{s_0} \times s_0 \ln\frac s {s_0} + \frac{1-s}{1-s_0}  \times (1-s_0) \ln \frac{1-s}{1-s_0}$

$D_{KL} (Q \Vert P) = s \ln \frac{s}{s_0} + (1-s) \ln  \frac{1-s}{1-s_0}$

I will prove that this choice of $Q$ is a global minimum of $D_{KL}(Q \Vert P)$ for a fixed $P$.

Consider a distribution $R = Q + \delta Q$, which involves moving some amount of probability mass $\delta$ from $x_1$ to $x_2$. Without loss of generality, take both to be in $\chi_s$ (they must both be in either $\chi_s$ or $\chi_s^C$ so that $R(x \in \chi_s) = s$ holds) Consider the value of

$D_{KL}(R \Vert P) - D_{KL}(Q \Vert P)$

Trivially we only need to look at the components relevant to $x_1$ and $x_2$:

$R(x_1) \ln \frac {R(x_1)}{P(x_1)} + R(x_2) \ln \frac {R(x_2)}{P(x_2)} - Q(x_1) \ln \frac {Q(x_1)}{P(x_1)} - Q(x_2) \ln \frac {Q(x_2)}{P(x_2)}$

Expand values of $R(x)$:

$(Q(x_1)- \delta) \ln \frac{\frac{s}{s_0} P(x_1) - \delta}{P(x_1)} + (Q(x_2)+ \delta) \ln \frac{\frac{s}{s_0} P(x_2) + \delta}{P(x_2)} -Q(x_1)\ln \frac{\frac{s}{s_0} P(x_1)}{P(x_1)} - Q(x_2) \ln \frac{\frac{s}{s_0} P(x_2)}{P(x_2)}$

Expand and collect factors of $Q(x)$, cancelling the $P(x)$ on the bottom:

$Q(x_1) \ln \frac {\frac{s}{s_0} P(x_1) - \delta}{\frac{s}{s_0} P(x_1)} - \delta \ln \frac{\frac{s}{s_0} P(x_1) - \delta}{ P(x_1)} + Q(x_2) \ln \frac {\frac{s}{s_0} P(x_2) + \delta}{\frac{s}{s_0} P(x_2)} + \delta \ln \frac{\frac{s}{s_0} P(x_2) + \delta}{ P(x_2)}$

Collect the factors of $\delta$, expand stuff to a $\ln(1 + y)$ form:

$Q(x_1) \ln(1 - \frac{s_0}{s} \frac{\delta}{P(x_1)}) + Q(x_2) \ln(1 + \frac {s_0}{s} \frac{\delta}{P(x_2)}) + \delta \big [ \ln \frac{\frac{s}{s_0} P(x_2) + \delta}{ P(x_2)} - \ln \frac{\frac{s}{s_0} P(x_1) - \delta}{ P(x_1)}\big ]$

$Q(x_1) \ln(1 - \frac{s_0}{s} \frac{\delta}{P(x_1)}) + Q(x_2) \ln(1 + \frac {s_0}{s} \frac{\delta}{P(x_2)}) + \delta\big [\ln (1 + \frac {s_0}{s} \frac {\delta}{P(x_2)}) - \ln(1 - \frac {s_0}{s} \frac {\delta}{P(x_1)})\big ]$

Use the taylor expansion $\ln(1+y) \approx y - \frac12 y^2 ...$ up to $\delta^2$.

$Q(x_1)(-\frac{s_0}{s} \frac{\delta}{P(x_1)} - \frac12(-\frac{s_0}{s}\frac{\delta}{P(x_1)})^2...) + Q(x_2)(\frac{s_0}{s} \frac{\delta}{P(x_2)}-\frac12(\frac{s_0}{s}\frac{\delta}{P(x_2)})^2...) + \delta\big[(\frac{s_0}{s} \frac{\delta}{P(x_2)}...) - (\frac{s_0}{s} \frac{\delta}{P(x_1)}...)\big]$

Sub in $Q(x) = \frac s{s_0} P(x)$, expand, cancel:

$-\frac 12\delta^2 \frac{s_0}{s}(\frac1{P(x_1)} + \frac1{P(x_2)}) + \delta^2\frac{s_0}{s}(\frac{1}{P(x_2)} + \frac{1}{P(x_1)})$

Subtract:

$\frac 12\delta^2 \frac{s_0}{s}(\frac1{P(x_1)} + \frac1{P(x_2)}) \ge 0$

Therefore $Q(x)$ is a local minimum of $D_{KL}(Q\Vert P)$ subject to our condition that $Q(X \in \chi_x) = s$. $D_{KL}(Q \Vert P)$ is convex in $Q$ for fixed $P$, therefore we have found the unique global minimum.

### Derivation of Differential Impact

The measure of $\mathrm{Op}$ based on entropy that I've used here was based on the following comparison to differential optimization:

Consider the network $A = P$, $F = P - (1-k) A$. This gives $\mathrm{C}(F,P;A) = k$ and $\mathrm{Op}(F,P;A) = -\ln k$.

This can be extended to an entropic measure of $\mathrm{Op}$ by considering uncertainty over $P$, specifically:

 $P \sim N(\mu_P, \sigma_P)$

$A \sim N(\mu_P, \sigma_P)$

$F \sim N(k\mu_P, k \sigma_P)$

Using split-histories we get:

$P', P'' \sim N(\mu_P, \sigma_P)$

$A' \sim N(\mu_P, \sigma_P)$

$F'|P''=p'' \sim N(\mu_P-(1-k)p'', \sigma_P)$

If we take $\mathrm{Op}(F,P;A ) =H(F'|P'')-H(F)$ this gives the familiar value of $\ln k$. We may instead investigate the value of $\mathrm{Imp}(F,P;A) = D_{KL}(F\Vert F'|P'')$. Letting $F\sim N(\mu_1, \sigma_1)$, $F'|P'' \sim N(\mu_2, \sigma_2)$ for brevity:

$\mathrm{Imp} = \ln(\frac{\sigma_2}{\sigma_1}) + \frac12 \frac{\sigma_1^2 + (\mu_1, \mu_2)^2}{\sigma_2^2} - \frac12$

Substituting:

$\mu_1 - \mu_2 = k \mu_P - \mu_P + (1-k)p'' = (1-k)p'' - (1-k)\mu_P$

$(\mu_1-\mu_2)^2 = (1-k)^2(\mu_P^2 - 2\mu_Pp'' + p''^2)$

Taking $\mathbb E((\mu_1-\mu_2)^2)$ with respect to $p''$ requires taking $\mathbb E(p'') = \mu_P$, $\mathbb E(p''^2) = \mu_P^2 + \sigma_P^2$:

$(\mu_1 - \mu_2)^2 = (1-k)^2(\mu_P^2 - 2\mu_P^2 +\mu_P^2 + \sigma_P^2) = \sigma_P^2$

$\sigma_1 = k\sigma_P$

$\sigma_2 = \sigma_P$

Substituting into our original equation:

$\mathrm {Imp} = \ln(1/k) + \frac12[k^2 + (1-k)^2\frac{\sigma_P^2}{\sigma_P^2}] - \frac12$

$\mathrm{Imp} = - \ln(k) + \frac12[k^2 + 1 - 2k + k^2] - \frac12$

$\mathrm{Imp} = -\ln(k) + k^2 - k$

$\mathrm{Imp} = - \ln(k) + k(k-1)$

Which, if we extend to $\mathrm{C}$, gives

$\mathrm{Imp} = -\ln(C) +C(C- 1)$

### Derivation of Multivariate Differential Impact and Optimization

Let us take vectors $\mathbf p, \mathbf a, \mathbf f$, and $\mathbf p', \mathbf p'', \mathbf a', \mathbf f'$ in the same manner as above. Assume around some value of $\mathbf p$ we have the following Jacobians.

$\mathbf J_f = \frac{\mathrm d \mathbf a}{\mathrm d \mathbf p}$ $\mathbf J_g = \frac{\partial\mathbf f}{\partial\mathbf p}$  $\mathbf J_h = \frac{\mathrm d\mathbf f}{\mathrm d\mathbf p}$

Without loss of generality, take the means of all of these variables to be $\mathbf 0$. There exists a formula for transforming a multivariate normal distribution[^3hb64deu3cw].

$\mathbf P \sim N(\mathbf 0, \mathbf \Sigma_\mathbf p)$

$\mathbf A \sim N(\mathbf 0, \mathbf J_f\mathbf\Sigma_\mathbf p\mathbf J_f^\mathrm T)$

$\mathbf F \sim N(\mathbf 0, \mathbf J_h \mathbf \Sigma_\mathbf p \mathbf J^\mathrm T_h)$

Now for $\mathbf f'|\mathbf p''$, the mean will no longer be zero:

$\mathbf F'|\mathbf P''=\mathbf p'' \sim N((\mathbf J_h - \mathbf J_g) \mathbf p'', \mathbf J_g \mathbf \Sigma_\mathbf p \mathbf J_g^\mathrm T)$

We can calculate the KL divergence of $D_{KL}(\mathbf F\Vert \mathbf F'\vert \mathbf P'')$ using another formula[^chms10yvueh]: 

$\frac12\big[\ln\frac{|\mathbf \Sigma_2|}{|\mathbf \Sigma_1|} - n_{dim} + \mathrm {Tr}(\mathbf \Sigma_2^{-1} \mathbf\Sigma_1) + (\mathbf \mu_2 -\mu_1)^{\mathrm T} \mathbf \Sigma_2^{-1}(\mu_2-\mu_1)]$

$\mathbf \Sigma_1 = \mathbf J_h \mathbf \Sigma_\mathbf p \mathbf J_h^\mathrm T$  
$\mathbf \Sigma_2 = \mathbf J_g \mathbf \Sigma_\mathbf p \mathbf J^\mathrm T_g$  
$\mu_2 - \mu_1 = (\mathbf J_h - \mathbf J_g ) \mathbf p''$

Therefore our impact will be:

$\frac12\Big [\ln\frac{|\mathbf J_g \mathbf \Sigma_\mathbf p\mathbf J_g^\mathrm T|}{|\mathbf J_h \mathbf \Sigma_\mathbf p\mathbf J_h^\mathrm T|} - n_{dim} + \mathrm Tr((\mathbf J_g \mathbf \Sigma_\mathbf p\mathbf J_g^\mathrm T)^{-1}\mathbf J_h \mathbf \Sigma_\mathbf p\mathbf J_h^\mathrm T) + \mathbf p''^\mathrm T (\mathbf J_h - \mathbf J_g)^\mathrm T(\mathbf J_g \mathbf \Sigma_\mathbf p\mathbf J_g^\mathrm T)^{-1}(\mathbf J_h - \mathbf J_g) \mathbf p''\Big]$

Taking the expected value of the third component is actually easy if you have access to the internet. We can see that it is of the form $\mathbb E(\mathbf v^\mathrm T \mathbf M \mathbf v)$ where $\mathbf v$ is multivariate normal. This has a closed-form solution[^ya6tgc8mdr8]:

$\mu^\mathrm T \mathbf M \mu + \mathrm{Tr}(\mathbf M \mathbf \Sigma)$

Therefore we have the following expression:

$\frac12\Big [\ln\frac{|\mathbf J_g \mathbf \Sigma_\mathbf p\mathbf J_g^\mathrm T|}{|\mathbf J_h \mathbf \Sigma_\mathbf p\mathbf J_h^\mathrm T|} - n_{dim} + \mathrm Tr((\mathbf J_g \mathbf \Sigma_\mathbf p\mathbf J_g^\mathrm T)^{-1}\mathbf J_h \mathbf \Sigma_\mathbf p\mathbf J_h^\mathrm T) + \mathrm{Tr}\Big((\mathbf J_h - \mathbf J_g)^\mathrm T(\mathbf J_g \mathbf \Sigma_\mathbf p\mathbf J_g^\mathrm T)^{-1}(\mathbf J_h - \mathbf J_g)\mathbf \Sigma_\mathbf p \Big)\Big]$

We can make some progress towards simplifying this if we take $\mathbf \Sigma_\mathbf p = \sigma_p \mathbb I$, which in this case lets us cancel everything out involving a $\mathbf \Sigma _\mathbf p$, since the scalar value of $\sigma_p$ commutes with all matrices, $|\sigma_p\mathbb I| = \sigma_p^{n_{dim}}$, and $(\sigma_p \mathbb I)^{-1} = \sigma_p^{-1} \mathbb I$. We will also assume that all the Jacobians are invertible.

$\frac12\Big [\ln\frac{|\mathbf J_g\mathbf J_g^\mathrm T|}{|\mathbf J_h \mathbf J_h^\mathrm T|} - n_{dim} +\mathrm Tr((\mathbf J_g \mathbf J_g^\mathrm T)^{-1}\mathbf J_h \mathbf J_h^\mathrm T) + \mathrm{Tr}\Big((\mathbf J_h - \mathbf J_g)^\mathrm T(\mathbf J_g \mathbf J_g^\mathrm T)^{-1}(\mathbf J_h - \mathbf J_g)\Big)\Big]$

$\frac12\Big [\ln\frac{|\mathbf J_g\mathbf J_g^\mathrm T|}{|\mathbf J_h \mathbf J_h^\mathrm T|} - n_{dim} + \mathrm Tr((\mathbf J_g\mathbf J_g^\mathrm T)^{-1}\mathbf J_h\mathbf J_h^\mathrm T) + \mathrm {Tr}(\mathbf J_h^\mathrm T (\mathbf J_g^{\mathrm T})^{-1}\mathbf J_g^{-1} \mathbf J_h) - \mathrm {Tr}(\mathbf J_h^\mathrm T (\mathbf J_g^{\mathrm T})^{-1}\mathbf J_g^{-1} \mathbf J_g) - \mathrm {Tr}(\mathbf J_g^\mathrm T (\mathbf J_g^{\mathrm T})^{-1}\mathbf J_g^{-1} \mathbf J_h) + \mathrm {Tr}(\mathbf J_g^\mathrm T (\mathbf J_g^{\mathrm T})^{-1}\mathbf J_g^{-1} \mathbf J_g) \Big]$

$\frac12\Big [\ln\frac{|\mathbf J_g\mathbf J_g^\mathrm T|}{|\mathbf J_h \mathbf J_h^\mathrm T|} - n_{dim} + \mathrm Tr((\mathbf J_g\mathbf J_g^\mathrm T)^{-1}\mathbf J_h\mathbf J_h^\mathrm T) + \mathrm {Tr}(\mathbf J_h^\mathrm T (\mathbf J_g\mathbf J_g^\mathrm T)^{-1} \mathbf J_h) - \mathrm {Tr}(\mathbf J_h^\mathrm T (\mathbf J_g^{\mathrm T})^{-1}) - \mathrm {Tr}(\mathbf J_g^{-1} \mathbf J_h) + \mathrm {Tr}(\mathbb I) \Big]$

$\frac12\Big [\ln\frac{|\mathbf J_g\mathbf J_g^\mathrm T|}{|\mathbf J_h \mathbf J_h^\mathrm T|} + \mathrm Tr((\mathbf J_g\mathbf J_g^\mathrm T)^{-1}\mathbf J_h\mathbf J_h^\mathrm T) + \mathrm {Tr}(\mathbf J_h^\mathrm T (\mathbf J_g \mathbf J_g^\mathrm T)^{-1} \mathbf J_h) - \mathrm {Tr}(\mathbf J_h^\mathrm T (\mathbf J_g^{\mathrm T})^{-1}) - \mathrm {Tr}(\mathbf J_g^{-1} \mathbf J_h)\Big]$

$\frac12\Big [\ln\frac{|\mathbf J_g\mathbf J_g^\mathrm T|}{|\mathbf J_h \mathbf J_h^\mathrm T|} + \mathrm Tr((\mathbf J_g\mathbf J_g^\mathrm T)^{-1}\mathbf J_h\mathbf J_h^\mathrm T) + \mathrm {Tr}(\mathbf J_h^\mathrm T (\mathbf J_g \mathbf J_g^\mathrm T)^{-1} \mathbf J_h) - 2 \mathrm {Tr}(\mathbf J_g^{-1} \mathbf J_h)\Big]$

Using the cyclic property of the trace:

$\frac12\Big [\ln\frac{|\mathbf J_g\mathbf J_g^\mathrm T|}{|\mathbf J_h \mathbf J_h^\mathrm T|} + 2 \mathrm Tr((\mathbf J_g\mathbf J_g^\mathrm T)^{-1}\mathbf J_h\mathbf J_h^\mathrm T) - 2 \mathrm {Tr}(\mathbf J_g^{-1} \mathbf J_h)\Big]$

$\frac12 \ln\frac{|\mathbf J_g \mathbf J_g^\mathrm T|}{|\mathbf J_h \mathbf J_h^\mathrm T|} + \mathrm Tr((\mathbf J_g^\mathrm T)^{-1}\mathbf J_g^{-1}\mathbf J_h \mathbf J_h^\mathrm T) - \mathrm {Tr}(\mathbf J_g^{-1} \mathbf J_h)$

$\frac12 \ln\frac{|\mathbf J_g \mathbf J_g^\mathrm T|}{|\mathbf J_h \mathbf J_h^\mathrm T|} + \mathrm Tr(\mathbf J_g^{-1}\mathbf J_h \mathbf J_h^\mathrm T(\mathbf J_g^\mathrm T)^{-1}) + \mathrm {Tr}(\mathbf J_g^{-1} \mathbf J_h)$

$\frac12 \ln|\mathbf J_g \mathbf J_g^\mathrm T(\mathbf J_h \mathbf J_h^\mathrm T)^{-1}| + \mathrm{Tr}(\mathbf J_g^{-1}\mathbf J_h (\mathbf J_g^{-1}\mathbf J_h)^\mathrm T) - \mathrm {Tr}(\mathbf J_g^{-1} \mathbf J_h)$

And if we define $\mathbf C = \mathbf J_g^{-1} \mathbf J_h$ we get:

$-\ln|\mathbf C| + \mathrm{Tr}(\mathbf C \mathbf C^\mathrm T) - \mathrm{Tr}(\mathbf C)$

This seems to have the form of $-\ln(C^2) + C^2 - C$, and in fact if we consider $\mathbf P$, $\mathbf A$, and $\mathbf F$ to just be concatenations of variables, which maeans all the $\mathbf J$ matrices are diagonal, we see that our equation has the form.

$- \ln (\prod C) + \sum C^2 - \sum C = \sum \big (- \ln C + C^2 - C \big)$

Which is a nice sanity check. The value of $\mathrm {Op}$ is just the entropy difference $\frac 12 \ln \mathbf \Sigma_2 - \frac12 \ln \mathbf \Sigma_1$ which simplifies to $- \ln|\mathbf C|$ for free.

If the Jacobians are not invertible, but we assume that $\mathbf{J}_g \mathbf J_g^\mathrm T$ is invertible, we instead get:

$\mathrm {Imp} = \frac12\ln|\mathbf J_g\mathbf J_g^\mathrm T| - \frac12\ln |\mathbf J_h \mathbf J_h^\mathrm T| + \mathrm{Tr}(\mathbf J_h \mathbf J_h^\mathrm T(\mathbf J_g \mathbf J_g^\mathrm T)^{-1}) - \mathrm{Tr}(\mathbf J_g \mathbf J_h^T(\mathbf J_g \mathbf J_g^\mathrm T)^{-1})$

Appendix C: Supplementary Plots
-------------------------------

### Other Ways to Visualize Impact Plots

Here I plotted the "Impact ratio" $\mathrm{Imp} / \mathrm{Imp}_{min}$ against $\mathrm{Imp}_{min}$:

![The next six plots are just re-visualizations of previously plotted and discussed data, so I'm not going to write a bespoke description for each one.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/6b209d23e14dc6d658100c209e31ad537204c23e0736ddd2.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e7b1438d8dbfe46668bd8d2b3fa20d8fb10e12ef6e4e7ac6.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/bdb98026069f2f0431dedf52d08dd38065aeafda0390c05c.png)

Here I plotted "Excess Impact" $\mathrm{Imp} - \mathrm{Imp}_{min}$ against $\mathrm{Imp}$:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d496aa398f262ad670bef3219c198ad5ebc2bb80fc71bb6e.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e2fc717a663d0572a6ea04a659a7920e22bb1d6671e2b444.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b0a9974cd043d95921007d19015d8560d01ccb7ffb364d80.png)

### Example training runs from $n_{good}$ = 4

![There's 16 plots here, but none are important to the key point. Screen reader users I'm afraid you'll have to get someone else to nitpick the raw data.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1d8403b705fe68e2dac014b0d5b981e96dd63c3b1b709e99.png)

### Example figures summarizing training runs:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/20d286bfc688f90b7bd2160812c64f6ac2b59309ff28a69d.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/fceee8f47aa6295b4c57911d9d92014469a39f1dbe22bbe2.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/3bfe2cf63f281f180808bc7506f56e7e8822f3f859e5de62.png)

[^3hb64deu3cw]: https://statproofbook.github.io/P/mvn-ltt.html 

[^chms10yvueh]: https://stats.stackexchange.com/questions/60680/kl-divergence-between-two-multivariate-gaussians 

[^ya6tgc8mdr8]: https://statproofbook.github.io/P/mean-qf.html