prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: What is the proposed approach in this research paper?

Read the following text and answer briefly.

Context 1: \section{Introduction}
\label{sec:introduction}

Probabilistic models have proven to be very useful in a lot of applications in signal processing where signal estimation is needed \cite{rabiner1989tutorial,arulampalam2002tutorial,ji2008bayesian}  Some of their advantages are that 1) they force the designer to specify all the assumptions of the model, 2) they provide a clear separation between the model and the algorithm used to solve it, and 3) they usually provide some measure of uncertainty about the estimation 

On the other hand, adaptive filtering is a standard approach in estimation problems when the input is received as a stream of data that is potentially non-stationary  This approach is widely understood and applied to several problems such as echo cancellation \cite{gilloire1992adaptive}, noise cancellation \cite{nelson1991active}, and channel equalization \cite{falconer2002frequency} 

Although these two approaches share some underlying relations, there are very few connections in the literature  The first important attempt in the signal processing community to relate these two fields was the connection between a linear Gaussian state-space model (i e  Kalman filter) and the RLS filter, by Sayed and Kailath \cite{sayed1994state} and then by Haykin \emph{et al } \cite{haykin1997adaptive}  The RLS adaptive filtering algorithm emerges naturally when one defines a particular state-space model (SSM) and then performs exact inference in that model  This approach was later exploited in \cite{van2012kernel} to design a kernel RLS algorithm based on Gaussian processes 

A first attempt to approximate the LMS filter from a probabilistic perspective was presented in \cite{park2014probabilistic}, focusing on a kernel-based implementation  The algorithm of \cite{park2014probabilistic} makes use of a Maximum a Posteriori (MAP) estimate as an approximation for the predictive step  However, this approximation does not preserve the estimate of the uncertainty in each step, therefore degrading the performance of the algorithm 

In this work, we provide a similar connection between state-space models and least-mean-squares (LMS)  Our approach is based on approximating the posterior distribution with an isotropic Gaussian distribution  We show how the computation of this approximated posterior leads to a linear-complexity algorithm, comparable to the standard LMS  Similar approaches have already been developed for a variety of problems such as channel equalization using recurrent RBF neural networks \cite{cid1994recurrent}, or Bayesian forecasting \cite{harrison1999bayesian}  Here, we show the usefulness of this probabilistic approach for adaptive filtering 

The probabilistic perspective we adopt throughout this work presents two main advantages  Firstly, a novel LMS algorithm with adaptable step size emerges naturally with this approach, making it suitable for both stationary and non-stationary environments  The proposed algorithm has less free parameters than previous LMS algorithms with variable step size \cite{kwong1992variable,aboulnasr1997robust,shin2004variable}, and its parameters are easier to be tuned w r t  these algorithms and standard LMS  Secondly, the use of a probabilistic model provides us with an estimate of the error variance, which is useful in many applications 

Experiments with simulated and real data show the advantages of the presented approach with respect to previous works  However, we remark that the main contribution of this paper is that it opens the door to introduce more Bayesian machine learning techniques, such as variational inference and Monte Carlo sampling methods \cite{barber2012bayesian}, to adaptive filtering \\


\section{Probabilistic Model}

Throughout this work, we assume the observation model to be linear-Gaussian with the following distribution,

\begin{equation}
p(y_k|{\bf w}_k) = \mathcal{N}(y_k;{\bf x}_k^T {\bf w}_k , \sigma_n^2),
\label{eq:mess_eq}
\end{equation}
where  $\sigma_n^2$ is the variance of the observation noise, ${\bf x}_k$ is the regression vector and ${\bf w}_k$ is the parameter vector to be sequentially estimated, both $M$-dimensional column vectors 


In a non-stationary scenario, ${\bf w}_k$ follows a dynamic process  In particular, we consider a diffusion process (random-walk model) with variance $\sigma_d^2$ for this parameter vector:


\begin{equation}
p({\bf w}_k|{\bf w}_{k-1})= \mathcal{N}({\bf w}_k;{\bf w}_{k-1}, \sigma_d^2 {\bf I}),
\label{eq:trans_eq}
\end{equation}
where $\bf I$ denotes the identity matrix  In order to initiate the recursion, we assume the following prior distribution on ${\bf w}_k$

\begin{equation}
p({\bf w}_0)= \mathcal{N}({\bf w}_0;0, \sigma_d^2{\bf I}) \nonumber
\end{equation}

\section{Exact inference in this model: Revisiting the RLS filter}

Given the described probabilistic SSM, we would like to infer the posterior probability distribution $p({\bf w}_k|y_{1:k})$ 
Since all involved distributions are Gaussian, one can perform exact inference, leveraging the probability rules in a straightforward manner  The resulting probability distribution is
\begin{equation}
p({\bf w}_k|y_{1:k}) =  \mathcal{N}({\bf w}_k;{\bf\boldsymbol\mu}_{k}, \boldsymbol\Sigma_{k}), \nonumber
\end{equation}
in which the mean vector ${\bf\boldsymbol\mu}_{k}$ is given by
\begin{equation}
{\bf\boldsymbol\mu}_k = {\bf\boldsymbol\mu}_{k-1} + {\bf K}_k (y_k - {\bf x}_k^T {\bf\boldsymbol\mu}_{k-1}){\bf x}_k, \nonumber
\end{equation}
where we have introduced the auxiliary variable
\begin{equation}
{\bf K}_k = \frac{ \left(\boldsymbol\Sigma_{k-1} + \sigma_d^2 {\bf I}\right)}{{\bf x}_k^T  \left(\boldsymbol\Sigma_{k-1} + \sigma_d^2 {\bf I}\right)  {\bf x}_k + \sigma_n^2}, \nonumber
\end{equation}
and the covariance matrix $\boldsymbol\Sigma_k$ is obtained as
\begin{equation}
\boldsymbol\Sigma_k = \left( {\bf I} -  {\bf K}_k{\bf x}_k {\bf x}_k^T \right) ( \boldsymbol\Sigma_{k-1} +\sigma_d^2), \nonumber
\end{equation}
Note that the mode of $p({\bf w}_k|y_{1:k})$, i e  the maximum-a-posteriori estimate (MAP), coincides with the RLS adaptive rule
\begin{equation}
{{\bf w}}_k^{(RLS)} = {{\bf w}}_{k-1}^{(RLS)} + {\bf K}_k (y_k - {\bf x}_k^T {{\bf w}}_{k-1}^{(RLS)}){\bf x}_k  
\label{eq:prob_rls}
\end{equation}
This rule is similar to the one introduced in \cite{haykin1997adaptive} 

Finally, note that the covariance matrix $\boldsymbol\Sigma_k$ is a measure of the uncertainty of the estimate ${\bf w}_k$ conditioned on the observed data $y_{1:k}$  Nevertheless, for many applications a single scalar summarizing the variance of the estimate could prove to be sufficiently useful  In the next section, we show how such a scalar is obtained naturally when $p({\bf w}_k|y_{1:k})$ is approximated with an isotropic Gaussian distribution  We also show that this approximation leads to an LMS-like estimation 
 


\section{Approximating the posterior distribution: LMS filter }

The proposed approach consists in approximating the posterior distribution $p({\bf w}_k|y_{1:k})$, in general a multivariate Gaussian distribution with a full covariance matrix, by an isotropic spherical Gaussian distribution 

\begin{equation}
\label{eq:aprox_post}
\hat{p}({\bf w}_{k}|y_{1:k})=\mathcal{N}({\bf w}_{k};{\bf \hat{\boldsymbol\mu}}_{k}, \hat{\sigma}_{k}^2 {\bf I} ) 
\end{equation}

In order to estimate the mean and covariance of the approximate distribution $\hat{p}({\bf w}_{k}|y_{1:k})$, we propose to select those that minimize the Kullback-Leibler divergence with respect to the original distribution, i e , 

\begin{equation}
\{\hat{\boldsymbol\mu}_k,\hat{\sigma}_k\}=\arg \displaystyle{  \min_{\hat{\boldsymbol\mu}_k,\hat{\sigma}_k}} \{ D_{KL}\left(p({\bf w}_{k}|y_{1:k}))\| \hat{p}({\bf w}_{k}|y_{1:k})\right) \}  \nonumber
\end{equation}

The derivation of the corresponding minimization problem can be found in Appendix A  In particular, the optimal mean and the covariance are found as
\begin{equation}
{\hat{\boldsymbol\mu}}_{k} = {\boldsymbol\mu}_{k};~~~~~~ \hat{\sigma}_{k}^2 = \frac{{\sf Tr}\{ \boldsymbol\Sigma_k\} }{M} 
\label{eq:sigma_hat}
\end{equation}


We now show that by using \eqref{eq:aprox_post} in the recursive predictive and filtering expressions we obtain an LMS-like adaptive rule  First, let us assume that we have an approximate posterior distribution at $k-1$, $\hat{p}({\bf w}_{k-1}|y_{1:k-1}) =  \mathcal{N}({\bf w}_{k-1};\hat{\bf\boldsymbol\mu}_{k-1}, \hat{\sigma}_{k-1}^2 {\bf I} )$  Since all involved distributions are Gaussian, the predictive distribution
is obtained as %
\begin{eqnarray}
\hat{p}({\bf w}_k|y_{1:k-1}) &=& \int p({\bf w}_k|{\bf w}_{k-1}) \hat{p}({\bf w}_{k-1}|y_{1:k-1}) d{\bf w}_{k-1} \nonumber\\
&=& \mathcal{N}({\bf w}_k;{\bf\boldsymbol\mu}_{k|k-1}, \boldsymbol\Sigma_{k|k-1}), 
\label{eq:approx_pred}
\end{eqnarray}
where the mean vector and covariance matrix are given by
\begin{eqnarray}
\hat{\bf\boldsymbol\mu}_{k|k-1} &=& \hat{\bf\boldsymbol\mu}_{k-1} \nonumber \\
\hat{\boldsymbol\Sigma}_{k|k-1} &=& (\hat{\sigma}_{k-1}^2 + \sigma_d^2 ){\bf I}\nonumber 
\end{eqnarray}

From \eqref{eq:approx_pred}, the posterior distribution at time $k$ can be computed using Bayes' Theorem and standard Gaussian manipulations (see for instance \cite[Ch  4]{murphy2012machine})  Then, we approximate the posterior $p({\bf w}_k|y_{1:k})$ with an isotropic Gaussian,
\begin{equation}
\hat{p}({\bf w}_k|y_{1:k}) =  \mathcal{N}({\bf w}_k ; {\hat{\boldsymbol\mu}}_{k}, \hat{\sigma}_k^2 {\bf I} ),\nonumber
\end{equation}
where 
\begin{eqnarray}
{\hat{\boldsymbol\mu}}_{k} &= & {\hat{\boldsymbol\mu}}_{k-1}+ \frac{ (\hat{\sigma}_{k-1}^2+ \sigma_d^2)  }{(\hat{\sigma}_{k-1}^2+ \sigma_d^2)  \|{\bf x}_k\|^2 + \sigma_n^2} (y_k - {\bf x}_k^T {\hat{\boldsymbol\mu}}_{k-1}){\bf x}_k  \nonumber  \\
&=& {\hat{\boldsymbol\mu}}_{k-1}+ \eta_k (y_k - {\bf x}_k^T {\hat{\boldsymbol\mu}}_{k-1}){\bf x}_k   
\label{eq:prob_lms}
\end{eqnarray}
Note that, instead of a gain matrix ${\bf K}_k$ as in Eq ~\eqref{eq:prob_rls}, we now have a scalar gain $\eta_k$ that operates as a variable step size 


Finally, to obtain the posterior variance, which is our measure of uncertainty, we apply \eqref{eq:sigma_hat} and the trick ${\sf Tr}\{{\bf x}_k{\bf x}_k^T\}= {\bf x}_k^T{\bf x}_k= \|{\bf x}_k \|^2$,

\begin{eqnarray}
\hat{\sigma}_k^2 &=& \frac{{\sf Tr}(\boldsymbol\Sigma_k)}{M} \\
&=& \frac{1}{M}{\sf Tr}\left\{ \left( {\bf I} -  \eta_k {\bf x}_k {\bf x}_k^T \right) (\hat{\sigma}_{k-1}^2 +\sigma_d^2)\right\} \\
&=& \left(1 - \frac{\eta_k \|{\bf x}_k\|^2}{M}\right)(\hat{\sigma}_{k-1}^2 +\sigma_d^2) 
\label{eq:sig_k}
\end{eqnarray}
If MAP estimation is performed, we obtain  an adaptable step-size LMS estimation

\begin{equation}
{\bf w}_{k}^{(LMS)} = {\bf w}_{k-1}^{(LMS)} + \eta_k (y_k - {\bf x}_k^T {\bf w}_{k-1}^{(LMS)}){\bf x}_k, 	
\label{eq:lms}
\end{equation}
with
\begin{equation}
\eta_k = \frac{ (\hat{\sigma}_{k-1}^2+ \sigma_d^2)  }{(\hat{\sigma}_{k-1}^2+ \sigma_d^2)  \|{\bf x}_k\|^2 + \sigma_n^2} \nonumber
\end{equation}
At this point, several interesting remarks can be made:

\begin{itemize}

\item The adaptive rule \eqref{eq:lms} has linear complexity since it does not require us to compute the full matrix $\boldsymbol\Sigma_k$ 

\item For a stationary model, we have $\sigma_d^2=0$ in \eqref{eq:prob_lms} and \eqref{eq:sig_k}  In this case, the algorithm remains valid and both the step size and the error variance, $\hat{\sigma}_{k}$, vanish over time $k$  

\item Finally, the proposed adaptable step-size LMS has only two parameters, $\sigma_d^2$ and $\sigma_n^2$, (and only one, $\sigma_n^2$, in stationary scenarios) in contrast to other variable step-size algorithms \cite{kwong1992variable,aboulnasr1997robust,shin2004variable}  More interestingly, both $\sigma_d^2$ and $\sigma_n^2$ have a clear underlying physical meaning, and they can be estimated in many cases  We will comment more about this in the next section  
\end{itemize}



\section{Experiments}
\label{sec:experiments}

We evaluate the performance of the proposed algorithm in both stationary and tracking experiments  In the first experiment, we estimate a fixed vector ${\bf w}^{o}$ of dimension $M=50$  The entries of the vector are independently and uniformly chosen in the range $[-1,1]$  Then, the vector is normalized so that $\|{\bf w}^o\|=1$  Regressors $\boldsymbol{x}_{k}$ are zero-mean Gaussian vectors with identity covariance matrix  The additive noise variance is such that the SNR is $20$ dB  We compare our algorithm with standard RLS and three other LMS-based algorithms: LMS, NLMS \cite{sayed2008adaptive}, VSS-LMS \cite{shin2004variable} \footnote{The used parameters for each algorithm are: for RLS $\lambda=1$, $\epsilon^{-1}=0 01$; for LMS $\mu=0 01$; for NLMS $\mu=0 5$; and for VSS-LMS $\mu_{max}=1$, $\alpha=0 95$, $C=1e-4$ } The probabilistic LMS algorithm in \cite{park2014probabilistic} is not simulated because it is not suitable for stationary environments 

In stationary environments, the proposed algorithm has only one parameter, $\sigma^2_n$  We simulate both the scenario where we have perfectly knowledge of the amount of noise (probLMS1) and the case where the value $\sigma^2_n$ is $100$ times smaller than the actual value (probLMS2)  The Mean-Square Deviation (${\sf MSD} = {\mathbb E} \| {\bf w}_0 - {\bf w}_k \|^2$), averaged out over $50$ independent simulations, is presented in Fig  \ref{fig:msd_statationary} 



\begin{figure}[htb]
\centering
\begin{minipage}[b]{\linewidth}
  \centering
  \centerline{\includegraphics[width=\textwidth]{results_stationary_MSD}}
\end{minipage}
\caption{Performance in terms of MSD of probabilistic LMS with both optimal (probLMS1) and suboptimal (probLMS2) compared to LMS, NLMS, VS-LMS, and RLS }
\label{fig:msd_statationary}
\end{figure}

The performance of probabilistic LMS is close to RLS (obviously at a much lower computational cost) and largely outperforms previous variable step-size LMS algorithms proposed in the literature  Note that, when the model is stationary, i e  $\sigma^2_d=0$ in \eqref{eq:trans_eq},  both the uncertainty $\hat{\sigma}^2_k$, and the adaptive step size $\eta_k$, vanish over time  This implies that the error tends to zero when $k$ goes to infinity  Fig  \ref{fig:msd_statationary} also shows that the proposed approach is not very sensitive to a bad choice of its only parameter, as demonstrated by the good results of probLMS2, which uses a $\sigma^2_n$ that is $100$ times smaller than the optimal value  


\begin{figure}[htb]
\centering
\begin{minipage}[b]{\linewidth}
  \centering
  \centerline{\includegraphics[width=\textwidth]{fig2_final}}
\end{minipage}
\caption{Real part of one coefficient of the measured and estimated channel in experiment two  The shaded area represents two standard deviations from the prediction {(the mean of the posterior distribution)} }
\label{fig_2}
\end{figure}


\begin{table}[ht]
\begin{footnotesize}
\setlength{\tabcolsep}{2pt}
\def1 5mm{1 5mm}
\begin{center}
\begin{tabular}{|l@{\hspace{1 5mm}}|c@{\hspace{1 5mm}}|c@{\hspace{1 5mm}}|c@{\hspace{1 5mm}}|c@{\hspace{1 5mm}}|c@{\hspace{1 5mm}}|c@{\hspace{1 5mm}}|}
\hline
Method &  LMS &  NLMS & LMS-2013 & VSSNLMS & probLMS & RLS \\
\hline
\hline
MSD (dB) &-28 45 &-21 07 &-14 36 &-26 90 &-28 36 &-25 97\\
\hline                                                                     
\end{tabular}
\end{center}
\caption{Steady-state MSD of the different algorithms for the tracking of a real MISO channel }
\label{tab:table_MSD}
\end{footnotesize}

\end{table}
\newpage
In a second experiment, we test the tracking capabilities of the proposed algorithm with {real} data of a wireless MISO channel acquired in a realistic indoor scenario  More details on the setup can be found in \cite{gutierrez2011frequency}  Fig  \ref{fig_2} shows the real part of one of the channels, and the estimate of the proposed algorithm  The shaded area represents the estimated uncertainty for each prediction, i e  $\hat{\mu}_k\pm2\hat{\sigma}_k$  Since the experimental setup does not allow us to obtain the optimal values for the parameters, we fix these parameters to their values that optimize the steady-state mean square deviation (MSD)  \hbox{Table \ref{tab:table_MSD}} shows this steady-state MSD of the estimate of the MISO channel with different methods  As can be seen, the best tracking performance is obtained by standard LMS and the proposed method  





\section{Conclusions and Opened Extensions}
\label{sec:conclusions}

{We have presented a probabilistic interpretation of the least-mean-square filter  The resulting algorithm is an adaptable step-size LMS that performs well both in stationary and tracking scenarios  Moreover, it has fewer free parameters than previous approaches and these parameters have a clear physical meaning  Finally, as stated in the introduction, one of the advantages of having a probabilistic model is that it is easily extensible:}

\begin{itemize}
\item If, instead of using an isotropic Gaussian distribution in the approximation, we used a Gaussian with diagonal covariance matrix, we would obtain a similar algorithm with different step sizes and measures of uncertainty, for each component of ${\bf w}_k$  Although this model can be more descriptive, it needs more parameters to be tuned, and the parallelism with LMS vanishes 
\item Similarly, if we substitute the transition model of \eqref{eq:trans_eq} by an Ornstein-Uhlenbeck process, 

\begin{equation}
p({\bf w}_k|{\bf w}_{k-1})= \mathcal{N}({\bf w}_k;\lambda {\bf w}_{k-1}, \sigma_d^2), \nonumber
\label{eq:trans_eq_lambda}
\end{equation}
a similar algorithm is obtained but with a forgetting factor $\lambda$ multiplying ${\bf w}_{k-1}^{(LMS)}$ in \eqref{eq:lms}  This algorithm may have improved performance under such a kind of autoregresive dynamics of ${\bf w}_{k}$, though, again, the connection with standard LMS becomes dimmer 

\item As in \cite{park2014probabilistic}, the measurement model \eqref{eq:mess_eq} can be changed to obtain similar adaptive algorithms for classification, ordinal regression, and Dirichlet regression for compositional data  

\item A similar approximation technique could be applied to more complex dynamical models, i e  switching dynamical models \cite{barber2010graphical}  The derivation of efficient adaptive algorithms that explicitly take into account a switch in the dynamics of the parameters of interest is a non-trivial and open problem, though the proposed approach could be useful 

\item Finally, like standard LMS, this algorithm can be kernelized for its application in estimation under non-linear scenarios 

\end{itemize}


\begin{appendices}

\section{KL divergence between a general gaussian distribution and an isotropic gaussian}
\label{sec:kl}

 We want to approximate  $p_{{\bf x}_1}(x) = \mathcal{N}({\bf x}; \boldsymbol\mu_1,\boldsymbol\Sigma_1)$ by $p_{{\bf x}_2}({\bf x}) = \mathcal{N}({\bf x}; \boldsymbol\mu_2,\sigma_2^2 {\bf I})$  In order to do so, we have to compute the parameters of $p_{{\bf x}_2}({\bf x})$, $\boldsymbol\mu_2$ and $\sigma_2^2$, that minimize the following Kullback-Leibler divergence,

\begin{eqnarray}
D_{KL}(p_{{\bf x}_1}\| p_{{\bf x}_2}) &=&\int_{-\infty}^{\infty} p_{{\bf x}_1}({\bf x}) \ln{\frac{p_{{\bf x}_1}({\bf x})}{p_{{\bf x}_2}({\bf x})}}d{\bf x} \nonumber  \\
&= &  \frac{1}{2} \{ -M + {\sf Tr}(\sigma_2^{-2} {\bf I}\cdot \boldsymbol\Sigma_1^{-1})  \nonumber \\
  & &  + (\boldsymbol\mu_2 - \boldsymbol\mu_1 )^T \sigma^{-2}_2{\bf I} (\boldsymbol\mu_2 - \boldsymbol\mu_1 )  \nonumber \\
 & &   +  \ln \frac{{\sigma_2^2}^M}{\det\boldsymbol\Sigma_1} \}   
\label{eq:divergence}
\end{eqnarray}
Using symmetry arguments, we obtain 
\begin{equation}
\boldsymbol\mu_2^{*} =\arg \displaystyle{  \min_{\boldsymbol\mu_2}} \{ D_{KL}(p_{{\bf x}_1}\| p_{{\bf x}_2}) \} = \boldsymbol\mu_1 
\end{equation}
Then, \eqref{eq:divergence} gets simplified into 

\begin{eqnarray}
D_{KL}(p_{{\bf x}_1}\| p_{{\bf x}_2}) = \frac{1}{2}\lbrace { -M + {\sf Tr}(\frac{\boldsymbol\Sigma_1}{\sigma_2^{2}}) + \ln \frac{\sigma_2^{2M}}{\det\boldsymbol\Sigma_1}}\rbrace 
\end{eqnarray}
The variance $\sigma_2^2$ is computed in order to minimize this Kullback-Leibler divergence as

\begin{eqnarray}
\sigma_2^{2*} &=& \arg\min_{\sigma_2^2} D_{KL}(P_{x_1}\| P_{x_2}) \nonumber \\
 &=& \arg\min_{\sigma_2^2}\{ \sigma_2^{-2}{\sf Tr}\{\boldsymbol\Sigma_1\} + M\ln \sigma_2^{2} \}  
\end{eqnarray}
Deriving and making it equal zero leads to

\begin{equation}
\frac{\partial}{\partial \sigma_2^2} \left[ \frac{{\sf Tr}\{\boldsymbol\Sigma_1\}}{\sigma_2^{2}} + M \ln \sigma_2^{2} \right] = \left  {\frac{M}{\sigma_2^{2}}-\frac{{\sf Tr}\{\boldsymbol\Sigma_1\}}{(\sigma_2^{2})^2}}\right|_{\sigma_2^{2}=\sigma_2^{2*}}\left  =0 \right   
\nonumber
\end{equation}
Finally, since the divergence has a single extremum in $R_+$,
\begin{equation}
\sigma_2^{2*} = \frac{{\sf Tr}\{\boldsymbol\Sigma_1\}}{M} 
\end{equation}




\end{appendices}

\vfill
\clearpage

\bibliographystyle{IEEEbib}

Context 2: Paper Info

Title: Environmental variability and network structure determine the optimal plasticity mechanisms in embodied agents
Publish Date: Unkown
Author List: Sina Khajehabdollahi (from Department of Computer Science, University of Tübingen)

Figure

Figure2: An outline of the network controlling the foraging agent The sensor layer receives inputs at each time step (the ingredients of the nearest food), which are processed by the plastic layer in the same way as the static sensory network, Fig 1 The output of that network is given as input to the motor network, along with the distance d and angle α to the nearest food, the current velocity v, and energy E of the agent These signals are processed through two hidden layers to the final output of motor commands as the linear and angular acceleration of the agent
Figure4: The evolved parameters θ = (θ 1 ,      , θ 8 ) of the plasticity rule for the reward prediction (a ) and the decision (b ) tasks, for a variety of parameters (p tr = 0 01, d e ∈ 0, 0 1,      , 1, and σ ∈ 0, 0 1,      , 1 in all 100 combinations) Despite the relatively small difference between the tasks, the evolved learning rules differ considerably For visual guidance, the lines connect θs from the same run 
Figure5: a The trajectory of an agent (blue line) in the 2D environment A well-trained agent will approach and consume food with positive values (green dots) and avoid negative food (red dots) b The learning rate of the plastic sensory network eta p grows with the distance between environments d e c  and decreases with the frequency of environmental change d The fitness of an agent (measured as the total food consumed over its lifetime) increases over generations of the EA for both the scalar and binary readouts in the sensory network e The Pearson correlation coefficient of an evolved agent's weights with the ingredient value vector of the current environment (E 1 -blue, E 2 -red) In this example, the agent's weights are anti-correlated with its environment, which is not an issue for performance since the motor network can interpret the inverted signs of food 

abstract

The evolutionary balance between innate and learned behaviors is highly intricate, and different organisms have found different solutions to this problem  We hypothesize that the emergence and exact form of learning behaviors is naturally connected with the statistics of environmental fluctuations and tasks an organism needs to solve 
Here, we study how different aspects of simulated environments shape an evolved synaptic plasticity rule in static and moving artificial agents  We demonstrate that environmental fluctuation and uncertainty control the reliance of artificial organisms on plasticity  Interestingly, the form of the emerging plasticity rule is additionally determined by the details of the task the artificial organisms are aiming to solve 
Moreover, we show that coevolution between static connectivity and interacting plasticity mechanisms in distinct sub-networks changes the function and form of the emerging plasticity rules in embodied agents performing a foraging task  One of the defining features of living organisms is their ability to adapt to their environment and incorporate new information to modify their behavior 
It is unclear how the ability to learn first evolved , but its utility appears evident  Natural environments are too complex for all the necessary information to be hardcoded genetically and more importantly, they keep changing during an organism's lifetime in ways that cannot be anticipated ;   The link between learning and environmental uncertainty and fluctuation has been extensively demonstrated in both natural ; , and artificial environments  
Nevertheless, the ability to learn does not come without costs  For the capacity to learn to be beneficial in evolutionary terms, a costly nurturing period is often required, a phenomenon observed in both biological , and artificial organisms   Additionally, it has been shown that in some complex environments, hardcoded behaviors may be superior to learned ones given limits in the agent's lifetime and envi-ronmental uncertainty ; ;  
The theoretical investigation of the optimal balance between learned and innate behaviors in natural and artificial systems goes back several decades  However, it has recently found also a wide range of applications in applied AI systems ;   Most AI systems are trained for specific tasks, and have no need for modification after their training has been completed 
Still, technological advances and the necessity to solve broad families of tasks make discussions about life-like AI systems relevant to a wide range of potential application areas  Thus the idea of open-ended AI agents that can continually interact with and adapt to changing environments has become particularly appealing 
Many different approaches for introducing lifelong learning in artificial agents have been proposed  Some of them draw direct inspiration from actual biological systems ;   Among them, the most biologically plausible solution is to equip artificial neural networks with some local neural plasticity , similar to the large variety of synaptic plasticity mechanisms ; ; that performs the bulk of the learning in the brains of living organisms  
The artificial plasticity mechanisms can be optimized to modify the connectivity of the artificial neural networks toward solving a particular task  The optimization can use a variety of approaches, most commonly evolutionary computation  The idea of meta-learning or optimizing synaptic plasticity rules to perform specific functions has been recently established as an engineering tool that can compete with stateof-the-art machine learning algorithms on various complex tasks ; ; Pedersen and Risi (2021);  
Additionally, it can be used to reverse engineer actual plasticity mechanisms found in biological neural networks and uncover their functions ;   Here, we study the effect that different factors (environ-arXiv:2303 06734v1 [q-bio NC] 12 Mar 2023 mental fluctuation and reliability, task complexity) have on the form of evolved functional reward-modulated plasticity rules 
We investigate the evolution of plasticity rules in static, single-layer simple networks  Then we increase the complexity by switching to moving agents performing a complex foraging task  In both cases, we study the impact of different environmental parameters on the form of the evolved plasticity mechanisms and the interaction of learned and static network connectivity 
Interestingly, we find that different environmental conditions and different combinations of static and plastic connectivity have a very large impact on the resulting plasticity rules  We imagine an agent who must forage to survive in an environment presenting various types of complex food particles  Each food particle is composed of various amounts and combinations of N ingredients that can have positive (food) or negative (poison) values 
The value of a food particle is a weighted sum of its ingredients  To predict the reward value of a given resource, the agent must learn the values of these ingredients by interacting with the environment  The priors could be generated by genetic memory, but the exact values are subject to change  To introduce environmental variability, we stochastically change the values of the ingredients 
More precisely, we define two ingredient-value distributions E 1 and E 2 and switch between them, with probability p tr for every time step  We control how (dis)similar the environments are by parametrically setting E 2 = (1 − 2d e )E 1 , with d e ∈ [0, 1] serving as a distance proxy for the environments; when d e = 0, the environment remains unchanged, and when d e = 1 the value of each ingredient fully reverses when the environmental transition happens 
For simplicity, we take values of the ingredients in E 1 equally spaced between -1 and 1 (for the visualization, see Fig  )  The static agent receives passively presented food as a vector of ingredients and can assess its compound value using the linear summation of its sensors with the (learned or evolved) weights, see Fig   
The network consists of N sensory neurons that are projecting to a single post-synaptic neuron  At each time step, an input X t = (x 1 ,       , x N ) is presented, were the value x i , i ∈ {1,       , N } represents the quantity of the ingredient i  We draw x i independently form a uniform distribution on the [0, 1] interval (x i ∼ U (0, 1)) 
The value of each ingredient w c i is determined by the environment (E 1 or E 2 )  The postsynaptic neuron outputs a prediction of the food X t value as y t = g(W X T t )  Throughout the paper, g will be either the identity function, in which case the prediction neuron is linear, or a step-function; however, it could be any other nonlinearity, such as a sigmoid or ReLU 
After outputting the prediction, the neuron receives feedback in the form of the real value of the input R t   The real value is computed as R t = W c X T t + ξ, where W c = (w c 1 ,       , w c N ) is the actual value of the ingredients, and ξ is a term summarizing the noise of reward and sensing system ξ ∼ N (0, σ) 
Figure : An outline of the static agent's network  The sensor layer receives inputs representing the quantity of each ingredient of a given food at each time step  The agent computes the prediction of the food's value y t and is then given the true value R t ; it finally uses this information in the plasticity rule to update the weight matrix 
For the evolutionary adjustment of the agent's parameters, the loss of the static agent is the sum of the mean squared errors (MSE) between its prediction y t and the reward R t over the lifetime of the agent  The agent's initial weights are set to the average of the two ingredient value distributions, which is the optimal initial value for the case of symmetric switching of environments that we consider here 
As a next step, we incorporate the sensory network of static agents into embodied agents that can move around in an environment scattered with food  To this end, we merge the static agent's network with a second, non-plastic motor network that is responsible for controlling the motion of the agent in the environment 
Specifically, the original plastic network now provides the agent with information about the value of the nearest food  The embodied agent has additional sensors for the distance from the nearest food, the angle between the current velocity and the nearest food direction, its own velocity, and its own energy level (sum of consumed food values) 
These inputs are processed by two hidden layers (of 30 and 15 neurons) with tanh activation  The network's outputs are angular and linear acceleration, Fig    The embodied agents spawn in a 2D space with periodic boundary conditions along with a number of food particles that are selected such that the mean of the food value distribution is ∼ 0  An agent can eat food by approaching it sufficiently closely, and each time a food particle is eaten, it is The sensor layer receives inputs at each time step (the ingredients of the nearest food), which are processed by the plastic layer in the same way as the static sensory network, Fig   
The output of that network is given as input to the motor network, along with the distance d and angle α to the nearest food, the current velocity v, and energy E of the agent  These signals are processed through two hidden layers to the final output of motor commands as the linear and angular acceleration of the agent re-spawned with the same value somewhere randomly on the grid (following the setup of ) 
After 5000 time steps, the cumulative reward of the agent (the sum of the values of all the food it consumed) is taken as its fitness  During the evolutionary optimization, the parameters for both the motor network (connections) and plastic network (learning rule parameters) are co-evolved, and so agents must simultaneously learn to move and discriminate good/bad food 
Reward-modulated plasticity is one of the most promising explanations for biological credit assignment   In our network, the plasticity rule that updates the weights of the linear sensor network is a rewardmodulated rule which is parameterized as a linear combination of the input, the output, and the reward at each time step:
Additionally, after each plasticity step, the weights are normalized by mean subtraction, an important step for the stabilization of Hebbian-like plasticity rules   We use a genetic algorithm to optimize the learning rate η p and amplitudes of different terms θ = (θ 1 ,       , θ 8 )  The successful plasticity rule after many food presentations must converge to a weight vector that predicts the correct food values (or allows the agent to correctly decide whether to eat a food or avoid it) 
To have comparable results, we divide θ = (θ 1 ,       , θ 8 ) by We then multiply the learning rate η p with θ max to maintain the rule's evolved form unchanged, η norm p = η p • θ max   In the following, we always use normalized η p and θ, omitting norm   To evolve the plasticity rule and the moving agents' motor networks, we use a simple genetic algorithm with elitism  
The agents' parameters are initialized at random (drawn from a Gaussian distribution), then the sensory network is trained by the plasticity rule and finally, the agents are evaluated  After each generation, the bestperforming agents (top 10 % of the population size) are selected and copied into the next generation 
The remaining 90 % of the generation is repopulated with mutated copies of the best-performing agents  We mutate agents by adding independent Gaussian noise (σ = 0 1) to its parameters  To start with, we consider a static agent whose goal is to identify the value of presented food correctly  The static reward-prediction network quickly evolves the parameters of the learning rule, successfully solving the prediction task 
We first look at the evolved learning rate η p , which determines how fast (if at all) the network's weight vector is updated during the lifetime of the agents  We identify three factors that control the learning rate parameter the EA converges to: the distance between the environments, the noisiness of the reward, and the rate of environmental transition 
The first natural factor is the distance d e between the two environments, with a larger distance requiring a higher learning rate, Fig    This is an expected result since the convergence time to the "correct" weights is highly dependent on the initial conditions  If an agent is born at a point very close to optimality, which naturally happens if the environments are similar, the distance it needs to traverse on the fitness landscape is small 
Therefore it can afford to have a small learning rate, which leads to a more stable convergence and is not affected by noise  A second parameter that impacts the learning rate is the variance of the rewards  The reward an agent receives for the plasticity step contains a noise term ξ that is drawn from a zero mean Gaussian distribution with standard deviation σ 
This parameter controls the unreliability of the agent's sensory system, i e , higher σ means that the information the agent gets about the value of the foods it consumes cannot be fully trusted to reflect the actual value of the foods  As σ increases, the learning rate η p decreases, which means that the more unreliable an environment becomes, the less an agent relies on plasticity to update its weights, Fig   
Indeed for some combinations of relatively small distance d e and high reward variance σ, the EA converges to a learning rate of η p ≈ 0  This means that the agent opts to have no adaptation during its lifetime and remain at the mean of the two environments  It is an optimal solution when the expected loss due to ignoring the environmental transitions is, on average, lower than the loss the plastic network will incur by learning via the (often misleading because of the high σ) environmental cues 
A final factor that affects the learning rate the EA will converge to is the frequency of environmental change during an agent's lifetime  Since the environmental change is modeled as a simple, two-state Markov process (Fig  ), the control parameter is the transition probability p tr   When keeping everything else the same, the learning rate rapidly rises as we increase the transition probability from 0, and after reaching a peak, it begins to decline slowly, eventually reaching zero (Fig  ) 
This means that when environmental transition is very rare, agents opt for a very low learning rate, allowing a slow and stable convergence to an environment-appropriate weight vector that leads to very low losses while the agent remains in that environment  As the rate of environmental transition increases, faster learning is required to speed up convergence in order to exploit the (comparatively shorter) stays in each environment 
Finally, as the environmental transition becomes too fast, the agents opt for slower or even no learning, which keeps them ) and the decision (b ) tasks, for a variety of parameters (p tr = 0 01, d e ∈ 0, 0 1,       , 1, and σ ∈ 0, 0 1,       , 1 in all 100 combinations)  Despite the relatively small difference between the tasks, the evolved learning rules differ considerably 
For visual guidance, the lines connect θs from the same run  near the middle of the two environments, ensuring that the average loss of the two environments is minimal (Fig  )  The form of the evolved learning rule depends on the task: Decision vs  Prediction The plasticity parameters θ = (θ 1 ,       , θ 8 ) for the rewardprediction task converge on approximately the same point, regardless of the environmental parameters (Fig  ) 
In particular, θ 3 → 1, θ 5 → −1, θ i → 0 for all other i, and thus the learning rule converges to: Since by definition y t = g(W t X T t ) = W t X T t (g(x) = x in this experiment) and R t = W c X T t + ξ we get: Thus the distribution of ∆W t converges to a distribution with mean 0 and variance depending on η p and σ and W converges to W c  
So this learning rule will match the agent's weight vector with the vector of ingredient values in the environment  We examine the robustness of the learning rule the EA discovers by considering a slight modification of our task  Instead of predicting the expected food value, the agent now needs to decide whether to eat the presented food or not 
This is done by introducing a step-function nonlinearity (g(x) = 1 if x ≥ 1 and 0 otherwise)  Then the output y(t) is computed as: Instead of the MSE loss between prediction and actual value, the fitness of the agent is now defined as the sum of the food values it chose to consume (by giving y t = 1)  Besides these two changes, the setup of the experiments remains exactly the same 
The qualitative relation between η p and parameters of environment d e , σ and p tr is preserved in the changed experiment  However, the resulting learning rule is significantly different (Fig  )  The evolution converges to the following learning rule: In both cases, the rule has the form ∆W t = η p X t [α y R t + β y ] 
Thus, the ∆W t is positive or negative depending on whether the reward R t is above or below a threshold (γ = −β y /α y ) that depends on the output decision of the network (y t = 0 or 1)  Both learning rules (for the reward-prediction and decision tasks) have a clear Hebbian form (coordination of preand post-synaptic activity) and use the incoming reward signal as a threshold 
These similarities indicate some common organizing principles of reward-modulated learning rules, but their significant differences highlight the sensitivity of the optimization process to task details  We now turn to the moving embodied agents in the 2D environment  To optimize these agents, both the motor network's connections and the sensory network's plasticity parameters evolve simultaneously 
Since the motor network is initially random and the agent has to move to find food, the number of interactions an agent experiences in its lifetime can be small, slowing down the learning  However, having the larger motor network also has benefits for evolution because it allows the output of the plastic network to be read out and transformed in different ways, resulting in a broad set of solutions 
The fitness of an agent (measured as the total food consumed over its lifetime) increases over generations of the EA for both the scalar and binary readouts in the sensory network  e  The Pearson correlation coefficient of an evolved agent's weights with the ingredient value vector of the current environment (E 1 -blue, E 2 -red) 
In this example, the agent's weights are anti-correlated with its environment, which is not an issue for performance since the motor network can interpret the inverted signs of food  The agents can solve the task effectively by evolving a functional motor network and a plasticity rule that converges to interpretable weights (Fig  ) 
After ∼ 100 evolutionary steps (Fig  ), the agents can learn the ingredient value distribution using the plastic network and reliably move towards foods with positive values while avoiding the ones with negative values  We compare the dependence of the moving and the static agents on the parameters of the environment: d e and the state transition probability p tr  
At first, in order to simplify the experiment, we set the transition probability to 0, but fixed the initial weights to be the average of E 1 and E 2 , while the real state is E 2   In this experiment, the distance between states d e indicates twice the distance between the agent's initial weights and the optimal weights (the environment's ingredient values) since the agent is initialized at the mean of the two environment distributions 
Same as for the static agent, the learning rate increases with the distance d e (Fig  )  Then, we examine the effect of the environmental transition probability p tr on the evolved learning rate η p   In order for an agent to get sufficient exposure to each environment, we scale down the probability p tr from the equivalent experiment for the static agents 
We find that as the probability of transition increases, the evolved learning rate η p decreases (Fig  )  This fits with the larger trend for the static agent, although there is a clear difference when it comes to the increase for very small transition probabil-ities that were clearly identifiable in the static but not the moving agents 
This could be due to much sparser data and possibly the insufficiently long lifetime of the moving agent (the necessity of scaling makes direct comparisons difficult)  Nevertheless, overall we see that the associations observed in the static agents between environmental distance d e and transition probability p tr and the evolved learning rate η p are largely maintained in the moving agents 
Still, more data would be needed to make any conclusive assertions about the exact effect of these environmental parameters on the emerging plasticity mechanisms  A crucial difference between the static and the moving agents is the function the plasticity has to perform  While in the static agents, the plasticity has to effectively identify the exact value distribution of the environment in order to produce accurate predictions, in the embodied agents, the plasticity has to merely produce a representation of the environment that the motor network can evolve to interpret adequately enough to make decisions about which food to consume 
To illustrate the difference, we plot the Pearson correlation coefficient between an agent's weights and the ingredient values of the environment it is moving in (Fig  )  We use the correlation instead of the MSE loss (which we used for the static agents in Fig  ) because the amplitude of the vector varies a lot for different agents and meaningful The evolved parameters of moving agents' plasticity rule for the g(s) = x, identity (a ) and the step function (Eq 
4) (b ) sensory networks (the environmental parameters here are d e ∈ [0, 1], σ = 0 and p tr = 0 001)  The step function (binary output) network evolved a more structured plasticity rule (e g , θ 3 > 0 for all realizations) than the linear network  Moreover, the learned weights for the identity network (c ) have higher variance and correlate significantly less with the environment's ingredient distribution compared to the learned weights for the thresholded network (d )
conclusions cannot be drawn from the MSE loss  For many agents, the learned weights are consistently anti-correlated with the actual ingredient values (an example of such an agent is shown in Fig  )  This means that the output of the sensory network will have the opposite sign from the actual food value 
While in the static network, this would lead to very bad predictions and high loss, in the foraging task, these agents perform exactly as well as the ones where the weights and ingredients values are positively correlated, since the motor network can simply learn to move towards food for which it gets a negative instead of a positive sensory input 
This additional step of the output of the plastic network going through the motor network before producing any behavior has a strong effect on the plasticity rules that the embodied agents evolve  Specifically, if we look at the emerging rules the top performing agents have evolved (Fig  ), it becomes clear that, unlike the very well-structured rules of the static agents (Fig  ), there is now virtually no discernible pattern or structure 
The difference becomes even clearer if we look at the learned weights (at the end of a simulation) of the best-performing agents (Fig  )  While there is some correlation with the environment's ingredient value distribution, the variance is very large, and they do not seem to converge on the "correct" values in any way 
This is to some extent expected since, unlike the static agents where the network's output has to be exactly correct, driving the evolution of rules that converge to the precise environmental distribution, in the embodied networks, the bulk of the processing is done by the motor network which can evolve to interpret the scalar value of the sensory network's output in a variety of ways 
Thus, as long as the sensory network's plasticity rule co-evolves with the motor network, any plasticity rule that learns to produce consistent information about the value of encountered food can potentially be selected  To further test this assumption, we introduce a bottleneck of information propagation between the sensory and motor networks by using a step-function nonlinearity on the output of the sensory network (Eq 
4)  Similarly to the decision task of the static network, the output of the sensory network now becomes binary  This effectively reduces the flow of information from the sensory to the motor network, forcing the sensory network to consistently decide whether food should be consumed (with the caveat that the motor network can still interpret the binary sign in either of two ways, either consuming food marked with 1 or the ones marked with 0 by the sensory network) 
The agents perform equally well in this variation of the task as before (Fig  ), but now, the evolved plasticity rules seem to be more structured (Fig  )  Moreover, the variance of the learned weights in the bestperforming agents is significantly reduced (Fig  ), which indicates that the bottleneck in the sensory network is in-creasing selection pressure for rules that learn the environment's food distribution accurately 
We find that different sources of variability have a strong impact on the extent to which evolving agents will develop neuronal plasticity mechanisms for adapting to their environment  A diverse environment, a reliable sensory system, and a rate of environmental change that is neither too large nor too small are necessary conditions for an agent to be able to effectively adapt via synaptic plasticity 
Additionally, we find that minor variations of the task an agent has to solve or the parametrization of the network can give rise to significantly different plasticity rules  Our results partially extend to embodied artificial agents performing a foraging task  We show that environmental variability also pushes the development of plasticity in such agents 
Still, in contrast to the static agents, we find that the interaction of a static motor network with a plastic sensory network gives rise to a much greater variety of wellfunctioning learning rules  We propose a potential cause of this degeneracy; as the relatively complex motor network is allowed to read out and process the outputs from the plastic network, any consistent information coming out of these outputs can be potentially interpreted in a behaviorally useful way 
Reducing the information the motor network can extract from the sensory system significantly limits learning rule variability  Our findings on the effect of environmental variability concur with the findings of previous studies that have identified the constraints that environmental variability places on the evolutionary viability of learning behaviors 
We extend these findings in a mechanistic model which uses a biologically plausible learning mechanism (synaptic plasticity)  We show how a simple evolutionary algorithm can optimize the different parameters of a simple reward-modulated plasticity rule for solving simple prediction and decision tasks 
Reward-modulated plasticity has been extensively studied as a plausible mechanism for credit assignment in the brain ; ; and has found several applications in artificial intelligence and robotics tasks ;   Here, we demonstrate how such rules can be very well-tuned to take into account different environmental parameters and produce optimal behavior in simple systems 
Additionally, we demonstrate how the co-evolution of plasticity and static functional connectivity in different subnetworks fundamentally changes the evolutionary pressures on the resulting plasticity rules, allowing for greater diversity in the form of the learning rule and the resulting learned connectivity 
Several studies have demonstrated how, in biological networks, synaptic plasticity heavily interacts with and is driven by network topology   Moreover, it has been recently demonstrated that biological plasticity mechanisms are highly redundant in the sense that any observed neural connectivity or recorded activity can be achieved with a variety of distinct, unrelated learning rules  
This observed redundancy of learning rules in biological settings complements our results and suggests that the function of plasticity rules cannot be studied independently of the connectivity and topology of the networks they are acting on  The optimization of functional plasticity in neural networks is a promising research direction both as a means to understand biological learning processes and as a tool for building more autonomous artificial systems 
Our results suggest that reward-modulated plasticity is highly adaptable to different environments and can be incorporated into larger systems that solve complex tasks  This work studies a simplified toy model of neural network learning in stochastic environments  Future work could be built on this basic framework to examine more complex reward distributions and sources of environmental variability 
Moreover, a greater degree of biological realism could be added by studying more plausible network architectures (multiple plastic layers, recurrent and feedback connections) and more sophisticated plasticity rule parametrizations  Additionally, our foraging simulations were constrained by limited computational resources and were far from exhaustive 
Further experiments can investigate environments with different constraints, food distributions, multiple seasons, more complex motor control systems and interactions of those systems with different sensory networks as well as the inclusion of plasticity on the motor parts of the artificial organisms 
Context 3: Sir Simon William English  (born 30 December 1961) is a New Zealand former National Party politician who served as the 39th prime minister of New Zealand from 2016 to 2017  He had previously served as the 17th deputy prime minister of New Zealand and minister of finance from 2008 to 2016 under John Key and the Fifth National Government 

A farmer and public servant before entering politics, English was elected to the New Zealand Parliament in  as the National Party's candidate in the Wallace electorate  He was elevated to Cabinet in 1996 and in 1999 was made minister of finance, although he served for less than a year due to his party's loss at the 1999 general election  In October 2001, English replaced Jenny Shipley as the leader of the National Party (and consequently as Leader of the Opposition)  He led the party to its worst defeat at the 2002 general election, and as a consequence, in October 2003 he was replaced as leader by Don Brash 

In November 2006, after Brash's resignation, English became deputy leader under John Key  After National's victory at the 2008 general election, he became deputy prime minister and was also made minister of finance for the second time  Under English's direction New Zealand's economy maintained steady growth during National's three terms of government  He became a list-only MP after stepping down as an electorate MP at the 2014 general election 

John Key resigned as leader of the National Party and prime minister in December 2016  English won the resulting leadership election unopposed and was sworn in as prime minister on 12 December 2016  His tenure was only ten months, and included a three-month election campaign  In the 2017 general election, National won the largest number of seats but fell short of a majority  The parties holding the balance of power declined to support the existing government, and English was subsequently replaced as prime minister by Jacinda Ardern, leader of the Labour Party  English initially continued on as Leader of the Opposition, but resigned as leader of the National Party on 27 February 2018 and left parliament two weeks later 

Early life
English was born on 30 December 1961 at Lumsden Maternity Centre in Lumsden  He is the eleventh of twelve children of Mervyn English and Norah (née O'Brien) English  His parents purchased Rosedale, a mixed sheep and cropping farm in Dipton, Southland from Mervyn's uncle, Vincent English, a bachelor, in 1944  English was born in the maternity unit at Lumsden 

English attended St Thomas's School in Winton, then boarded at St  Patrick's College in Upper Hutt, where he became head boy  He played in the first XV of the school's rugby team  English went on to study commerce at the University of Otago, where he was a resident at Selwyn College, and then completed an honours degree in English literature at Victoria University of Wellington 

After finishing his studies, English returned to Dipton and farmed for a few years  From 1987 to 1989, he worked in Wellington as a policy analyst for the New Zealand Treasury, at a time when the free market policies favoured by Labour's finance minister Roger Douglas (known collectively as "Rogernomics") were being implemented 

English joined the National Party in 1980, while at Victoria University  He served for a period as chairman of the Southland branch of the Young Nationals, and became a member of the Wallace electorate committee  After moving to Wellington, he served for periods on the Island Bay and Miramar electorate committees, respectively 

Fourth National Government (1990–1999)

At the 1990 general election, English stood as the National candidate in Wallace, replacing the retiring Derek Angus, and was elected with a large majority  He would hold this seat, renamed Clutha-Southland in 1996, until 2014  He and three other newly elected National MPs (Tony Ryall, Nick Smith, and Roger Sowry) were soon identified as rising stars in New Zealand politics, and at various points were dubbed the "brat pack", the "gang of four", and the "young Turks"  In his first term in parliament, English chaired a select committee into social services  He was made a parliamentary under-secretary in 1993, serving under the Minister of Health 

First period in cabinet (1996–1999)
In early 1996, English was elevated to cabinet by Prime Minister Jim Bolger, becoming the Minister for Crown Health Enterprises and Associate Minister of Education (to Wyatt Creech)  He was 34 at the time, becoming the cabinet's youngest member  After the 1996 general election, the National Party was forced into a coalition with New Zealand First to retain government  In the resulting cabinet reshuffle, English emerged as Minister of Health  However, as a condition of the coalition agreement, NZ First's Neil Kirton (a first-term MP) was made Associate Minister of Health, effectively becoming English's deputy  This arrangement was described in the press as a "shotgun marriage", and there were frequent differences of opinion between the two ministers  After their relationship became unworkable, Kirton was sacked from the role in August 1997, with the agreement of NZ First leader Winston Peters 

As Minister of Health, English was responsible for continuing the reforms to the public health system that National had begun after the 1990 general election  The reforms were unpopular, and health was perceived as one of the government's weaknesses, with the health portfolio consequently being viewed as a challenge  English believed that the unpopularity of the reforms was in part due to a failure in messaging, and encouraged his National colleagues to avoid bureaucratic and money-focused language (such as references to "balance sheets" and "user charges") and instead talk about the improvements to services the government's reforms would bring  He also rejected the idea that public hospitals could be run as commercial enterprises, a view which some of his colleagues had previously promoted 

By early 1997, as dissatisfaction with Bolger's leadership began to grow, English was being touted as a potential successor, along with Jenny Shipley and Doug Graham  His age (35) was viewed as the main impediment to a successful leadership run  National's leadership troubles were resolved in December 1997, when Bolger resigned and Shipley was elected to the leadership unopposed  English had been a supporter of Bolger as leader, but Shipley reappointed him Minister of Health in her new cabinet 

English was promoted to Minister of Finance in a reshuffle in January 1999, a position which was at the time subordinate to the Treasurer, Bill Birch  After a few months, the pair switched positions as part of Birch's transition to retirement, with English assuming the senior portfolio  In early interviews, he emphasised his wish to be seen as a pragmatist rather than an ideologue, and said that the initiatives of some of his predecessors (Roger Douglas's "Rogernomics" and Ruth Richardson's "Ruthanasia") had focused on "fruitless, theoretical debates" when "people just want to see problems solved" 

Opposition (1999–2008)

After the National Party lost the 1999 election to Helen Clark's Labour Party, English continued on in the shadow cabinet as National's spokesperson for finance  He was elected deputy leader of the party in February 2001, following the resignation of Wyatt Creech, with Gerry Brownlee being his unsuccessful opponent



ground_truth: This research paper proposed an approach based on approximating the posterior distribution with an isotropic Gaussian distribution.

prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: How does the transition probability of the environment affect the learning rate in the static agent?

Read the following text and answer briefly.

Context 3: 

Leader of the Opposition
In October 2001, after months of speculation, Jenny Shipley resigned as leader of the National Party after being told she no longer had the support of the party caucus  English was elected as her replacement unopposed (with Roger Sowry as his deputy), and consequently became Leader of the Opposition  However, he did not openly organise against Shipley, and according to The Southland Times "there was almost an element of 'aw, shucks, I'll do it then' about Mr English's ascension" 

Aged 39 when he was elected, English became the second-youngest leader in the National Party's history, after Jim McLay (who was 38 when elected in 1984)  He also became only the third Southlander to lead a major New Zealand political party, after Joseph Ward and Adam Hamilton  However, English failed to improve the party's performance  In the 2002 election, National suffered its worst electoral defeat ever, gaining barely more than twenty percent of the vote  English described it as "the worst day of my political life"  Both party insiders and the general public were split as to how much to blame him for the loss, but most of the party believed that English would be able to rebuild National's support 

By late 2003, however, National's performance in opinion polls remained poor  The party had briefly increased its popularity in the year following the election, but by October its support had fallen to levels only slightly better than what it achieved in the last ballot  English also appeared in a boxing match for a charity against entertainer Ted Clarke  This did not boost his polling or that of the National party either, with suggestions that it devalued his image as a serious politician  Don Brash, former governor of the Reserve Bank and a relative newcomer to politics, began to build up support to replace English  On 28 October, Brash gained sufficient backing in Caucus to defeat English in a leadership contest 

Shadow cabinet roles and deputy leader
On 2 November 2003, when Brash changed responsibilities for certain MPs, English became National's spokesman for education, ranked at fifth place in the party's parliamentary hierarchy  He remained in parliament after the 2005 election  In his new shadow education portfolio, English performed strongly, and remained a party favourite despite his election defeat as leader in 2002, eventually being returned to the finance portfolio in August 2004 as deputy spokesman (while still retaining responsibility for education) 

In November 2006, Brash resigned as leader  English was considered as a potential replacement leader (running against John Key) or deputy leader (against incumbent Gerry Brownlee) in the ensuing leadership election  However, a contest was avoided when the MPs agreed a Key/English ticket would run unopposed in a display of party unity  English took over the deputy leadership and the finance portfolio in the Key shadow cabinet 

Fifth National Government (2008–2017)

Deputy Prime Minister and Minister of Finance (2008–2016)

At the 2008 election, English was re-elected by his electorate, winning by a margin of about 15,500 votes  He became Deputy Prime Minister of New Zealand and Minister of Finance in the fifth National Government, being sworn into office on 19 November 2008 and continued to serve in those roles until becoming Prime Minister on 12 December 2014  He was also made Minister of Infrastructure in National's first term of government and Minister responsible for Housing New Zealand Corporation and minister responsible for the New Zealand flag consideration process in its third 

He was comfortably re-elected in Clutha-Southland in the 2011 election but opted to run as a party-list candidate in 2014  

The pairing of John Key as leader of the National Party and English as his deputy has been compared to that of Bob Hawke and Paul Keating (in Australia) and Tony Blair and Gordon Brown (in the UK) 

English acceded to the role of Finance Minister in the continuing wake of the financial crisis  In response to New Zealand's rising debt, English made budget deficit-reduction his main priority  His first budget outlined three focuses in New Zealand's financial recovery: "improving the business environment and removing roadblocks to growth; investment in productive infrastructure; and improving the way government works"  One of his first acts was creating the National Infrastructure Unit, charged with formulating a plan for infrastructure projects and investments  He commissioned a government-wide spending review, with an aim to reducing government expenditure—with the exceptions of a two-year stimulus package and long-term increases on infrastructure spending 

In April 2011, the Opposition criticised English for suggesting that New Zealand businesses could use New Zealand's low wages to help it compete with Australia  The National Government campaigned for re-election in 2011 on its economic record  The Government boasted growth for five consecutive quarters up to mid-2010, totalling 1 6% of real GDP 

Strong growth resulted in a surplus of $473 million for the 2015/16 financial year, projected to rise to $8 5 billion by 2020/21  In his 2016 Economic and Fiscal Update address, English stated that reducing debt and tackling the costs of the 2016 Kaikōura earthquake were higher priorities than reducing rates of tax 

Allowances issue
In 2009, the media, including TVNZ and TV3 revealed that English was receiving about NZ$900 a week as part of a living allowance for ministers, to live in his own NZ$1 2 million Wellington home  At the time, English also received $276,200 in his annual salary as Deputy Prime Minister  It was also revealed other ministers with homes in the capital city were also claiming accommodation allowances  On 3 August 2009, Prime Minister John Key started a review of the housing allowances claimed by cabinet ministers  English subsequently paid back $12,000 and only claimed about $24,000 a year in living allowances  The Auditor-General's office said in September 2009 that they were making "preliminary enquiries" into parliamentary housing expenses in response to a letter of complaint from Progressive party leader Jim Anderton  Two days later English stated that he would no longer take up any housing allowance and had paid back all the allowance he had received since the November 2008 election 

Prime Minister (2016–2017)

John Key resigned on 12 December, and endorsed English as his successor in the resulting leadership election  Following the drop-out of both Judith Collins and Jonathan Coleman from the leadership election, English was sworn in as the 39th Prime Minister of New Zealand on 12 December 2016 

English appointed his first cabinet on 18 December  In a reshuffle, he appointed Steven Joyce to succeed him as Finance Minister, while most ministerial portfolios remained the same 

In February 2017, English did not attend Waitangi Day commemorations at the historic treaty grounds, reportedly in response to the Ngāpuhi iwi's decision to stop the Prime Minister from speaking at the marae  Ngāpuhi have protested the Government's negotiation of the Trans Pacific Partnership Agreement (TPPA), which the iwi believe infringes upon Māori sovereignty, and thus does not adhere to the Treaty of Waitangi  English had been invited to attend in an official capacity; his non-attendance was criticised by a Ngāpuhi elder and Opposition leader Andrew Little



ground_truth: As the transition probability increases, the learning rate initially rises and then declines.

prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: When did Simon English become the leader of the National Party?

Read the following text and answer briefly.

Context 3: 

In his first overseas trip as Prime Minister, English travelled to Europe to discuss trade ties, including a prospective New Zealand–European Union free trade agreement  He first travelled to London on 13 January 2017 to meet British Prime Minister Theresa May  Discussing trade relations, English said the two nations were "natural partners" and would "continue to forge ties" after the UK's withdrawal from the EU  He also arranged to meet with London Mayor Sadiq Khan, Belgian Prime Minister Charles Michel and German Chancellor Angela Merkel   In a meeting with Merkel, English received crucial backing from Germany for a trade deal with the EU  On 16 January, English stated that his government would continue to promote TPPA, despite the United States' decision to withdraw from the agreement  He explained that Southeast Asian countries would now be treated as a priority in negotiations—he also asserted that the United States was ceding influence to China by its rejection of the trade pact 

At a press conference at the Beehive on 1 February 2017, English announced that the 2017 general election would be held on 23 September  The Prime Minister later confirmed that his party would approach ACT, United Future and the Māori Party if confidence and supply agreements were required to form a government following the election  In his second cabinet reshuffle on 24 April, English appointed Gerry Brownlee as his new Foreign Affairs Minister; he also promoted Nikki Kaye to the portfolio of Education Minister, and moved Mark Mitchell into the cabinet to become Defence Minister  The reshuffle was perceived as an election preparation 

On 13 February 2017, English welcomed Australian Prime Minister Malcolm Turnbull to Wellington  The two leaders reaffirmed their shared trade agenda, and discussed changes to the Australian citizenship pathway which will affect permanent residents originating from New Zealand 

On 19 June, it was reported that Todd Barclay, who succeeded English as MP for Clutha-Southland, had clandestinely recorded one of his employee's conversations the previous year, and that John Key's leaders' budget was used to pay a confidential settlement after the employee resigned  English admitted that he had been aware of the illegal recording and the settlement, and thus implicated in the scandal 

During the 2017 National campaign launch, English introduced a $379 million social investment package including digital learning academies for high school students, more resources for mathematics, and boosting support for teaching second languages in schools, and maintaining National Standards in the school curriculum  Prime Minister English also sought to defend National's financial management and economic track record and claimed that the opposition Labour Party would raise taxes  Early opinion polling had forecast a poor showing in the election for the Labour Party, but in early August 37-year-old Jacinda Ardern took over as Labour leader and seemingly energised younger voters 

At the 2017 general election, National won the largest share of the party vote (44 4%) and the largest number of seats (56) in the House Representatives  However, National lacked enough seats to govern alone due to two of the party's support partners, the Māori Party and United Future, losing their parliamentary seats  In response, English stated that the party would be entering into talks to form a coalition with New Zealand First  Following talks with the two largest parties, New Zealand First entered a coalition arrangement with the Labour Party  English was succeeded as prime minister by Jacinda Ardern on 26 October 

Opposition (2017–2018)

Leader of the Opposition
English was re-elected as National Party leader on 24 October 2017  At the time of his re-election, English announced his intention to stay on as leader until the next general election  On 13 February 2018, however, he stood down as National Party leader due to personal reasons, and instructed the party to put into motion the processes to elect a new leader  He also retired from Parliament  English's resignation followed weeks of speculation that he would step aside for a new leader  On 27 February, he was succeeded as party leader by Simon Bridges as the result of the leadership election held that day 

Post-premiership 
In 2018, English joined the board of Australian conglomerate, Wesfarmers  English serves in Chairmanships of Mount Cook Alpine Salmon, Impact Lab Ltd and Manawanui Support Ltd  He is also a director of The Instillery, Centre for Independent Studies and The Todd Corporation Limited, and is a member of the Impact Advisory Group of Macquarie Infrastructure and Real Assets 

Political and social views

English is regarded as more socially conservative than his predecessor, John Key  He has stated his opposition to voluntary euthanasia and physician-assisted suicide, same-sex civil unions, and the decriminalisation of prostitution  As Prime Minister he opposed any "liberalisation" of abortion law 

In 2004, English voted against a bill to establish civil unions for both same-sex and opposite-sex couples  In 2005, he voted for the Marriage (Gender Clarification) Amendment Bill, which would have amended the Marriage Act to define marriage as only between a man and a woman  English voted against the Marriage (Definition of Marriage) Amendment Bill, a bill that legalised same-sex marriage in New Zealand  However, in December 2016 he stated, "I'd probably vote differently now on the gay marriage issue  I don't think that gay marriage is a threat to anyone else's marriage" 

In 2009, English voted against the Misuse of Drugs (Medicinal Cannabis) Amendment Bill, a bill aimed at amending the Misuse of Drugs Act so that cannabis could be used for medical purposes 

Personal life 
English met his future wife, Mary Scanlon, at university  She was studying medicine at the time, and became a general practitioner  Both her parents were immigrants, her father being Samoan and her mother Italian, born on the island of Stromboli  They have six children: a daughter and five sons 

English is a practising Roman Catholic, but has stated that he considers his religious beliefs personal and thus separate from politics 

In June 2002, English took part in TV3's Fight For Life, a celebrity boxing fundraiser to raise money for the Yellow Ribbon anti-youth-suicide campaign, influenced by the death of a teenage nephew in 1997  He lost a split decision to former university colleague Ted Clarke 

Honours
In the 2018 Queen's Birthday Honours, English was appointed a Knight Companion of the New Zealand Order of Merit, for services of over 27 years to the State 

See also

List of New Zealand governments
Politics of New Zealand

References

External links

Profile at National Party 
Profile on Parliament nz
Releases and speeches at Beehive govt nz

|-

|-

|-

|-

|-

|-

|-

|-

|-

|-

|-

|-

|-

|-

|-

|-

|-

|-

1961 births
21st-century New Zealand politicians
Candidates in the 2017 New Zealand general election
Deputy Prime Ministers of New Zealand
Leaders of the Opposition (New Zealand)
Living people
Members of the Cabinet of New Zealand
Members of the New Zealand House of Representatives
New Zealand farmers
New Zealand finance ministers
New Zealand list MPs
New Zealand MPs for South Island electorates
New Zealand National Party MPs
New Zealand National Party leaders
New Zealand Roman Catholics
New Zealand people of Irish descent
People educated at St  Patrick's College, Silverstream
People from Dipton, New Zealand
People from Lumsden, New Zealand
Prime Ministers of New Zealand
University of Otago alumni
Victoria University of Wellington alumni
Knights Companion of the New Zealand Order of Merit




ground_truth: October 2001.