text: 158,144,1236913875804536834,1211825303388995584,Russell Tsuchida,"Back in February, @Tea_Pearce, Chris van der Heide, Fred Roosta, @marcus_marcusg and I did some work on the kernels of infinitely wide deep neural networks with GELU and ELU activations. We also studied the fixed points of these kernels. Check it out here: <LINK>",https://arxiv.org/abs/2002.08517,"Analysing and computing with Gaussian processes arising from infinitely wide neural networks has recently seen a resurgence in popularity. Despite this, many explicit covariance functions of networks with activation functions used in modern networks remain unknown. Furthermore, while the kernels of deep networks can be computed iteratively, theoretical understanding of deep kernels is lacking, particularly with respect to fixed-point dynamics. Firstly, we derive the covariance functions of multi-layer perceptrons (MLPs) with exponential linear units (ELU) and Gaussian error linear units (GELU) and evaluate the performance of the limiting Gaussian processes on some benchmarks. Secondly, and more generally, we analyse the fixed-point dynamics of iterated kernels corresponding to a broad range of activation functions. We find that unlike some previously studied neural network kernels, these new kernels exhibit non-trivial fixed-point dynamics which are mirrored in finite-width neural networks. The fixed point behaviour present in some networks explains a mechanism for implicit regularisation in overparameterised deep models. Our results relate to both the static iid parameter conjugate kernel and the dynamic neural tangent kernel constructions. Software at github.com/RussellTsuchida/ELU_GELU_kernels. ","Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite