Package: high-order-layers-torch
Piecewise Polynomial in PyTorchThis is a PyTorch implementation of my tensorflowrepositoryand is more complete due to the flexibility of PyTorch.Lagrange Polynomial, Piecewise Lagrange Polynomial, Discontinuous Piecewise Lagrange Polynomial, Fourier Series, sum and product layers in PyTorch.  The sparsity of using piecewise polynomial layers means that by adding new segments the representational power of your network increases, but the time to complete a forward step remains constant. Implementation includes simple fully connected layers, convolution layers and deconvolutional layers using these models. This is a PyTorch implementation of thispaperincluding extension to Fourier Series and convolutional neural networks.Collab NotebookUsing simple high order layersSimple function approximationUsing simple high order MLP2d function approximationIdeaThe idea is extremely simple - instead of a single weight at the synapse, use n-weights.  The n-weights describe a piecewise polynomial (or other complex function) and each of the n-weights can be updated independently. A Lagrange polynomial and Gauss Lobatto points are used to minimize oscillations of the polynomial.  The same approach can be applied to any "functional" synapse, and I also have Fourier series synapses in this repo as well.  This can be implemented as construction of a polynomial or Fourier kernel followed by a standard pytorch layer where a linear activation is used.In the image below each "link" instead of being a single weight, is a function of both x and a set of weights.  These functions can consist of an orthogonal basis functions for efficient approximation.WhyUsing higher order polynomial representations might allow networks with much fewer total weights.Fully Connected Layer TypesAll polynomials are Lagrange polynomials with Chebyshev interpolation points.A helper function is provided in selecting and switching between these layersfromhigh_order_layers_torch.layersimport*layer1=high_order_fc_layers(layer_type=layer_type,n=n,in_features=784,out_features=100,segments=segments,)wherelayer_typeis one oflayer_typerepresentationcontinuouspiecewise polynomial using sum at the neuroncontinuous_prodpiecewise polynomial using products at the neurondiscontinuousdiscontinuous piecewise polynomial with sum at the neurondiscontinuous_proddiscontinous piecewise polynomial with product at the neuronpolynomialsingle polynomial (non piecewise) with sum at the neuronpolynomial_prodsingle polynomial (non piecewise) with product at the neuronproductProductfourierfourier series with sum at the neuronnis the number of interpolation points per segment for polynomials or the number of frequencies for fourier series,segmentsis the number of segments for piecewise polynomials,alphais used in product layers and when set to 1 keeps the linear part of the product, when set to 0 it subtracts the linear part from the product.Convolutional Layer Typesconv_layer=high_order_convolution_layers(layer_type=layer_type,n=n,in_channels=3,out_channels=6,kernel_size=5,segments=segments,rescale_output=rescale_output,periodicity=periodicity)All polynomials are Lagrange polynomials with Chebyshev interpolation points.layer_typerepresentationcontinuous(1d,2d)piecewise continuous polynomialdiscontinuous(1d,2d)piecewise discontinuous polynomialpolynomial(1d,2d)single polynomialfourier(1d,2d)fourier series convolutionh and p refinementp refinement is taking an existing network and increasing the polynomial order of that network without changing the network output.  This allow the user to train a network at low polynomial order and then use that same network to initialize a network with higher polynomial order.  This is particularly useful since a high order polynomial network will often converge poorly without the right initialization, the lower order network provides a good initial solution.  The function for changing the order of a network isfrom high_order_layers_torch.networks import interpolate_high_order_mlp
interpolate_high_order_mlp(
    network_in: HighOrderMLP, network_out: HighOrderMLPcurrent implementation only works with high order MLPs, not with convnets.  A similar function exists for h refinement.  h refinement is
refining the number of segments in a layer, and is used for similar reasoning.  Layers with lots of segments may be slow to converge
so the user starts with a small number of segments (1 or 2) and then increases the number of segments (h) using the lower initialization.  The following function currently only works for high order MLPs, not with convnetsfrom high_order_layers_torch.network import hp_refine_high_order_mlp
hp_refine_high_order_mlp(
    network_in: HighOrderMLP, network_out: HighOrderMLP
)InstallingInstalling locallyThis repo uses poetry, so runpoetry installand thenpoetry shellInstalling from pypipipinstallhigh-order-layers-torchorpoetry add high-order-layers-torchExamplesSimple function approximationApproximating a simple function using a single input and single output (single layer) with no hidden layers
to approximate a function using continuous and discontinuous piecewise polynomials (with 5 pieces) and simple
polynomials and fourier series.  The standard approach using ReLU is non competitive.  To see more complex see
the implicit representation pagehere.pythonexamples/function_example.pyXOR : 0.5 for x*y > 0 else -0.5Simple XOR problem using the standard network structure (2 inputs 2 hidden 1 output) this will also work with no hidden layers. The function is discontinuous along the axis and we try and fit that function. Using piecewise discontinuous layers the model can match the function exactly.With piecewise continuous it doesn't work quite as well.Polynomial doesn't work well at all (expected).MNIST (convolutional)pythonexamples/mnist.pymax_epochs=1train_fraction=0.1layer_type=continuous2dn=4segments=2CIFAR100 (convolutional)python examples/cifar100.py -m max_epochs=20 train_fraction=1.0 layer_type=polynomial segments=2 n=7 nonlinearity=False rescale_output=False periodicity=2.0 lr=0.001 linear_output=FalseVariational AutoencoderStill a WIP.  Does work, but needs improvement.python examples/variational_autoencoder.py -m max_epochs=300 train_fraction=1.0run with nevergrad for parameter tuningpython examples/variational_autoencoder.py -mInvariant MNIST (fully connected)Without polynomial refinementpythonexamples/invariant_mnist.pymax_epochs=100train_fraction=1mlp.layer_type=continuousmlp.n=5mlp.p_refine=Falsemlp.hidden.layers=4with polynomial refinement (p-refinement)python examples/invariant_mnist.py max_epochs=100 train_fraction=1 layer_type=mlp.continuous mlp.n=2 mlp.target_n=5 mlp.p_refine=TrueI've also added hp refinement, but it needs a lot of testing.Implicit RepresentationAn example of implicit representation for image compression, language generation can be foundhere.  I intend to explore generative models in natural language furtherherePDEs in Fluid DynamicsAn example using implicit representation to solve hyperbolic (nonlinear) wave equations can be foundhereNatural Language GenerationExamples using these networks for natural language generation can be foundhereGenerative musicNo real progress herehereTest and CoverageAfter installing and runningpoetry shellrunpytestfor coverage, runcoverage run -m pytestand thencoverage reportA note on the product unit (I rarely use anymore)The layers used here do not require additional activation functions and use a simple sum or product in place of the activation.
I almost always use sum units, but product units are performed in this manner$$ product=-1+\prod_{i}(1 + f_{i})+(1-\alpha)\sum_{i}f_{i} $$The 1 is added to each function output to as each of the sub products is also computed.  The linear part is controlled by
the alpha parameter.Notes on optimizerThe Lion optimizer seems to be the best choice since it performs better than Adam in general, but seems to work especially well
for the case of polynomials.Notes on normalizationAlthough you can use batchnorm, layernorm etc... work better, I've found that you can actually just use the infinity norm ("max_abs" norm) which has no parameters
for this formulation (same approach seems not to work very well for standard relu networks - but need to investigate this further).
The max_abs normalization is defined this waynormalized_x = x/(max(abs(x))+eps)where the normalization is done per sample (as opposed to per batch).  The way the layers are formulated, we don't want the neuron
values to extend beyond [-1, 1] as the polynomial values grow rapidly beyond that range.  You can also use mirror periodicity to keep the
values within from growing rapidly. We want the values to cover the entire range [-1, 1] of the polynomials as the weights
are packed towards the edges of each segment (though using even number of segments means you'll have a lot of weights near the origin).Reference@misc{Loverich2020,
  author = {Loverich, John},
  title = {High Order Layers Torch},
  year = {2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/jloveric/high-order-layers-torch}},
}