Package: arpkeras
Auto-Rotating Perceptrons (ARP)This repository contains the Keras implementation of the Auto-Rotating Perceptrons (Saromo, Villota, and Villanueva) for dense layers of artificial neural networks. These neural units were presented inthis paperwith anoral expositionat theLXAI workshopatNeurIPS 2019.The ARP library was developed byDaniel SaromoandMatias Valdenegro-Toro. This repository contains implementations that are not present in theLXAI @ NeurIPS 2019 paper.What is an Auto-Rotating Perceptron?The ARP are a generalization of the perceptron unit that aims to avoid the vanishing gradient problem by making the activation function's input near zero, without altering the inference structure learned by the perceptron.Classic perceptronAuto-Rotating PerceptronHence, a classic perceptron becomes the particular case of an ARP withrho=1.Basic principle: The dynamic regionWe define thedynamic regionas the symmetric numerical range (w.r.t.0) from where we would like the pre-activation values to come from in order to avoid node saturation. Recall that, in order to avoid the vanishing gradient problem (VGP), we do not want the derivative of the activation function to take tiny values. For the ARP, the dynamic region goes from-Lto+L.For example, in the unipolar sigmoid activation function (logistic curve) shown below, we would like it to receive values from-4to4. For inputs whose absolute values are higher than4, the derivative of the activation is too low. Hence, theLvalue could be4. The resultingdynamic regionprojected on the derivative curve is depicted as a gray shade.What isL?Lis the hyperparameter that defines the limits of the desired symmetricdynamic region.How do I chooseL?You need to analyze the activation function and its derivative. For the dynamic region, there is a trade-off. For a biggerL, you accept more non-linearity for the activation function, but at the same time, you get more saturation.Below you have the suggested values forL, according to the activation function of the neuron:Activation functionLtanh2sigmoid4arctan7In the figure below, you can see that for inputs whose absolute values are higher than the values from the table, the derivative of the activation functions is very small.What aboutxQ?In theoriginal ARP paper, you needed to set this value manually. Currently, by default,xQis automatically calculated usingL. However, the ARP library supports a custom selection of thexQvalue.A deeper explanation can be found in the journal version of the ARP paper (in preparation).ARP Vs. Classic perceptronsAs shown in the example notebook (examples/example_CIFAR10_Keras.ipynb) that compares ARP and classic perceptrons, the ARP can lead to a faster convergence and lower loss values.Furthermore, there is an application of the ARP to calibrate a wearable sensor where the test loss was reduced by a factor of 15 when changing from classic perceptrons to ARP. You can check the paperhere.In machine learning, the advantages of using one technique or another are problem-dependant. We encourage you to apply ARP in your research and discover its potential.Keras implementationInstalationThe ARP library is available on thePython Package Index.
To install the library, first installpipand then use the following command:pipinstallarpkerasYou may need to update thepipmanager. You can use:python-mpipinstall–upgradepipImportfromARPkerasimportAutoRotDenseCreating an ARP modelTheAutoRotDenseclass implementation inherits from the KerasDenseclass. Hence, you can use it as a typical KerasDenselayer, but adding the following arguments:xmin_lim: The lower limit of the values that will enter the neuron. For example, if we scale our input data to the range0to1, and we choosetanhas the activation function (which goes from-1to+1), then the lowest input value will bexmin_lim=-1.xmax_lim: The upper limit of the values that will enter the neuron. Analogous toxmin_lim.L: The limit of the desired symmetricaldynamic region. This value is theonly hyperparameterneeded for the Auto-Rotating layers. The two variables described above depend on the activation function you choose and your data preprocessing.This is an example of use for the unipolar'sigmoid'activation (whose output goes from0to+1) with data scaled to the range0to1:fromtensorflow.keras.modelsimportSequentialfromtensorflow.keras.layersimportDensefromARPkerasimportAutoRotDensexmin_lim=0# min( 0,0)xmax_lim=1# max(+1,1)L=4model=Sequential()model.add(Dense(20),input_shape=(123,))model.add(AutoRotDense(10,xmin_lim=xmin_lim,xmax_lim=xmax_lim,L=L,activation='sigmoid'))#By default the `AutoRot` flag of the Auto-Rotating layers is True.model.summary()This is another example, when using the'tanh'activation (whose output goes from-1to+1) with data scaled to the range0to1:fromtensorflow.keras.modelsimportSequentialfromtensorflow.keras.layersimportDensefromARPkerasimportAutoRotDensexmin_lim=-1# min(-1,0)xmax_lim=+1# max(+1,1)L=7model=Sequential()model.add(AutoRotDense(20,input_shape=(123,),xmin_lim=xmin_lim,xmax_lim=xmax_lim,L=L,activation='tanh'))model.add(AutoRotDense(10,xmin_lim,xmax_lim,L,activation='tanh'))#By default the `AutoRot` flag of the Auto-Rotating layers is True.model.summary()BeyondDenselayersIs that all for the ARP? No! The journal version of the ARP paper is being finished, with the support of Dr. Edwin Villanueva and Dr. Matias Valdenegro-Toro. There, the Auto-Rotating concept was extrapolated to other layer types, creating theAuto-Rotating Neural Networks.These are the Keras layers implemented with the Auto-Rotating operation (Tip: Just addAutoRotbefore the layer name):Keras Original LayerAuto-Rotating ImplementationDenseAutoRotDenseSimpleRNNAutoRotSimpleRNNLSTMAutoRotLSTMGRUAutoRotGRUConv1DAutoRotConv1DConv2DAutoRotConv2DConv3DAutoRotConv3DConv2DTransposeAutoRotConv2DTransposeConv3DTransposeAutoRotConv3DTransposeSeparableConvAutoRotSeparableConvSeparableConv1DAutoRotSeparableConv1DSeparableConv2DAutoRotSeparableConv2DDepthwiseConv2DAutoRotDepthwiseConv2DComing soon :sunglasses::Auto-Rotating Neural Networks(Saromo, Villanueva, and Valdenegro-Toro).CitationThis code is free to use for research purposes, and if used or modified in any way, please consider citing:@article{saromo2019arp,
  title={{A}uto-{R}otating {P}erceptrons},
  author={Saromo, Daniel and Villota, Elizabeth and Villanueva, Edwin},
  journal={LatinX in AI Workshop at NeurIPS 2019 (arXiv:1910.02483)},
  year={2019}
}Other inquiries:daniel.saromo@pucp.pe