White Box Network: Obtaining a right composition ordering of functions

1 INTRODUCTION . Reverse engineering can be defined as the deconstruction of an object to reveal its design or architecture , or to extract knowledge from the object [ Eldad , 2005 ] . Typically , in machine learning , obtaining the internal functions of a device only with its input and output values is a regression problem . Before machine learning was developed , devices were disassembled to analyze them . However , advances in deep learning have allowed us to approximate various functions in regression tasks ; therefore , we can now solve this regression problem without actually looking inside the object . While this development in deep learning ( Bengio , 2009 ; Bengio et al. , 2013 ) has benefitted reverse engineering significantly , some limitations still exist . First , deep learning is a black box model ; therefore , the results may not be interpreted . In this case , information regarding the object such as what it does and how it works may not be extracted . This causes field engineers to not entirely trust the results of deep learning even if the correct values were reflected . Next , deep learning approximates the target function and does not reveal the actual design and architecture of the device . An artificial neural network is a mathematical simulation of a neural network composed of human neurons and synapses that simulate the manner in which the human brain recognizes patterns . However , a device typically comprises an electronic circuit and block functions ; therefore , it may not fit perfectly with the neural network structure , which can cause large errors when certain data are used . We herein propose a white box network ( WBN ) with a selection layer , specially designed for reverse engineering . This network continuously composites function blocks to create a target function . We assume that all types of function blocks that can constitute the target function are provided in advance . This differs from the normal neural network because the WBN reveals the exact functions with the correct inputs and their ordering to construct the target function , instead of merely approximating them . Our WBN is different from normal neural networks as the model architecture is discretized using a discretized matrix called the selection layer . Because discretization is interpretable ( Chen et al. , 2016 ) , our WBN can reconstruct the target function . The WBN is motivated by the questions , Can we benefit from knowing the functions that are used in certain fields ? . For example , in a programmable logic controller ( PLC ) , a target function can be obtained through a ladder logic diagram comprising well-known function blocks . Therefore , we can obtain a target function by ordering those function blocks with the correct inputs . As such , in this study , we create a WBN that can automate reverse engineering . For the experiment , we imitated the PLC data and verified whether our model could obtain their ladder logic diagram . Additionally , we conducted other experiments to validate the selection layer . We assumed that we may not know the functions used in a certain field . Therefore , we regarded the given function blocks as neural networks . Subsequently , the network was ordered and connected to the appropriate neural networks to obtain the target function that resembles PathNet ( Fernando et al. , 2017 ) . Using this concept , we create an end-to-end PathNet structure and perform experiments to verify whether it functions like PathNet . 2 RELATED WORKS . In machine learning , regression is a task of obtaining a real-valued function y = f ( x ) from the hypothesis set ( X , Y ) . However , typically , actual data may contain noise or exhibit data loss owing to measurement limitations or other causes . Therefore , regression is often considered as a black box process and an exact function may not be obtained . For example , in Gaussian process regression [ Williams and Ras-mussen , 2006 ] and support vector regression ( SVR ) ( Smola & Schölkopf , 2004 ) , we reproduced a kernel and obtained an approximated value instead of obtaining a function . Another typical regression method is using a multilayered neural network ( Bengio , 2009 ; Bengio et al. , 2013 ) , which approximates a function by using the human neuron model . The aim of reverse engineering is to obtain the exact structure of a function rather than merely approximating it . In some fields , we may know the type of function forms that typically appear ; hence , we can use that information to construct a function . An equation learner ( EQL ) ( Martius & Lampert , 2017 ; Sahoo et al. , 2018 ) is designed for regression task that can be described by analytical expressions of mechanical systems such as a pendulum or a robotic arm . An EQL has been shown more effective at solving dynamics than conventional neural nets ; additionally , it reduces extrapolation errors . Furthermore , PDE-net ( Long et al. , 2018b ; a ) accurately predicts the dynamics of complex systems and uncovers the underlying hidden partial differential equation models using a convolution kernel filter . This type of model renders regression more effective if certain types of function blocks or processes that appear in a specific field are known . A discrete variable can be used variously , e.g. , learning probabilistic latent representations ( Kingma et al. , 2014 ) , image region ( Xu et al. , 2015 ) , and memory access ( Graves et al. , 2014 ; 2016 ) . Especially for the memory access , it resembles an attention mechanism that can help interpret the focus of the model . The end-to-end memory network ( Kingma et al. , 2014 ) uses softmax to access the memory in questions and answering task . This allows us to determine the sentence that the network refers to when it answers a specific question . The neural Turing machine ( NTM ) ( Graves et al. , 2014 ) and differentiable neural computer ( Graves et al. , 2016 ) use a sharpening method instead of merely using softmax . As such , those models select information from the memory without disordering information , thus rendering them more interpretable . Therefore , we can determine how the model functions . For example , we can reconstruct the Copy processes of the NTM and create a pseudocode by referring to the headers in the model . Therefore , discretization can be used such that the model is more interpretable ( Chen et al. , 2016 ) , and it is sometimes more efficient for computing ( Rae et al. , 2016 ) . PathNet ( Fernando et al. , 2017 ) is a modular deep neural network with pathways . It uses a genetic algorithm to obtain the best pathway for learning . The pathways are represented as integer-valued matrices ; therefore , the modules can be selected by accessing those matrices . PathNet reuses the pathways for a different task as a transfer learning paradigm and shows the potential in selecting the appropriate modules to obtain the target results . 3 METHODS . 3.1 WHITE BOX NETWORK ARCHITECTURE . The WBN is a multilayered feed forward network . For each hidden layer , it consists of a selection layer followed by function block mappings . In this study , we assumed that all function blocks constituting the target function were known before we built the model . However , the exact order of the function blocks with the correct inputs must be obtained . Each function block can be multivalued with multiple inputs ; however , for simplicity , we assume that the function blocks are unitary and binary operations . Suppose that the model is L-layered , and we have m ( l ) unitary functions and n ( l ) binary functions in the lth layer . In the selection layer , we select inputs to enter into each function block . The selection layer can be written as z ( l ) = Ŵ ( l ) y ( l−1 ) Here , y ( l−1 ) is the output of the previous layer and Ŵ ( l ) is the selection matrix . If each row of Ŵ ( l ) has only one 1 and the remaining 0s , then , it selects one element of the y ( l−1 ) as an element of z ( l ) . To make Ŵ ( l ) differentiable , we approximate Ŵ ( l ) by Ŵ ( l ) = softmax ( W ( l ) /τ , axis=-1 ) Here , τ > 0 is a temperature parameter that renders Ŵ sharper as τ → 0. τ can be chosen as both a constant and a learnable parameter . After multiplying the selection matrix , z ( l ) goes through the function block mappings . Let f ( l ) 1 , . . . , f ( l ) m ( l ) and g ( l ) 1 , . . . , g ( l ) n ( l ) be the given unitary and binary function blocks on the lth layer . Then , our new output y ( l ) is y ( l ) = ( f ( l ) 1 ( z ( l ) 1 ) , . . . , f ( l ) m ( l ) ( z ( l ) m ( l ) ) , g ( l ) 1 ( z ( l ) m ( l ) +1 , z ( l ) m ( l ) +2 ) , . . . , g ( l ) n ( l ) ( z ( l ) m ( l ) +2n ( l ) −1 , z ( l ) m ( l ) +2n ( l ) ) ) As such , when we select the function blocks of each layer , the dimensions of the variables are fixed . For example , y ( l ) represents ( m ( l ) +n ( l ) ) -dimensional vectors and z ( l ) represents ( m ( l ) +2n ( l ) ) dimensional vectors . Finally , we convert the last layer ’ s output to fit the target output dimension . Therefore , we use the selection layer again to match with the output dimension . o = Ŵ ( L+1 ) y ( L ) Here , the final output is a composition of function blocks from each layer with selective inputs . 3.2 END-TO-END PATHNET ARCHITECTURE . Suppose we do not have any information regarding the target function . In that case , we can select our function blocks as a single-layered neural network followed by nonlinear activation . Then , our model becomes modular neural network that selects the right modules to find the best pathway to obtain the target value . This concept is the same as that of a previous study , i.e. , PathNet ( Fernando et al. , 2017 ) . To apply one key component of our WBN to existing PathNet architecture , we need to switch the order of the selection layer and the function block mapping for each hidden layer . Therefore , function block mapping is proceeded by the selection layer . In function block mapping , the function blocks are one-layered neural networks with learnable parameters . Each of them is assigned a single input and yields a single output , which is typically multidimensional . Now , suppose that our model is a modular deep neural network with L layers with each layer containing M neural network modules . Let N be the number of maximum distinct modules per layer that can be selected in a pathway . Let φ ( l ) 1 , , φ ( l ) M be neural network modules in the lth layer and let k ( l ) be the dimension ( number of channels ) of hidden output variables . Then , the function mapping layer is written as Z ( l ) = ( φ ( l ) 1 ( y ( l−1 ) ) , φ ( l ) 2 ( y ( l−1 ) ) , . . . , φ ( l ) M ( y ( l−1 ) ) ) Here , Z ( l ) is M by k ( l ) matrix . Then , selection layer selects the neural network outputs Ŷ ( l ) = Ŵ ( l ) Z ( l ) The selection matrix Ŵ ( l ) is N ×M with each row of Ŵ ( l ) ) containing only one almost 1 and the remaining almost 0s ; Ŵ ( l ) can be approximated in the manner described previously . Now , we add all rows of Ŷ ( l ) such that the hidden output y ( l ) follows the existing PathNet . y ( l ) = reduce sum ( Ŷ ( l ) , axis=0 ) Finally , there is a linear layer that can be used to convert the last hidden layer to the output value . o = Wy ( L ) + b This architecture is inspired by the PathNet in which a selection matrix instead of a genetic algorithm is used to select a pathway . It is differentiable , and hence we can easily train the network by gradient descent .

Summary: This paper investigates the question of identifying concise equations from data to understand the functional relations. In particular, a set of base functions are given in hand and the goal is to obtain the right composition of these functions which fits the target function. The main contribution of the paper is to introduce a selection layer, which enhances sparse connections in the network. Several experiments are conducted to show the effectiveness of the method.