White Box Network: Obtaining a right composition ordering of functions

1 INTRODUCTION . Reverse engineering can be defined as the deconstruction of an object to reveal its design or architecture , or to extract knowledge from the object [ Eldad , 2005 ] . Typically , in machine learning , obtaining the internal functions of a device only with its input and output values is a regression problem . Before machine learning was developed , devices were disassembled to analyze them . However , advances in deep learning have allowed us to approximate various functions in regression tasks ; therefore , we can now solve this regression problem without actually looking inside the object . While this development in deep learning ( Bengio , 2009 ; Bengio et al. , 2013 ) has benefitted reverse engineering significantly , some limitations still exist . First , deep learning is a black box model ; therefore , the results may not be interpreted . In this case , information regarding the object such as what it does and how it works may not be extracted . This causes field engineers to not entirely trust the results of deep learning even if the correct values were reflected . Next , deep learning approximates the target function and does not reveal the actual design and architecture of the device . An artificial neural network is a mathematical simulation of a neural network composed of human neurons and synapses that simulate the manner in which the human brain recognizes patterns . However , a device typically comprises an electronic circuit and block functions ; therefore , it may not fit perfectly with the neural network structure , which can cause large errors when certain data are used . We herein propose a white box network ( WBN ) with a selection layer , specially designed for reverse engineering . This network continuously composites function blocks to create a target function . We assume that all types of function blocks that can constitute the target function are provided in advance . This differs from the normal neural network because the WBN reveals the exact functions with the correct inputs and their ordering to construct the target function , instead of merely approximating them . Our WBN is different from normal neural networks as the model architecture is discretized using a discretized matrix called the selection layer . Because discretization is interpretable ( Chen et al. , 2016 ) , our WBN can reconstruct the target function . The WBN is motivated by the questions , Can we benefit from knowing the functions that are used in certain fields ? . For example , in a programmable logic controller ( PLC ) , a target function can be obtained through a ladder logic diagram comprising well-known function blocks . Therefore , we can obtain a target function by ordering those function blocks with the correct inputs . As such , in this study , we create a WBN that can automate reverse engineering . For the experiment , we imitated the PLC data and verified whether our model could obtain their ladder logic diagram . Additionally , we conducted other experiments to validate the selection layer . We assumed that we may not know the functions used in a certain field . Therefore , we regarded the given function blocks as neural networks . Subsequently , the network was ordered and connected to the appropriate neural networks to obtain the target function that resembles PathNet ( Fernando et al. , 2017 ) . Using this concept , we create an end-to-end PathNet structure and perform experiments to verify whether it functions like PathNet . 2 RELATED WORKS . In machine learning , regression is a task of obtaining a real-valued function y = f ( x ) from the hypothesis set ( X , Y ) . However , typically , actual data may contain noise or exhibit data loss owing to measurement limitations or other causes . Therefore , regression is often considered as a black box process and an exact function may not be obtained . For example , in Gaussian process regression [ Williams and Ras-mussen , 2006 ] and support vector regression ( SVR ) ( Smola & Schölkopf , 2004 ) , we reproduced a kernel and obtained an approximated value instead of obtaining a function . Another typical regression method is using a multilayered neural network ( Bengio , 2009 ; Bengio et al. , 2013 ) , which approximates a function by using the human neuron model . The aim of reverse engineering is to obtain the exact structure of a function rather than merely approximating it . In some fields , we may know the type of function forms that typically appear ; hence , we can use that information to construct a function . An equation learner ( EQL ) ( Martius & Lampert , 2017 ; Sahoo et al. , 2018 ) is designed for regression task that can be described by analytical expressions of mechanical systems such as a pendulum or a robotic arm . An EQL has been shown more effective at solving dynamics than conventional neural nets ; additionally , it reduces extrapolation errors . Furthermore , PDE-net ( Long et al. , 2018b ; a ) accurately predicts the dynamics of complex systems and uncovers the underlying hidden partial differential equation models using a convolution kernel filter . This type of model renders regression more effective if certain types of function blocks or processes that appear in a specific field are known . A discrete variable can be used variously , e.g. , learning probabilistic latent representations ( Kingma et al. , 2014 ) , image region ( Xu et al. , 2015 ) , and memory access ( Graves et al. , 2014 ; 2016 ) . Especially for the memory access , it resembles an attention mechanism that can help interpret the focus of the model . The end-to-end memory network ( Kingma et al. , 2014 ) uses softmax to access the memory in questions and answering task . This allows us to determine the sentence that the network refers to when it answers a specific question . The neural Turing machine ( NTM ) ( Graves et al. , 2014 ) and differentiable neural computer ( Graves et al. , 2016 ) use a sharpening method instead of merely using softmax . As such , those models select information from the memory without disordering information , thus rendering them more interpretable . Therefore , we can determine how the model functions . For example , we can reconstruct the Copy processes of the NTM and create a pseudocode by referring to the headers in the model . Therefore , discretization can be used such that the model is more interpretable ( Chen et al. , 2016 ) , and it is sometimes more efficient for computing ( Rae et al. , 2016 ) . PathNet ( Fernando et al. , 2017 ) is a modular deep neural network with pathways . It uses a genetic algorithm to obtain the best pathway for learning . The pathways are represented as integer-valued matrices ; therefore , the modules can be selected by accessing those matrices . PathNet reuses the pathways for a different task as a transfer learning paradigm and shows the potential in selecting the appropriate modules to obtain the target results . 3 METHODS . 3.1 WHITE BOX NETWORK ARCHITECTURE . The WBN is a multilayered feed forward network . For each hidden layer , it consists of a selection layer followed by function block mappings . In this study , we assumed that all function blocks constituting the target function were known before we built the model . However , the exact order of the function blocks with the correct inputs must be obtained . Each function block can be multivalued with multiple inputs ; however , for simplicity , we assume that the function blocks are unitary and binary operations . Suppose that the model is L-layered , and we have m ( l ) unitary functions and n ( l ) binary functions in the lth layer . In the selection layer , we select inputs to enter into each function block . The selection layer can be written as z ( l ) = Ŵ ( l ) y ( l−1 ) Here , y ( l−1 ) is the output of the previous layer and Ŵ ( l ) is the selection matrix . If each row of Ŵ ( l ) has only one 1 and the remaining 0s , then , it selects one element of the y ( l−1 ) as an element of z ( l ) . To make Ŵ ( l ) differentiable , we approximate Ŵ ( l ) by Ŵ ( l ) = softmax ( W ( l ) /τ , axis=-1 ) Here , τ > 0 is a temperature parameter that renders Ŵ sharper as τ → 0. τ can be chosen as both a constant and a learnable parameter . After multiplying the selection matrix , z ( l ) goes through the function block mappings . Let f ( l ) 1 , . . . , f ( l ) m ( l ) and g ( l ) 1 , . . . , g ( l ) n ( l ) be the given unitary and binary function blocks on the lth layer . Then , our new output y ( l ) is y ( l ) = ( f ( l ) 1 ( z ( l ) 1 ) , . . . , f ( l ) m ( l ) ( z ( l ) m ( l ) ) , g ( l ) 1 ( z ( l ) m ( l ) +1 , z ( l ) m ( l ) +2 ) , . . . , g ( l ) n ( l ) ( z ( l ) m ( l ) +2n ( l ) −1 , z ( l ) m ( l ) +2n ( l ) ) ) As such , when we select the function blocks of each layer , the dimensions of the variables are fixed . For example , y ( l ) represents ( m ( l ) +n ( l ) ) -dimensional vectors and z ( l ) represents ( m ( l ) +2n ( l ) ) dimensional vectors . Finally , we convert the last layer ’ s output to fit the target output dimension . Therefore , we use the selection layer again to match with the output dimension . o = Ŵ ( L+1 ) y ( L ) Here , the final output is a composition of function blocks from each layer with selective inputs . 3.2 END-TO-END PATHNET ARCHITECTURE . Suppose we do not have any information regarding the target function . In that case , we can select our function blocks as a single-layered neural network followed by nonlinear activation . Then , our model becomes modular neural network that selects the right modules to find the best pathway to obtain the target value . This concept is the same as that of a previous study , i.e. , PathNet ( Fernando et al. , 2017 ) . To apply one key component of our WBN to existing PathNet architecture , we need to switch the order of the selection layer and the function block mapping for each hidden layer . Therefore , function block mapping is proceeded by the selection layer . In function block mapping , the function blocks are one-layered neural networks with learnable parameters . Each of them is assigned a single input and yields a single output , which is typically multidimensional . Now , suppose that our model is a modular deep neural network with L layers with each layer containing M neural network modules . Let N be the number of maximum distinct modules per layer that can be selected in a pathway . Let φ ( l ) 1 , , φ ( l ) M be neural network modules in the lth layer and let k ( l ) be the dimension ( number of channels ) of hidden output variables . Then , the function mapping layer is written as Z ( l ) = ( φ ( l ) 1 ( y ( l−1 ) ) , φ ( l ) 2 ( y ( l−1 ) ) , . . . , φ ( l ) M ( y ( l−1 ) ) ) Here , Z ( l ) is M by k ( l ) matrix . Then , selection layer selects the neural network outputs Ŷ ( l ) = Ŵ ( l ) Z ( l ) The selection matrix Ŵ ( l ) is N ×M with each row of Ŵ ( l ) ) containing only one almost 1 and the remaining almost 0s ; Ŵ ( l ) can be approximated in the manner described previously . Now , we add all rows of Ŷ ( l ) such that the hidden output y ( l ) follows the existing PathNet . y ( l ) = reduce sum ( Ŷ ( l ) , axis=0 ) Finally , there is a linear layer that can be used to convert the last hidden layer to the output value . o = Wy ( L ) + b This architecture is inspired by the PathNet in which a selection matrix instead of a genetic algorithm is used to select a pathway . It is differentiable , and hence we can easily train the network by gradient descent .

Summary: This paper presents White Box Network (WBN), which allows for composing function blocks from a given set of functions to construct a target function. The main idea is to introduce a selection layer that only selects one element of the previous layer as an input to a function block. This allows for both introducing function priors as well as interpreting the learned function. The paper also presents a setting where each function block is a neural network that can be learned end-to-end using a PathNet style setting and shows positive transfer across MNIST and CIFAR classification tasks.