Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper discusses a parametrization of the set of inputs that a layer of ReLUs maps to a given output value. 

PROS. 
* The question about the preimages of ReLU networks is quite interesting. 
* Parametrizing the preimages of a layer of ReLUs is a neat idea. 
* The contribution is not very substantial in its current form, but it seems that with more time it could evolve further. 

CONS. 
* The focus of the workshop is on late-breaking developments, very novel ideas and position papers. While the paper at hand includes a neat observation, I hardly regard it as a late-breaking development, very novel idea, or a position. 
* The discussion is quite narrow, assuming the same number of input and output units and that the weight matrix is regular. The biases are not adequately discussed. The situation for a network with multiple layers is not further discussed. 
* The practical implementation and computational aspects are not further discussed. 
* The terminology is not used adequately, confusing notions such as  kernel, nullspace, linear vs affine, etc. The discussion is confusing, with the reader having to guess often what could be meant. For instance, quantifiers are missing or wrong. The assumptions are not properly and timely stated, for instance regarding the number of outputs, the biases, the invertibility of the weight matrix, and so on. The typesetting is very poor and contains many typos. 

review_title: review of ``THE PREIMAGE OF RECTIFIER NETWORK ACTIVITIES''

final_score: rating: 7: Good paper, accept
review_content: This paper studies the question: what is the set of points in the input space of a linear + relu layer that map to a given point in output space? This question has been studied before in an empirical manner (for multi-layer networks), but this paper is the first to give a simple linear algebraic answer.

The central construction of the paper is a basis e_i that has the property e_i^T w_j = delta(i,j): that is, e_i is orthogonal to all weight vectors w_j except w_i. Although it is not often taught in linear algebra courses, this is a well known concept called the dual basis of w. Since the paper uses quite a lot of space to explain the idea, it may be more efficient to simply state that every basis has a unique dual basis (with said property) and then continue to work with it, providing a reference to the reader who is interested in learning more about the concept and how one might compute a dual basis efficiently.

The analysis seems to be mostly correct for the case where W is a square, invertible matrix. This and other assumptions should be clearly stated. If W is not square or not invertible, many complicated phenomena may occur (https://en.wikipedia.org/wiki/Arrangement_of_hyperplanes).

Since it is immediately clear that the pre-image of a relu net must be piecewise linear, the value of this paper is in a precise characterization of this linear manifold. To do this thoroughly, we cannot ignore the fact that e.g. not all layers in a neural network have the same dimensionality (although the full rank assumption may be reasonable for dimension-increasing layers).

One issue I'm slightly uncertain about is the role of the bias, which is sometimes ignored in the paper. Looking at the equation w_j^T x = ... on page 2, I think the constraints on alpha_i should involve b_i.

The question of preimages of multi-layer networks is treated very briefly at the end. I think the paper could be much more significant if you used some of the space used to explain the dual basis to go into greater depth on this question. E.g. how can we parameterize the piecewise linear manifold of a multi-layer net, given some values for the weights of the network?

I think the equation for Pi_i on page 1 should read:
Pi_i = {x : w_i^T x + b_i = 0}   (i=1..n)

The paper contains many typos and missing periods:
explicitely
the matrix W . and bias
"sets of hyperplanes" -> hyperplanes
easily using by just
intersection For
= 1 The
a_j <= 0 By
e_i Any
a_i >= 0 The
region they will 
can therefore me considered
review_title: Interesting paper, can be streamlined

Decision: Accept