Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Revised Review:

The authors have largely addressed my concerns with the revised manuscript. I still have some doubts about the C > N setting (the new settings of C / N of 4 and 2 aren't C >> N, and the associated results aren't detailed clearly in the paper), but I think the paper warrants acceptance.

Original Review:

The paper proposes fixing the classification layers of neural networks, replacing the traditional learned affine transformation with a fixed (e.g., Hadamard) matrix. This is motivated by the observation that classification layers frequently constitute a non-trivial fraction of a network's overall parameter count, compute requirements, and memory usage, and by the observation that removal of pre-classification fully-connected layers has often been found to have minimal impact on performance. Experiments are performed on a range of datasets and network architectures, in both image classification and NLP settings.

First, I'd like to note that the empirical component of this paper is strong: I was impressed by the breadth of architectures and settings covered, and the experiments left me reasonably convinced that the classification layer can often be fixed, at least for image classification tasks, without significant loss of accuracy.

I have two general concerns. For one, removing the fully connected classification layer is not a novel idea; All Convolutional Networks (https://arxiv.org/abs/1412.6806) reported excellent results without an additional fully connected affine transform (just a global average pooling after the last convolutional layer). I think it would be worth at least referencing/discussing differences with this and other all-convolutional architectures. Including a fixed Hadamard matrix for the classification layer is I believe new (although related to an existing literature on using structured matrices in neural networks).

However, I have doubts about the ability of the approach to scale to problems with a larger number of classes, which arguably is a primary motivation of the paper ("parameters ... grow linearly with the number of classes"). Specifically, the idea of using a fixed N x C matrix with C orthogonal columns (such as Hadamard) is only possible when N > C. This is a critical point: in the N > C regime, a final hidden representation with N dimensions can be chosen to achieve *any* C-dimensional output, regardless of the projection matrix used (so long as it is full rank). This makes it seem fairly reasonable to me that the network can (at least approximately, and complicated by the ReLU nonlinearities) fold the "desired" classification layer into the previous layer, especially with a learned scaling and bias term. In fact it's not clear to me that the fixed classification layer accomplishes anything here, beyond projecting from N -> C (i.e., if N = C, I'd guess it could be removed entirely similar to all convolutional nets, as long as the learned scaling and bias were retained).

On the other hand, when C > N, it is not possible to have mutually orthogonal columns, and in general the output is constrained to lie in an N-dimensional subspace of the overall C-dimensional output space. Picking somewhat randomly a *fixed* N-dimensional subspace seems like a bad idea when N << C, since it is unlikely to select a subspace in which it is possible to adequately capture correlations between the different classes. This makes the proposed technique much less appealing for precisely the family of problems where it would be most effective in reducing compute/memory requirements. It also provides (in my view) a clearer explanation for the failure of the approach in the NLP setting. These issues were not discussed anywhere in the text as far as I can tell, and I think it's necessary to at least acknowledge that mutually orthogonal columns can't be chosen when C > N in section 2.2 (and probably include a longer discussion on the probable implications).

Overall, I think the paper provides a useful observation that clearly isn't common knowledge, since classification layers persist in many popular recent architectures. But the notion of fixing or removing the classification layer isn't particularly novel, and I don't believe the proposed technique would scale well to settings with many classes. As is I think the paper falls slightly short.
review_title: Good experiments, concerns about novelty and scalability

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes to use a fixed weight matrix to replace the final linear projection in a deep neural network.
This fixed classifier is combined with a global scaling and per output shift that are learned.
The authors claim that this can be used as a drop in replacement for standard architectures and does not result in reduced performance.
The key advantage is that it generates a reduction in parameters (e.g. for resent 50 8% of parameters are eliminated).

The idea is extremely simple and I like it conceptually.
Currently it looks like my reimplementation on resent 50 is working. 
I do lose a about 1% in accuracy compared to my baseline learned projection implementation.
Is the scale and bias regularized?

I have assigned a score of 6 now.  but I will wait for my final rating when I get the actual results.
Overall the evaluation is seems reasonably thorough many tasks were presented and the model was applied to different architectures.

I also think the manuscript could benefit from the following experiments:
- how does the chosen projection matrix affect performance.
- is the scale needed
I assume the authors did these experiments when they developed the method but it is unclear how important these choices are. 
Including these experiments would make it a more scientific contribution.

The amount of computation saved seems rather limited? Especially since the gradient of the scale parameter has to go through the weight vector?
Therefore my assumption is that only the application of the gradients save a limited amount of time and memory?
At least in my experiments reproducing these results, the computational benefit is not there/obvious.

While I like the idea, the way the manuscript is written is a bit strange at times. 
The introduction appears to be there to be because you need a introduction, not to explain the background. 
For this reason some of the cited work seems a bit out of place.
Especially the universal approximation and data memorization references.
What I find interesting is that this work is the complement of the reservoir computing/extreme learning machines approach.
There the final output layer is trained but the network itself uses random weights.
 
It would be nice if Fig 2 had a better caption. Which dataset, model, ….
Is there an intuition why the training error remains higher but the validation error is identical? This is difficult to get my head round.
Also, it would be nice if an analysis was provided where the computational cost of not doing the gradient update was computed.

review_title: Interesting idea

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes replacing the weights of the final classifier layer in a CNN with a fixed projection matrix.  In particular a Hadamard matrix can be used, which can be represented implicitly.

I'd have liked to see some discussion of how to efficiently implement the Hadamard transform when the number of penultimate features does not match the number of classes, since the provided code does not do this.

How does this approach scale as the number of classes grows very large (as it would in language modeling, for example)?

An interesting experiment to do here would be to look this technique interacts with distillation, when used in the teacher or student network or both.   Does fixing the features make it more difficult to place dog than on boat when classifying a cat?  Do networks with fixed classifier weights make worse teachers for distillation?

review_title: .

final_score: rating: null
review_content: One thing that is not clear to me from the paper are the experiments done with the Hadamard version AND scaling?


review_title: Quick comment

Meta-review: This paper proposes an interesting new idea which creates an interesting discussion.

Decision: Accept (Poster)