Title: Trusting SVM for Piecewise Linear CNNs
Authors: Leonard Berrada
Andrew Zisserman
M. Pawan Kumar
Venue: ICLR 2017

Abstract: We present a novel layerwise optimization algorithm for the learning objective of Piecewise-Linear Convolutional Neural Networks (PL-CNNs), a large class of convolutional neural networks. Specifically, PL-CNNs employ piecewise linear non-linearities such as the commonly used ReLU and max-pool, and an SVM classifier as the final layer. The key observation of our approach is that the prob- lem corresponding to the parameter estimation of a layer can be formulated as a difference-of-convex (DC) program, which happens to be a latent structured SVM. We optimize the DC program using the concave-convex procedure, which requires us to iteratively solve a structured SVM problem. This allows to design an opti- mization algorithm with an optimal learning rate that does not require any tuning. Using the MNIST, CIFAR and ImageNet data sets, we show that our approach always improves over the state of the art variants of backpropagation and scales to large data and large network settings.

TL;DR: Formulating CNN layerwise optimization as an SVM problem

Review discussion:
1) The baseline in the experiment is a strawman variant of backprop (without batch norm, drop out, etc). I think this is unfair. Batch norm and dropout have been proved to be very useful for CNN models. If the proposed method cannot take advantage of batch norm/dropout, then this is an advantage of backprop.
2) The dataset is too small for today's standard. Why not use something like ImageNet?
We thank the reviewer for the questions regarding the experiments section.
Our main goal is to compare different algorithms that optimize the same objective function, namely problem (3), in terms of three measures: (i) training accuracy; (ii) the testing accuracy; and (iii) the learning objective value. Our layerwise optimization is complementary to backprop. To highlight this, we have incorporated new results that minimize the learning objective of a network that incorporates batch normalization (subsection 5.2). Similar to the original experiments, our algorithm leads to significant improvements on all three measures.
It is worth noting that our method can be extended to handle other forms of difference-of-convex non-linearities, such as batch normalization.
However, in this work we focus on PL functions that have the added advantage of being amenable to Frank-Wolfe style optimization.
In the appendix of the revised paper, we also include new results that highlight the robustness of our method to hyperparameter settings compared to all the standard backprop variants. Please note that all the methods use the same learning objective. By obtaining lower objective values, our method is able to escape the bad local minima that are encountered by backprop.
This paper proposes a new approaches for optimizing the objective of CNNs. The proposed method uses a lay-wise optimization, i.e. at each step, it optimizes the parameters in one layer of CNN while fixing the parameters in other layers. The key insight of this paper is that, for a large class of CNNs, the optimization problem at a particular can be formulated as optimizing a piecewise linear (PL) function. This PL function optimization happens to be the optimization problem commonly encountered in latent structural SVM. This connection allows this paper to borrows ideas from the latent structural SVM literature, in particular concave-convex procedure, to learn the parameters of CNNs.
Overall, the paper is well-written. Traditional, CNNs and structural SVMs are almost two separate research communties. The connection of CNNs to latent structural SVM is interesting, and might bridge the gap and facilitate the transferring of ideas between these two camps.
Of course, the proposed method also has some limitations. 1) It is limited to layer-wise optimization. Nowadays layer-wise optimization is essentially a coordinate descent algorithm and is not really a competitive strategy in learning CNNs.
When you choose layer-wise optimization, you already lose something in terms of optimizing the objective (since you are using coordinate descent, instead of gradient descent).
Of course, you also gain something since now you can guarantee that each coordinate descent step always improve the objective. It is not clear to me how the loss/gain balances each other. 2) This paper focues on improving the optimization of CNN objective.
However, we all know that a better objective does not necessarily correspond to a good model (e.g. due to overfitting).
Although the SGD with backprop in standard CNN learning does not always improve the solution of the objective (unlike the proposed method in this paper), but to me, this might be a good thing since it can prevent overfitting (the goal of learning is not to get better solution for the optimization problem in the first place -- the optimization problem is merely a proxy to learn a model with good generalization ability).
The experiment is a bit weak.
1) Only CIFAR10 is used. This is a very small dataset by today's standard, while CNNs are typically used in large-scale datasets, such as ImageNet. It is not clear whether the conclusions of this paper still hold when applied on ImageNet.
2) This paper only compares with a crippled variant of SGD (without batch normalization, dropout, etc).
Although this paper mentions that the reason is that it wants to focus on optimization.
But I mentioned earlier, SGD is not designed to purely obtain the best solution that optimizes the objective, the goal of SGD is to reasonably optimize the objective, while preventing overfitting.
So the comparison to SGD purely in terms of the optimization is that meaningful in the first place.
We thank the reviewer for the comments, and provide some clarifications below.
Comment: “layer-wise optimization is essentially a coordinate descent algorithm and is not really a competitive strategy in learning CNNs [...] It is not clear to me how the loss/gain balances each other”
Response:  The performance impact of using coordinate descent is indeed an open question. For this work, we point out that:
(i) LW-SVM improves over SGD solutions despite optimizing only one layer at a time,
(ii) LW-SVM trained from scratch can also reach competitive results on CIFAR-10,
(iii) The connection between PL-CNNs and latent SVMs opens an interesting research direction that may further improve on LW-SVM (for example,  by extending LW-SVM to a global optimization algorithm that updates all layers simultaneously).
Comment: “This paper only compares with a crippled variant of SGD (without batch normalization, dropout, etc).”
Response: In the revised version of the paper, we report results with batch-normalization and show similar or greater improvements of LW-SVM over the baselines.
Comment: “a better objective does not necessarily correspond to a good model (e.g. due to overfitting).
Although the SGD with backprop in standard CNN learning does not always improve the solution of the objective (unlike the proposed method in this paper), but to me, this might be a good thing since it can prevent overfitting”
Response: The experiments of this work suggest that the improvement made by LW-SVM on the training data generalize well in our settings.
In particular, results with batch-normalization show significant improvements on the testing accuracy. A better optimization algorithm may also lead to a better understanding of the deficiencies of the current learning objective, which could in itself be of interest to the representation learning community.
Comment: “Only CIFAR10 is used. This is a very small dataset by today's standard, while CNNs are typically used in large-scale datasets, such as ImageNet.”
Response: In principle the architectures used on ImageNet (such as VGG-16, ResNets etc.) can be treated like the models used in our experiments, and we expect to obtain similar results on these.
In addition to the current results, we are currently evaluating our method on CIFAR-100 and ImageNet.
A layer wise optimization for CNNs with ReLU activations and max-pooling is proposed and shown to correspond to a series of latent structured SVM problems. Using CCCP style optimization a monotonic decrease of the overall objective function can be guaranteed.
Summary:
I think the discussed insights are very interesting but not presented convincingly.
Firstly, claims are emphasized which are often violated in practice (e.g., no convergence guarantees due to mini-batches), statements could be validated more convincingly (e.g., is monotone convergence a curse or a blessing), the experimental evaluation should be extended.
In summary, I think the paper requires some more attention to form a compelling story.
Quality: I think some of the techniques could be described more carefully to better convey the intuition. At times apples are compared to oranges, e.g., back propagation is contrasted with CCCP.
Clarity: Some of the derivations and intuitions could be explained in more detail.
Originality: The suggested idea is reasonable albeit heuristics are required.
Significance: Since the experimental setup is somewhat limited according to my opinion, significance is hard to judge at this point in time.
Details:
1. I think the provided guarantees for the optimization procedure are certainly convenient theoretically but their practical relevance still needs to be demonstrated more convincingly, e.g., mini-batch optimization alleviates any form of monotonic decrease.
Hence the emphasize in the paper is somewhat misguided according to my opinion and given he current experimental evaluation.
2. In spirit similar is work by B. Amos and J. Kolter, Input-Convex Deep Networks (https://openreview.net/pdf?id=ROVmA279BsvnM0J1IpNn) ICLR Workshop Track 2016; which should probably be mentioned.
3. I think the comparison between backprop and the discussed CCCP approach is not really appropriate. Note that backprop is a mechanism to compute gradients and is not at all related to any optimization technique/algorithm. This means that backprop combined with Armijo line-search would for example result in convergence guarantees.
Hence a statement like `one of the main advantages of our approach compared to back propagation and its variants, which fail to provide similar guarantees on the value of the objective function from one iteration to the next’ is according to my opinion superficial.
4. More justification regarding the argument that search for the step-size is a disadvantage seems necessary. There is evidence that noise introduced by mini-batches and inaccurate/no line search is actually beneficial.
In contrast the proposed CCCP procedure may converge pre-maturely to a local optimum close to the initialization. Since mini-batches are used no guarantees are available in any case.
Hence I think additional evidence for the fact that such convergence guarantees don’t result in premature stopping seems necessary.
5. The observation 1 required to convert the layer wise procedure into an SVM seems rather ad-hoc. Note that you can easily construct examples where Eq. (9) and Eq. (7) differ significantly.
6. I personally think the experimental evaluation is conducted on examples that are too small and some results are obvious. E.g., LW-SVM always improves over the solution of the SGD algorithm. If the authors were to use backtracking line search they could also improve upon LW-SVM.
We thank the reviewer for the comments and provide some clarifications below.
Comment: “comparison between backprop and the discussed CCCP approach is not really appropriate”
Response: We use the term ‘backpropagation’ to denote the overall subgradient descent (SGD) algorithm. This includes the forward pass and the backward pass (which computes the subgradient), as well as the state of the art variants of the step size computations, namely, Adagrad, Adadelta and Adam. As our main aim is to improve the optimization of the learning objective (3), a comparison with the existing optimization algorithms provides an appropriate baseline.
Comment: “mini-batch optimization alleviates any form of monotonic decrease”.
Response: Does the reviewer mean that by using mini-batch optimization we have lost our monotonic decrease guarantee? If so, this is not correct. There are two distinct monotonic improvements that we guarantee here, even when using mini-batches:
(i) a monotonic decrease in the overall objective function at each outer iteration of the CCCP algorithm. This is because we solve a convex problem and therefore obtain the optimal solution, which is at least as good as the initial point.
(ii) a monotonic increase in the dual of the convex problem at each step. Because the dual is smooth and quadratic, we perform a coordinate ascent on the dual variables with an optimal step-size. This feature of the BCFW algorithm (Lacoste-Julien et al., 2013) guarantees a monotonic increase despite the fact that the conditional gradient is computed using samples from a mini-batch / block of coordinates.
Comment: “More justification regarding the argument that search for the step-size is a disadvantage seems necessary”, “line-search would for example result in convergence guarantees”.
Response: Another noteworthy advantage of our work is that we can analytically compute the optimal step-size for the conditional gradient obtained over a mini-batch.
Indeed, this fact has been exploited successfully for structural SVM optimization with significant improvements over SGD methods.
For backpropagation style algorithms (that is, SGD on learning objective (3)), the computation of the step-size is not computationally feasible. This is due to the fact that the evaluation of the objective for each putative step-size would require an entire epoch.
Furthermore, for PL-CNN, the objective (3) is only sub-differentiable.
Hence, even an exhaustive line search cannot guarantee a monotonic improvement.
In other words, by establishing a connection between deep networks and structural SVMs, we have been able to successfully transfer the two advantages of BCFW to the deep learning domain (monotonic improvement of dual using mini-batches, efficient computation of the optimal step-size).
Comment: “ In spirit similar is work by B. Amos and J. Kolter, Input-Convex Deep Networks (https://openreview.net/pdf?id=ROVmA279BsvnM0J1IpNn) ICLR Workshop Track 2016”.
Response: We thank the reviewer for the reference (we have found a full version at https://arxiv.org/pdf/1609.07152v2.pdf).
While our method deals with convexity w.r.t. the parameters to improve the training task, the aforementioned work analyzes how convexity w.r.t input or the output of the network helps with the inference (for cases with complex outputs). We will include this reference in the future revision of the paper.
Comment: “The observation 1 required to convert the layer wise procedure into an SVM seems rather ad-hoc.”
Response: In our revised version of the paper, we have updated Observation 1 with a principled algorithm that is guaranteed to find the optimal solution of the convex problem.
In summary, all the theoretical guarantees presented in our work are applicable, despite the use of mini-batches and the use of a modified conditional gradient algorithm.
This paper presents a novel layer-wise optimization approach for learning CNN with piecewise linear nonlinearities.  The proposed approach trains piecewise linear CNNs layer by layer and reduces the sub-problem into latent structured SVM, which has been well-studied in the literature.
In addition, the paper presents improvements of the BCFW algorithm used in the inner procedure.
Overall, this paper is interesting.
However, unfortunately, the experiment is not convincing.
Pros:
- To my best knowledge, the proposed approach is novel, and the authors provide nice theoretical analysis.
- The paper is well-written and easy to follow.
Cons:
- Although the proposed approach can be applied in general structured prediction problem, the experiments only conduct on a simple multi-class classification task. This makes this work less compelling.
- The test accuracy performance on CIFAR-10 reported in the paper doesn't look right. The accuracy of the best model reported in this paper is 70.2% while existing work often reports 90+%.
For example, https://arxiv.org/pdf/1412.6806.pdf showed an accuracy of 91% without data augmentation.
Also, CIFAR-10 is a relatively small dataset,
Other comments:
- If I understand correctly, BCFW only guarantees monotonically increasing in the dual objective and does not have guarantees on the primal objective. Especially, in practice, the inner optimization process often stops pretty early (i.e., stops when duality gap is still large).
Therefore, when putting them together, the CCCP procedure may not monotonically decrease as the inner procedure is only solved approximately. The authors should add this note when they discuss the properties of their algorithm.
We thank the reviewer for his comments, which we discuss below:
Comment: “Although the proposed approach can be applied in general structured prediction problem, the experiments only conduct on a simple multi-class classification task.”
Response: In principle the architectures used on ImageNet (such as VGG-16, ResNets etc.) can be treated like the models used in our experiments, and we expect to obtain similar results on these.
In addition to the current results, we are currently evaluating our method on CIFAR-100 and ImageNet, which are more challenging multi-class classification tasks. We do take note that it would be interesting to test the method on structured prediction tasks as well.
Comment: “The test accuracy performance on CIFAR-10 reported in the paper doesn't look right. The accuracy of the best model reported in this paper is 70.2% while existing work often reports 90+%.”
Response: The best score of 70% is obtained with a network which does not use batch normalization, which makes an important difference. The results with batch-normalization in the revised version of the paper reach around 78% testing accuracy. Taking into account the reviewer’s comments, we are currently running experiments with deeper architectures to improve this score.
Comment: “If I understand correctly, BCFW only guarantees monotonically increasing in the dual objective and does not have guarantees on the primal objective.
In practice, the inner optimization process often stops pretty early (i.e., stops when duality gap is still large).
Therefore, when putting them together, the CCCP procedure may not monotonically decrease as the inner procedure is only solved approximately. The authors should add this note when they discuss the properties of their algorithm.”
Response: The reviewer is correct that during the inner optimization, only the dual objective is guaranteed a monotonic improvement.
In practice, we do run BCFW with a sufficient number of iterations to yield a very small duality gap.
Therefore the method provides a monotonic decrease in practice. To illustrate this, we have plotted the training objective and the dual gap for an experiment presented in the paper (Adadelta with batch-norm + LW-SVM, Figure 3 in the revised paper): https://drive.google.com/file/d/0BxXMf_vDT8vCSFQwOWl0aldzeWs/view?usp=sharing (one data point at the end of each inner iteration of the CCCP). We will add a note about these points in the future revision of the paper.
tldr: New results on ImageNet, CIFAR-100, improved results on CIFAR-10.
We thank the reviewers for their helpful feedbacks. We list here the changes made in the revisions of the paper (version 1 being the original submission read by the reviewers).
List of changes in version 2:
1) New results with batch-normalization on CIFAR-10 (new subsection 5.2)
2) Clarification of the objective of the paper and experiments (Methods paragraph in subsection 5.1)
List of changes in version 3:
1) New results on CIFAR-10: deeper architecture for a stronger baseline (subsection 5.2)
2) New results on CIFAR-100 (subsection 5.2)
3) Re-wording of the experiments section and removal of previous experiments on CIFAR-10 (with and without batch normalization) (section 5)
4) New Appendix about the computation of the feature vectors and detailed example (Appendix B).
5) Infeasibility of standard line-search in Introduction
6) New references, including suggestions from the reviewers (section 2)
7) More compact abstract
8) Inclusion of batch normalization in Discussion (section 6)
9) Minor rewording and typo fixes throughout the paper.
List of changes in version 4:
1) Added ImageNet results (subsection 5.3)

Meta-review:
role: PC
data: comment: The authors present a novel layer-wise optimization approach for learning convolutional neural networks with piecewise linear nonlinearities. The proposed approach trains piecewise linear ConvNets layer by layer, reduces the sub-problem into latent structured SVM. 
 
 Reviewers mainly expressed concerns about the experimental results, which the authors have diligently addressed in their revised versions. While the reviewers haven't updated explicitly their reviews, I believe the changes made should have been sufficient for them to do so.
 
 Thus, I recommend this paper be accepted.

Decision: Accept (Poster)