Title: Charged Point Normalization: An Efficient Solution to the Saddle Point Problem
Authors: Armen Aghajanyan
Venue: ICLR 2017
Keywords: Deep learning
Computer vision
Optimization

Abstract: Recently, the problem of local minima in very high dimensional non-convex optimization has been challenged and the problem of saddle points has been introduced. This paper introduces a dynamic type of normalization that forces the system to escape saddle points. Unlike other saddle point escaping algorithms, second order information is not utilized, and the system can be trained with an arbitrary gradient descent learner. The system drastically improves learning in a range of deep neural networks on various data-sets in comparison to non-CPN neural networks.


Review discussion:
Please test the following hypothesis:
equation (8) introduces a momentum term that helps to demonstrate better performance than SGD without momentum,
\alpha=0.95 was a good enough choice to show better performance than non-tuned algorithms on the monkey saddle function.
Thank you for reading my paper!
Sorry, I'm a little unclear on the hypothesis. Are you recommending to increase the SGD momentum to 0.95, on page 8?
it is up to you how to check the hypothesis
when you consider momentum in other approaches, you often set it to 0.9 and not 0.95 as for CPN (for SGD in section 3.3.1, for ADAM in section 3.4.2, SGD and ADAM in section 4.3, etc), why?
I see.
I'll edit my paper to include explanations behind choice of parameters behind CPN.
I uploaded a new version of the paper that talks about choice of hyper-parameters for the saddle point toy problem. I also added clarifications throughout the paper.
Please let me know what you think.
- You mention  about "while the function R contains parameters: β, p, φ and". Where do you use φ as a parameter of R(.)?
- Do you have more theoretically grounded justification for the charged point normalization? (In terms of optimization theory)
- Can you compare against batch-norm? Does CPN also work well with batch/layer-norm?
- Can you add captions to your Figures? For most of the figures it is not really clear whether you are showing validation or training curves.
- Can you also report the generalization performance for the tasks that you have tested CPN.
For example it would be interesting to see how CPN effects the generalization on bAbI task.
Thank you for your comments.
To respond to your questions:
- The $\phi$ parameter is a function that combines collapses two parameters into one parameter of the same shape. It can be thought of as a merging function. In this paper, we stick to an exponential weighted average update as the merging function. Future works can analyze various more complicated schemes for merging these two parameters.
- In terms of grounded justification via optimization theory, we had trouble showing CPN's success analytically due to its dynamic nature. The paper analytically describes the problem of saddle points and shows that SGD is good enough, in principle, to escape saddle points. The problem arises in the time until convergence which we experimentally show CPN decreases.
- BatchNorm would be an interesting thing to test out. I'm not sure there is enough time to rerun all experiments with BatchNorm+HyperParameter optimization.
But if you believe this will significantly improve the paper, we are willing to try. BatchNorm scales inner activations in order to reduce covariate shifting between the layers.
In theory, this scaling should not have much effect on CPN.
- Yes. The next set of revisions on the paper will include captions and explanation on choices of hyper-parameters.
- The reason we didn't include validation losses is because we are attempting to improve optimization routines, whether or not it is overfitting is due to the regularization scheme being employed, another minor reason is including more plots would cause the number of pages to exceed the limit. If you would like I can place generalization results on the BaBl dataset.
Thank you very much for your comments.
Armen A.
I've added revisions about hyper-parameters and clearer expectations. I also stated that all the graphs in the paper represent train loss.
Please let me know what you think.
The hyperparameters of gradient descent seem to be chosen once and fixed. Would optimizing the gradient descent hyperparameters lead to equivalent performance as the CPN method?
Following up on another reviewer's question: CPN seems closely related to momentum. Can you provide a clear example to show how CPN is qualitatively distinct from momentum? (I believe it is, but this could be clarified further in the paper)
Thank you for reading.
A grid search was run for GD hyperparameters and the only parameters that successfully escaped the monkey saddle point toy problem were those that are unrealistic. They all contained extremely high learning rates and high momentums, that are never used when training real nets. I'll add that distinction in the paper.
I can add a section to describe the similarities between momentum and CPN, mainly being that CPN charges are self-adapting, and contain more potential to move points out of saddle points. The toy problem shows that momentum based methods alone, with reasonable parameters, are not enough to escape saddle points. The paper has a set of other experiments which show that CPN allows momentum based methods to escape saddle points, where without CPN it wouldn't. Would you recommend showing another toy problem?
Thank you.
The research direction taken by this paper is of great interest.
However, the empirical results are not great enough to pay for the weaknesses of the proposed approach (see Section 6).
"Throughout this paper the selection of hyper-parameters was kept rather simple." but the momentum term of CPN is set to 0.95
and not 0.9 as in all/most optimizers CPN is compared to. I suppose that the positive effect of CPN (if any) is mostly due to its momentum term.
Thank you for your review.
But I would please ask you to re-evaluate.
The paper ran multiple experiments that tested various architecures using various momentum methods ranging from RMSProp to Adam. If the positive effect of CPN was simply due to the momentum we would not see uniform improvements over all experiments using a large varying set of optimization routines. And for the toy-problem the only momentums that solved the toy-problem were unrealistic parameters that could not be used in a real setting.
We also believe that the theoretical perspective of this paper on saddle points is important. I ask you respectfully to reconsider.
Thank you.
Armen.
Thanks for your request.
1) MNIST experiments
Section 3.2.1 and 3.2.2
You compare SGD without momentum and CPN where momentum is present. The difference in performance is negligible.
This supports my arguments "results are not great enough"
2) CIFAR experiments
You compare SGD with momentum *0.9* and CPN with momentum *0.95*. The default approach outperforms CPN during the first 30 epochs, then CPN outperforms SGD. How you selected 0.9 for SGD and β = 0.01, λ = 0.1 for CPN? The paper suggests that "by hand around 4-8 times". Which values of momentum you tried exactly? If only 0.9 for SGD with momentum and 4-8 for CPN then feel free to scale up the x-axis of CPN by 4-8.
3) BABI dataset
"For the LSTM architecture, CPN hyper-parameters were: β = 0.0025, λ = .03, α = 0.95. For the GRU
architecture, CPN hyper-parameters were: β = 0.1, λ = .1, α = 0.95."
Here, you jumped from 0.0025, λ = .03 to β = 0.1, λ = .1 with "4-8" trials. Does not look trivial to me. The grid had to be β in [0.0025; 0.1] and λ in [0.03; 0.1] to get your hyperparameter settings in 4 trials! Adam was not optimized I suppose.
4) Toy example
You show that by combing some baseline algorithms and CPN with *tuned hyperparameters* we can leave the saddle point. Great. Did you try to optimize hyperparameters of the baselines as well? This would be more fair, I guess.
I suppose that any reasonable random walk (optionally, gradient-biased) could help to escape the saddle point of this 2-dimensional problem.
I do not claim that the approach does not make sense.
Instead, I think that the current comparison analysis is not convincing enough.
Thank you for your comments.
I mentioned in the similarities to momentum in the paper. If momentum was the only positive effect of CPN than just increasing momentum would cause training to escape saddle points, but in reality, this does not happen. Hyperparameter search through standard training also showed that higher momentums wouldn't work.
Thank you.
This paper proposes a novel method for accelerating optimization near saddle points. The basic idea is to repel the current parameter vector from a running average of recent parameter values. This method is shown to optimize faster than a variety of other methods in a variety of datasets and architectures.
On the surface, the proposed method seems extremely close to momentum. It would be very valuable to think of a clear diagram illustrating how it differs from momentum and why it might be better near a saddle point. The illustration of better convergence on the toy saddle example is not what I mean here—optimization speed comparisons are always difficult due to the many details and hyper parameters involved, so seeing it work faster in one specific application is not as useful as a conceptual diagram which shows a critical case where CPN will behave differently from—and clearly qualitatively better than—momentum.
Another way of getting at the relationship to momentum would be to try to find a form for R_t(f) that yields the exact momentum update. You could then compare this with the R_t(f) used in CPN.
The overly general notation $\phi(W,W)$ etc should be dropped—Eqn 8 is enough.
The theoretical results (Eqn 1 and Thm 1) should be removed, they are irrelevant until the joint density can be specified.
Experimentally, it would be valuable to standardize the results to allow comparison to other methods.
For instance, recreating Figure 4 of Dauphin et al, but engaging the CPN method rather than SFN, would clearly demonstrate that CPN can escape something that momentum cannot.
I think the idea here is potentially very valuable, but needs more rigorous comparison and a clear relation to momentum and other work.
Thank you for comments.
I'd like to note that if momentum was capable of escaping saddle points with the same proficiency as CPN then all that would be necessary is raising the momentum term.
But in reality, this approach never works as it causes the gradients to expode.
I mentioned the similarity to momentum in an update to the paper. The theoretical result show the asymptotic growth of saddle points independently of the joint probability.
Thank you.
Summary:
This paper proposes a regularizer that is claimed to help escaping from the saddle points. The method is inspired from physics, such that thinking of the optimization process is moving a positively charged particle would over the error surface which would be pushed away from saddle points due to the saddle point being positively changed as well. Authors of the paper show results over several different datasets.
Overview of the Review:
Pros:
- The idea is very interesting.
- The diverse set of results on different datasets.
Cons:
- The justification is not strong enough.
- The paper is not well-written.
- Experiments are not convincing enough.
Criticisms:
I liked the idea and the intuitions coming from the paper.
However, I think this paper is not written well. There are some variables introduced in the paper and not explained good-enough, for example in 2.3, the authors start to talk about p without introducing and defining it properly. The only other place it appears before is Equation 6. The Equations need some work as well, some work is needed in terms of improving the flow of the paper, e.g., introducing all the variables properly before using them.
Equation 6 appears without a proper explanation and justification. It is necessary to explain it what it means properly since I think this is one of the most important equation in this paper. More analysis on what it means in terms of optimization point of view would also be appreciated.
$\phi$ is not a parameter, it is a function which has its own hyper-parameter $\alpha$.
It would be interesting to report validation or test results on a few tasks as well. Since this method introduced as an additional cost function, its effect on the validation/test results would be interesting as well.
The authors should discuss more on how they choose the hyper-parameters of their models.
The Figure 2 and 3 does not add too much to the paper and they are very difficult to understand or draw any conclusions from.
There are lots of Figures under 3.4.2 without any labels of captions. Some of them are really small and difficult to understand since the labels on the figures appear very small and somewhat unreadable.
A small question:
* Do you also backpropagate through $\tilde{\mW}_i^t$?

Meta-review:
role: PC
data: comment: The paper proposes a method for accelerating optimization near saddle points when training deep neural networks. The idea is to repel the current parameter vector from a running average of recent parameter values. The proposed method is shown to optimize faster than a variety of other methods in a variety of datasets and architectures.
 
 The author presents a fresh idea in the area of stochastic optimization for deep neural networks. However the paper doesn't quite appear to be above the Accept bar, due to remaining doubts about the thoroughness of the experiments.We therefore invite this paper for presentation at the Workshop track.

Decision: Invite to Workshop Track