Title: Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning
Authors: Werner Zellinger
Thomas Grubinger
Edwin Lughofer
Thomas Natschläger
Susanne Saminger-Platz
Venue: ICLR 2017
Keywords: Transfer Learning
Deep learning
Computer vision

Abstract: The learning of domain-invariant representations in the context of domain adaptation with neural networks is considered. We propose a new regularization method that minimizes the  domain-specific latent feature representations directly in the hidden activation space. Although some standard distribution matching approaches exist that can be interpreted as the matching of weighted sums of moments, e.g. Maximum Mean Discrepancy (MMD), an explicit order-wise matching of higher order moments has not been considered before.
We propose to match the higher order central moments of probability distributions by means of order-wise moment differences. Our model does not require computationally expensive distance and kernel matrix computations. We utilize the equivalent representation of probability distributions by moment sequences to define a new distance function, called Central Moment Discrepancy (CMD). We prove that CMD is a metric on the set of probability distributions on a compact interval. We further prove that convergence of probability distributions on compact intervals w.r.t. the new metric implies convergence in distribution of the respective random variables.
We test our approach on two different benchmark data sets for object recognition (Office) and sentiment analysis of product reviews (Amazon reviews). CMD achieves a new state-of-the-art performance on most domain adaptation tasks of Office and outperforms networks trained with MMD, Variational Fair Autoencoders and Domain Adversarial Neural Networks on Amazon reviews. In addition, a post-hoc parameter sensitivity analysis shows that the new approach is stable w. r. t. parameter changes in a certain interval. The source code of the experiments is publicly available.

TL;DR: A new method for hidden activation distribution matching in the context of domain adaptation.

Review discussion:
What about central moments specifically makes them appealing vs just computing the raw moments themselves? Do the magnitudes of the central moments decay as k grows larger?
That is, do their contributions to the objective become smaller, so that the lower-order moments dominate?
Q1) What about central moments specifically makes them appealing vs just computing the raw moments themselves?
We utilize central moments because of their translation invariance and natural geometric interpretations.
Q2+Q3) Do the magnitudes of the central moments decay as K grows larger?
That is, do their contributions to the objective become smaller, so that the lower-order moments dominate?
Yes, for central moments on the compact interval [0,1] there exists a strictly decreasing upper bound. We introduced a normalization factor, a proposition and some text for clearification. The experimental evaluations on the parameter sensitivity of K further strengthens this answer.
We added some text to better clarify the five main novelties of our paper:
- explicit order-wise matching of higher order moments
- the application of central moments
- the mathematical introduction of the cmd metric including proves for metrical and convergence properties
- the new state-of-the-art accuracy on benchmark data sets
- the accuracy insensitivity w.r.t. parameter changes
One of the main contributions claimed by this paper is that high-order moments are explicitly matched, rather than as in kernel MMD approaches where an implicit weighted sum is used.
However, if the explicit high-order moment matching is critical, then maybe MMD with explicit high-order moment matching can also work well?  This seems to be something to be tested.
In terms of computation cost, there are N^k terms in a kth order moment in N-dimentional space.  If we compute all of them then the computation cost grows exponentially as k grows, I wonder how can the N^5 order computation be done efficiently in the experiments?  Or maybe I misunderstood something?
Q1) MMD with explicit moment computation.
Using the Taylor expansion of the Gaussian kernel in order to obtain an approximation of the MMD with explicit moment terms (sums of moments) was one of our initial ideas.
In particular, we defined a new regularizer MMD5 by bounding the number of terms in the Taylor expansion and setting all weights to 1. An implementation of this early approach can be found in our source code. Due to the low classification accuracy of MMD5-based networks and the recommended paper length of 9 pages, we did not include the results in our paper.
Q2) Computation cost.
We compute the N-dimensional central moments vector as E((X-E(X))^k) with E(.) being the N-dimensional expectation vector.
That is, for each k we compute only N scalar central moments, one for each dimension. The differences ||E((X-E(X))^k)-E((Y-E(Y))^k)|| of the source activations X and the target activations Y are then aggregated and minimized. For clarification, we added some words of explanation in the introduction and in the definitions.
Is the performance of CMD sensitive to the dimensionality of the hidden activations? Will more data be required to estimate the central moments when the number of hidden nodes are large? Will MMD have an advantage over CMD in this case?
Q1) Is the performance of CMD sensitive to the dimensionality of the hidden activations?
Not for the Amazon reviews benchmark data and the model architectures under consideration. This answer is underpinned by a new experiment in subsection Parameter Sensitivity. Please note that this experiment additionally shows that the default setting (\lambda=1, K=5) of the CMD is applicable independently of the number of hidden nodes.
Q2) Will more data be required to estimate the central moments when the number of hidden nodes are large?
No, as we compute the central moments for each hidden node (activation coordinate) separately.
That is, the introduction of new nodes defines new central moments. Each central moment estimation is based on all input examples. We formulated a statement about the effect of more data: The replacement of the expected value with the empirical expectation creates a consistent estimator of the CMD metric.
That is, by increasing the number of input examples, the resulting approximation of the CMD metric converges in probability to the true CMD metric. This can be proven by the joint application of the weak law of large numbers (Billingsley, 2008) with the continuous mapping theorem (Billingsley, 2013). For clarification, we added this statement, the proof sketch and some words on the activation coordinates with the last revisions.
Q3) Has MMD an advantage in the case of more hidden nodes?
In terms of computational complexity, MMD has a clear disadvantage. MMD needs O(N(n^2+mn+m^2+1)) arithmetic operations to be computed whereas CMD needs only O(N(n+m+1)) operations, with N being the number of hidden nodes, n being the number of source samples and m being the number of target samples.
That is, the computation time increase, induced by increasing N, is higher for the MMD.
In terms of parameter sensitivity, the new experiments show that the default setting (\lambda=1, K=5) of the CMD can also be used for higher number of hidden nodes, whereas the MMD needs additional parameter tuning steps. Please note that the parameter tuning process for the Gaussian kernel parameter of the MMD is sophisticated since no labels are available in the target domain. The tuning of the Gaussian kernel parameter is even complicated if labels are available (Chih-Wei Hsu, 2003).
In terms of prediction accuracy, some state-of-the-art networks with weighted sums of different Gaussian kernels and large number of hidden nodes, see e.g. Long et al., 2015 with 4096 hidden nodes and fine-tuning of AlexNet, show lower classification accuracy on the Office data set compared to our CMD networks (subsection Results).
Variational auto-encoders, adversarial networks, and kernel scoring rules like MMD have recently gained popularity as methods for learning directed generative models and for other applications like domain adaptation. This paper gives an additional method along the scoring rules direction that uses the matching of central moments to match two probability distributions. The technique is simple, and in the case of domain adaptation, highly effective.
CMD seems like a very nice and straightforward solution to the domain adaptation problem. The method is computationally straightforward to implement, and seems quite stable with respect to the tuning parameters when compared to MMD. I was skeptical reading through this, especially given the fact that you only use K=5 in your experiments, but the results seem quite good. The natural question that I have now is: how will this method do in training generative models? This is beyond the scope of this paper, but it’s the lowest hanging fruit.
Below I give more detailed feedback.
One way to speed up MMD is to use a random Fourier basis as was done in “Fastmmd: Ensemble of circular discrepancy for efficient two-sample test” by Zhao and Meng, 2015. There are also linear time estimators, e.g., in “A Kernel Two-Sample Test“ by Gretton et al., 2012. I don’t think you need to compare against these approaches since you compare to the full MMD, but they should be cited.
The paper “Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy” by Sutherland et al. submitted to ICLR 2017 as well, discusses techniques for optimizing the kernel used in MMD and is worth citing in section 3.
How limiting is the assumption that the distribution has independent marginals?
The sample complexity of MMD depends heavily on the dimensionality of the input space - do you have any intuitions about the sample complexity of CMD? It seems like it's relatively insensitive based on the results in Figure 4, but I would be surprised if this were the case with 10,000 hidden units. I mainly ask this because with generative models, the output space can be quite high-dimensional.
I’m concerned that the central moments won’t be numerically stable at higher orders when backpropagating. This doesn’t seem to be a problem in the experimental results, but perhaps the authors could comment a bit on this? I’m referring to the fact that ck(X) can be very large for k >= 3. Proposition 1 alleviates my concerns that the overall objective is unstable, I’m referring specifically to the individual terms within.
Figure 3 is rather cluttered, and aside from the mouse class it’s not clear to me from the visualization that the CMD regularizer is actually helping. It would be useful to remove some of the classes for the purpose of visualization.
I would like some clarification about the natural geometric interpretations of K=5. Do you mean that the moments up to K=5 have been well-studied? Do you have any references for this? Why does K >= 6 not have a natural geometric interpretation?
Figure 4 should have a legend
We added your remarks about fast MMD estimators and kernel parameter choices to the Related Work section.
Independent marginal distributions are assumed in order to guarantee that the CMD regularizer (empirical estimate) still has the property of matching the joint distributions.
However, even without this assumption, a zero regularizer value implies equal marginal distributions and convergence in CMD implies convergence in distribution of the marginals.
That is, we don’t need the independence assumption for the application of our method but for guaranteeing some metrical and convergence properties w.r.t. the joint distributions. In the new version of our paper, we added a detailed description of the properties that are obtained with the independence assumption and the properties that are obtained without the independence assumption.
The question about the higher numbers of hidden units (>10.000) is very interesting and we consider the answer as future work.
The absolute value of a central moment ck(X) of a distribution is smaller than the absolute central moment (proof of proposition 1). For this absolute central moment there exists an upper bound (Egozcue et al., 2012). For distributions on the interval [0,1], this bound is decreasing with increasing moment order.
That is, for random variables on [0,1], the absolute values of the higher central moments have a small upper bound.
In order to guarantee distributions on [0,1] we introduce the normalization factor 1/|b-a|^k in the CMD metric, where [a,b] is the interval of the activation functions. This is equivalent to dividing the activations by their maximum range. Please note that we restrict our method on bounded activation functions. Unbounded activations need to be clipped or normalized to be bounded.
We removed figure 3 as it is hardly interpretable and it does not contain any additional information not contained elsewhere.
Indeed, also the sixth and the seventh central moment are related to geometrical properties (hyperskewness, hyperflatness). K=5 is heuristically set a priori large enough to capture rich geometrical information (mean, variance, skewness, kurtosis), and small enough to be computationally efficient.
However, the experiments in subsection Parameter Sensitivity show that similar results are obtained for K>=6. Thanks to your review we changed our argumentation with the current revision.
The previous figure 4 (now figure 3) has a combined legend for all four experiments.
This paper proposed a new metric central moment discrepancy (CMD) for matching two distributions, with applications to domain adaptation.  Compared to a more well-known variant, MMD, CMD has the benefit of not over penalizing the mean, and therefore can focus more on the shape of distribution around the center.
In terms of discriminative power (the ability to tell two distributions apart), MMD and CMD should be equivalent, but in practice I can understand that CMD may be better as MMD tries to match the raw moments which may over penalize data that are not zero centered.
In the paper CMD is used only up to Kth order, and not all the central moments are used, but rather only the diagonal entries are considered in the CMD objective, I think this is mostly motivated for computation efficiency.  A natural comparison with MMD therefore can be made, by also explicitly include raw moments up to Kth order.
Another thing to compare against is to include all moments, not just the diagonal terms, in the objective.  This is computationally expensive, but can be done for e.g. 1st and 2nd orders.
Since the experiments only compare CMD in the above form with kernelized MMD, the claim that explicit moment matching is helpful is not very well supported.  To make this a solid claim CMD should be compared against MMD with explicit raw moments.
The claim that the kernel parameter in MMD is hard to tune and CMD does not have such parameters only applies to kernel MMD, not explicit MMD.  For kernel MMD, there are also studies on how to set these parameters, for example:
Sriperumbudur et al.  Kernel choice and classifiability for rkhs embeddings of probability distributions.
Gretton et al.  A kernel two-sample test.
and also using multiple kernels (Li et al. 2015) which removes the need to tune them.  Tuning the beta directly like done in this paper is usually not the way MMD is tuned.
At least simple heuristics like dividing |x-y|^2 by dimensionality or mean pairwise distance first should be applied first before trying beta in the way done in this paper.
Overall I think CMD could be better than MMD, and could have applications in many domains.
But it also has the problem of not easily kernelizable (you can argue this both ways though).  The experiments demonstrating that CMD is better could be done more convincinly.
We try to approach your advice w.r.t. the experiments and your proposals for further improvement of our investigations. We would be happy to present additional experimental findings regarding explicit moment matching at the conference.
We added the remarks about different MMD versions and kernel parameter choices to the Related Work section.
We decided to use the reverse cross-validation procedure of Zhong et al., 2010 with some modifications for neural networks (Ganin et al., 2016). The method jointly utilizes the source inputs, source labels and target inputs. Each parameter of our models, including the domain regularizer weights, is tuned using this transfer learning specific cross-validation method. With this simple method a one-layer neural network with MMD domain-regularization shows higher accuracy than the state-of-the-art model VFAE on the Amazon reviews benchmark data set. There are also other papers that utilize cross-validation for the optimal kernel choice, see Sutherland et al., 2016 subsection 2.2 for a comprehensive summary.
We added your references and notes for studies on the unsupervised tuning of the parameter beta with the new revision to the Related Work section.
In addition, we added your suggestions and your proposals for further improvements of the experiments to the Conclusion and Outlook section.
The work introduces a new regularization for learning domain-invariant representations with neural networks. The regularization aims at matching the higher order central moments of the hidden activations of the NNs of the source and target domain. The authors compared the proposed method vs MMD and two state-of-art NN domain adaptation algorithms on the Amazon review and office datasets, and showed comparable performance.
The idea proposed is simple and straightforward, and the empirical results suggest that it is quite effective. The biggest limitation I can see with the proposed method is the assumption that the hidden activations are independently distributed.
For example, this assumption will clearly be violated for the hidden activations of convolutional layers, where neighboring activations are dependent. I guess this is why the authors start with the output of dense layers for the image dataset. Do the authors have insight on if it is beneficial to start adaptation from lower level? If so, do the authors have insight on how to relax the assumption? In these scenarios, if MMD has an advantage as it does not make this assumption?
Figure 3 does not seems to clearly support the boost of performance shown in table 2. The only class where the new regularization brings the source and target domain closer seem to be the mouse class pointed by the authors. Is the performance improvement only coming from this single class?
Independent marginal distributions are assumed to guarantee that the CMD regularizer (empirical estimate) with infinite many terms has the property of matching the joint distributions.
However, even without this assumption, a zero regularizer value implies equal marginal distributions and convergence in CMD implies convergence in distribution of the marginals.
That is, we don’t need the independence assumption for the application of our method but to guarantee some metrical and convergence properties w.r.t. the joint distributions. From your review we have learned that it is not clear why the assumption is needed. In the new version of our paper, we added a detailed description of the properties that are obtained with the independence assumption and the properties that are obtained without the independence assumption.
Recent findings reveal that deep features must eventually transition from general to specific along the network, and the transferability of features drops significantly in higher layers with increasing domain discrepancy (Long et al., 2015; Yosinski et al., 2014).
Therefore, we try to force the network to train transferable features in the higher layers by adaptation. We compare the accuracy of our method with various state-of-the-art models on Office and most of them (AdaBN, CORAL, Deep CORAL, DANN, DAN, DDC) start adaptation from higher layers (section 5.2). Tzeng et al., 2014 report higher accuracies based on MMD regularization when starting from the highest layer compared to lower ones.
However, our method is also applicable to lower layers without any modification. We were just not able to produce competitive results with lower layer adaptation.
Although not clearly visible in figure 3 of the last revision, 58% of the individual classes are improved in this example.
However, we removed the figure as it is hardly interpretable and it does not contain any additional information not contained elsewhere.

Meta-review:
role: PC
data: comment: The proposed Central Moment Discrepancy criterion is well-described and supported in this paper. Its performance on domain adaptation tasks is good as well. The reviewers had several good comments and suggestions and the authors have taken most of these into account and improved the paper considerably. The paper thus makes a nice contribution to the distribution matching literature and toolbox.

Decision: Accept (Poster)