Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes to quantify the inter-identify variation by utilizing pairs of similar expressions for identify-invariant facial expression recognition. To find pairs of similar expressions from different identities, the authors define the inter-feature similarity as a transportation cost. Then, they perform optimal identity matching to find the optimal flow with minimum transportation cost by Sinkhorn-Knopp iteration. 
strengths_and_weaknesses: Strengths:
1. The authors propose a method to quantify the inter-identify variation, which is different from the previous works in a non-quantitative way.
2. The experimental results exceed most existing works
Weaknesses:
1. As mentioned in the paper, inter-identify variation is quantified by pairs of similar expressions. However, when the expressions of individuals with different IDs differ greatly in a batch, that is, the precondition fails. The methods may not work.
2. The variations between reference and targets are calculated in a batch, but different references will be chosen in different batches. The variations among references are not considered.
3. Although the authors describe the training process in detail, they do not mention the testing process, which I believe is a non-trivial process.
4. The authors only verify that the proposed method performs better in VA prediction, but it is lack experiments on expression and AU prediction.
questions: 1. Please give reasonable explanations for the above weaknesses.
2. In line 169, I do not find the definition of $L_{va}$ in the cited paper [50]. Maybe the author made a mistake.
limitations: Please see the Questions.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a method for facial expression recognition (by means of valence and arousal) which aims to remove the factors that relate to identity from the backbone features, so as to make the classifier be robust against identity-specific features. In particular, a method is proposed to remove from the feature representation the identity shift, by computing the optimal transport between the batch-specific features in a way to represent the image-specific domain shift from the other images in the batch. The features are then normalized according to the computing shifting mean and scale and passed through the final classifier which regresses the values of valence and arousal. The use of optimal transport to assign the image-specific mean vector is grounded on the idea that the batch-specific features can be rearranged, while preserving the total mass distribution, so as to remove the identity-based similarities encountered in the batch. The method is validated in a set of challenging datasets achieving competitive results. 
strengths_and_weaknesses: The paper is technically novel and achieves competitive results and hence has the merits to be accepted at NeurIPS. The use of OT to disentangle the factors of identity is novel, the motivation is clear and the experiments are compelling. The paper is well accompanied by algorithms and descriptions that help understanding the derivation, with further pseudo-code for the crucial parts in the Supplementary Material. The paper could benefit from a better clarity of presentation and elaboration on the significance and meaning of the learned shift, and thus I would like to particularly ask the authors to improve the manuscript’s presentation and writing substantially for better clarity.

My main comments and concerns are listed below:

- Digging under the hood of Equations 3 and 4 as well as how the cost matrix is computed based on similarity, I can spot that there are some substantial similarities and differences between using the OT assignment and using cross-attention to attenuate the influence of the global features. Plugging the cost matrix into Eqn 3 and observing the “residual” behaviour of Eqn. 4 can derive to expressing the new features as a linear combination of the other images’ features, given by a weighted similarity between them. This resembles to me the use of cross-attention with X being the attention weights. Obviously, the authors propose a completely different way to assign these attention weights, but after reading the full derivation I would like to ask the authors if a small transformer encoder instead to compute the mean features in Equation (3) would also enforce the learned features to be identity-invariant. I think that such exploration is important and the paper would gain significant strength such comparison would be given. 

- It is not mentioned in the paper (or I missed that), but there is no reference to the process of inference. My understanding is that at test time no shift is obtained because the features are already expected to be “centered”, but some clarification in this regard would be appreciated.

- I think the paper would gain strength by including the performance of the proposed approach on AffectNet and SEWA, while AffWild2 and AffWild are challenging, they are a subset of each other, and AFEW-VA has recently given way to the aforementioned datasets.

- What makes the number of parameters on Table 1 be different for the proposed method and that of CAF for the AL-tuned and ResNet-18 backbones? 

- I am a bit confused about the results shown in Figure 5. First, the behaviour of the curves is a bit counterintuitive as e.g. there is a drop in two points of CCC between using 18 and 19 subjects, which is recovered after using 20 subjects. What is this behaviour due to? In addition, it is not clear to me what does w/o L_va mean, considering that this is the main training loss. The blue curve, does it correspond to a training using the PCC + CCC losses? Please do clarify. 

- It is important also to understand the contribution of the training losses in the method. How does this compare against competing methods? If they had been trained without L_ccc or P_ccc then I believe the authors should also consider a fair evaluation using their method trained with the same losses. This will indicate the contribution of the losses in the performance gain.

- I didn’t get quite well what Figure 4 attempts to represent, so I would like to request the authors to elaborate on it.

- Finally, as mentioned before, I believe the paper would greatly benefit from better writing and presentation. The readability of the paper is somewhat poor affecting its reach. 


questions: As stated above, my main concerns or questions can be summarized as:
1) Exploring or comparing against of attention-based methods for residual learning. 

2) Inference

3) Use of SEWA or AffectNet to compare against CAF

4) Different number of parameters for same networks in Table 1.

5) Different losses and comparison against existing methods.

6) Explanation of Figure 4

limitations: The limitations are properly addressed in the paper and hence I have no further questions in this regard.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes an approach to identity invariant facial expression recognition. Identity matching is formulated as an optimal transport problem. The optimal flow for this problem is found using Sinkhorn-Knopp. Evaluations are conducted on three public FER datasets. 
strengths_and_weaknesses: Strengths

Paper proposes a solution to a challenging problem in FER. Problem is well formulated in paper including needed background information to setup proposed solution. State of the art results are detailed on multiple public datasets. Results show the proposed approach can easily be integrated into different models. Ablation study on the relevance weights, and impact of number of IDs is well conducted. 

Weaknesses

As the paper focuses on identity invariant FER, other factors such as age, gender, and race could have a large impact on the ID matching and FER learning. While this is briefly discussed in Section 5.4 (A5), the discussion is largely superficial with a tech report being cited that talk about differences in facial expression across age. It is not clear how the proposed approach can handle these changes across different demographics. Handling this type of shift is important for generalized FER.


questions: How can the proposed approach handle shift due to demographics such as age, gender, or race?
limitations: Negative impact has been addressed. Limitation regarding the reference ID is detailed, however, limitations on the scope of demographics and the proposed approach are not detailed.
review_title: null

Meta-review: metareview: Authors propose a new strategy for a hard problem that reviewers found compelling and novel. The  experimental details are complex and we encourage the authors to address the many issues the reviewers raise. 
confidence: Certain
Decision: Accept