Venue: UAI 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
review_content: Q1 Summary and contributions: This paper deals with the general problem of neural architecture design in machine learning (ML). The authors here introduce a general deep architecture for learning functions on multiple permutation-invariant sets, and demonstrate that it is a universal approximator of these functions. Experimental comparisons show superior results to existing methods on a variety of ML tasks including counting tasks, alignment tasks, distinguishability tasks and statistical distance measurements.

Q3 Main strengths: - Neural architecture design for a specific application is a difficult problem
- The paper is technically sound
- Experimentations are provided to support the model efficiency
Q4 Main weakness: - Experiment protocols are not identical for the 4 datasets
- Explanations of the type of algorithms (encoder, cnn, ...) used for experiments are missing
- Scalability of the proposal is not discussed. Only small size datasets are considered during experiments.


Q5 Detailed comments to the authors: The concept proof of the proposal relies mainly on the experiments conducted. However, there is no information provided on the hyper-parameters of the compared approaches. How those hyper-parameters are optimized is not discussed.

Accuracy can be misleading, especially when dealing with imbalanced data. How does this affect the experimentations ? Critical difference diagram can also be useful to measure the statistical significance of the comparison between several algorithms.

Minors :
- Table 3 : Our ModelS
- Several references are incomplete.
Q2(4) Quality of experiments (Optional): 2: Fair: The experimental evaluation is weak: important baselines are missing, or the results do not adequately support the main claims.
Q7 Justification for your score: The experimentations need improvement, wrt to the remarks above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate to high impact on more than one area of AI, with good to excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations. 
review_content: Q1 Summary and contributions: This paper presents a generalization of deep permutation invariant functions to multiple permutation invariant sets. They also extend their proposed method to sets of encode variable dimensions. The authors prove that their extended architecture is a universal approximator of continuous partially permutation equivariant function. The proposed method is then evaluated on multiple different tasks including alignment and statistical distance measurement. 
Q3 Main strengths: -- The paper explores an interesting problem of extending previous work to sets of sets.

-- The idea of capturing different type of interaction using a separate transformer and concatenating to get a contextual representation of a set is novel (sec 3.3). 

-- Showing that the resultant architecture is a universal approximator of partially permutation invariant function is simple and adequate. It would be better if in the main paper the authors could provide in the paper an outline of the proof and of where the is difficulty in proving 4.2. There is some white space that could be removed to get space for it, for example figure 1 can fit in single column. 

-- I like the experiments section since they not only evaluate the performance of multi set transformer but they also provide ablation study of the different component. It is great to see that across all the application their proposed method is performing better than baselines, although there is no one clear winner.

Q4 Main weakness: -- The theoretical analysis shows that their proposed method is a universal approximator, but what is not clear is that their proposed architecture is the only way in which we can achieved that result.

-- Even though the ablation study is great, I would have liked to see how the method performs agains a simple baseline: T(S={X U Y}): R^d x R^d -> R^d'. Where the transformer looks at the entire set of X and Y. My guess is that in terms of performance this model will be at par with the best performing model.
Q5 Detailed comments to the authors: -- If we try to extend this framework to 3 sets, 4 sets and beyond how many cross terms will be required in this given framework? Does this grow exponentially with the number of sets? Have you thought about such extension?

-- Topic models and other bag-of-bag of models have similar permutation invariances equivariances, have the authors tried experimenting with such models? 


Q2(4) Quality of experiments (Optional): 4: Excellent: The experimental evaluation is comprehensive and the results are compelling.
Q7 Justification for your score: Overall the paper is solving an interesting problem with a novel architecture. The theoretical and empirical results shows that the proposed architecture does lead to better learning of partially equivariant functions. The paper is well written and easy to follow.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate to high impact on more than one area of AI, with good to excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations. 
review_content: Q1 Summary and contributions: This paper extends previous work on single set transformer to handle inputs with multiple sets. Extensive experiments demonstrate the superiority of the proposed method.
Q3 Main strengths: 1. the method is theoretically sound and straightforward.
2. they prove the method is an universal approximator on multiset functions.
3. the experiments are thorough and cover many different tasks.
Q4 Main weakness: 1. the experiments are somewhat trivial. For example, the learned statistical distances are on Gaussian mixtures, which is questionable how good it can generalize to other distributions.
2. all experiments are supervised. It would be great to see some unsupervised applications.
Q5 Detailed comments to the authors: The paper is well-written.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q7 Justification for your score: A novel method with sufficient experiments and significantly better results.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept, e.g., novelty, outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: Q1 Summary and contributions: This paper proposes a general deep architecture for learning functions on multiple permutation-invariant sets. This architecture is further generalized to sets of elements of any dimension by dimension equivariance. Both theoretical analysis and experimental analysis are provided to show the effectiveness of the proposed architecture.
Q3 Main strengths: + A general deep architecture for learning functions on multiple permutation-invariant sets is proposed.
+ Theoretical analysis shows the generalization ability of the proposed architecture.
+ Experimental analysis also shows the effectiveness of the proposed architecture in some ways. 

Q4 Main weakness: - My major concern is on the insufficient comparison in Tables 2-3. Since Table 2 shows the results of multi-task image classification, many other representative multi-task learning methods should be included. Moreover, Table 3 shows the results of cross-modal learning in computer vision, and the latest multi-modal should be carefully discussed. At this moment, the reported results in Tables 2-3 are not that convincing. 
- One important reference is missing, i.e., “Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions, CVPR 2020”. Keeping this reference in mind, I wonder if the proposed architecture can be applied to few-shot learning. 
Q5 Detailed comments to the authors: See my detailed comments in Q4.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q7 Justification for your score: A general deep architecture for learning functions on multiple permutation-invariant sets is proposed. Both theoretical analysis and experimental analysis are provided to show the effectiveness of the proposed architecture. However, the comparisons in Tables 2-3 are insufficient.
review_title: null

Meta-review: Meta Review: The authors present a new deep learning architecture for learning functions on multiple permutation-invariant sets. They also extend the architecture to handle variable dimensions. The paper presented both theoretical results that showing the architecture is a universal approximator of the functions, but also empirical results demonstrating nice performance on a number of benchmark tasks.  Overall the proposed method is novel, and the results are promising. The current architecture is illustrated using 2 sets. The authors explained that modeling k sets needs to model k^2 pairs of sets. Is that a limitation of the architecture? What about applying a set-invariant transformer on top of the sets?
Decision: Accept (Poster)