Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper addresses the estimation of the relative density ratio using only a few instances from two datasets.  The authors propose a meta-learning method in which a neural network is trained to output the embeddings reflect the characteristics of two datasets.  In addition, they proposed to use the closed-solution of a linear model on the embedded space.  The experimental results show that the proposed method outperforms conventional methods on dataset comparison problems and outlier detection when a few instances are available.
main_review: Originality: 
The few-shot estimation of the relative density ratio is a novel problem setting and valuable for applications.
Although there are attempts to integrate deep neural networks into the estimation of the relative density ratio, the application of meta-learning is also novel for the DNN-based estimation of the relative density ratio.

Quality: 
The proposed approach is reasonable, and empirical experiments prove its effectiveness.
The paper also provides an ablation study to confirm the effectiveness of the specific design choices, such as the representation learning of two datasets.

Clarity: 
The paper is well-organized.
Although most of the paper is clearly written, integrating the closed-form solution of w into the training of DNN and the estimation of the relative density ratio is not clear enough to implement the proposed method.  I think it will be more clear if the details of NoSadapt and NoSadapt-FT are explained.


Significance: 
The few-shot estimation of the relative density ratio is an important problem for real-world applications.   The proposed meta-learning approach can be a baseline for future work.
Since the few-shot setting in the experiments is artificial, I would like to see a real-world application.
limitations_and_societal_impact: The limitations described in the Conclusion and the supplemental material are adequate.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: In this paper, a meta-learning method for relative density ratio estimation is proposed. The proposed method uses a permutation-invariant network to learn a network that outputs the relative density ratio of source and target data, and then uses the learned network to output the relative density ratio of a new source-target pair.
main_review: This paper is well motivated and well written so easy to follow.
I acknowledge the importance of relative density ratio estimation, its difficulty in a data-scarce environment, and the idea of meta-learning for addressing the problem. As far as I know, this is the first attempt to solve the relative density ratio estimation problem in the meta-learning framework. It is also highly regarded that the proposed method empirically outperforms conventional methods. However, it is unclear what is mechanism enables to transfer of knowledge.
As briefly mentioned in lines 144 to 146, it is contributed to the sharing of the network structure, but there is no theoretical or experimental evaluation of when/how it will work and when it will not. There are no guarantees in terms of model selection or learning theory.
Overall, the present paper lacks theoretical underpinning and the contribution is not enough for acceptance to this venue.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors are proposing to use meta-learning to estimation the density ratio which is a very important problem in machine learning. Density ratio estimation is a very challenging problem as it involves matching an infinite number of statistical moments, which can cause stability issues. The authors have constrained the problem so that they are addressing so that they have limited data from the source domain to estimate the density. The density-ratio is expressed using a permutation-invariant neural network.

The authors derive the loss function for the density ratio that is represented by a neural network. Then show that the parameters of the neural network can be estimated by splitting the data set into support instances and test instances. The authors have demonstrated its capability for relative density ratio estimation with applications to inlier-based outlier detection. Their results have shown to be superior under all matrix.
main_review: Overall, the paper is quite well written. The technical part is well explained and well structured. I do think there is some novelty in introducing meta-learning for relative density ratio estimation. My only comment is that there could be stronger experimental results. For example, the authors have compared their results to a number of approach. They have claimed that their approach is a convex optimization, so that there should be no variation in the AUC. However, other approaches may not be convex, it may be good to show some variance in the model after each run, with different initializations or parameters.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper shows how one can estimate density ratios when very few data points of the two sets of interested are available. In order to do that, the idea is to leverage other datasets that contain related tasks. The ratio is estimated via a linear model to a latent space that represents each dataset.
main_review: In general, the paper is well written and the problem is relevant. However, the notation is a bit convoluted and could be improved. For instance, $S_{nu}$ is used to denote target datasets not used for training (Section 4.1), however, the training algorithm itself also uses $S_{nu}$ to denote a subset of one of the training sets.  

Major comments:

- The results are encouraging overall, however, I miss comparisons to simpler approaches that would be more appropriate for small datasets than kernel-based and nnets methods. For instance, knn (see Kremer, Jan, et al. "Nearest neighbour density ratio estimation for large-scale applications in astronomy." Astronomy and Computing 12 (2015): 67-72.) and (parametric) probabilistic classification-based ratio methods would provide a more fair baseline for Section "5.2 Relative Density-ratio Estimation". 

- As expected, all methods in Table 2 improve their performance as $N_S$ grows. In particular, MMD starts to give better results for $N_S=4$ (even though it does not use the auxiliary datasets). I think it is important to understand what are the limitations of the method. If $N_S$ increases even further, would the other methods that don't use auxiliary datasets be more competitive?

- Regarding the ablation study (especially Table 4): I struggle to understand how your method without latent vectors ("No Latent") is able to learn something meaningful. If the ratio estimator has the shape $w^t h(x)$ (which is what I understand by "no latent"), how can such an estimate adapt to the new datasets? In particular, won't it give the same estimated ratio if you swipe the dataset to be used on the denominator with the one to be used on the numerator? Or are w and h retrained with the support instances of the true target datasets? If the answer is the latter, the text and Algorithm 1 are not clear, and a follow-up question would be whether the AUC results are based on data points not used in this last step.

- I also miss a discussion about the intuition behind the method. For instance, how should the auxiliary datasets relate to the target datasets in order for this procedure to work?

Minor comments:

- Why is NoSadapt-FT worse than NoSadapt? Shouldn't it improve NoSadapt because of the fine-tuning?

- The code has not been submitted (and they say it won't be available because it is proprietary), so it is not possible to check all details about the experiments, nor it is possible to reproduce the examples.

- Why is $N_S$ set to 10 on the toy example but to at most 5 in MNIST and ISOLET?

limitations_and_societal_impact: The limitations mentioned in "Main Review" were not addressed in the paper; however other limitations were addressed.
review_title: null

Meta-review: The reviews are mostly positive and the general sentiment is that the meta-learning approach (for estimating density ratios when there are few instances) is well-motivated and the results encouraging. There were some concerns of lack of theoretical underpinnings (Vnfc), the method limited in terms of applications (Q8XR), and experiments being artificial (r99y). The authors need to address the last two concerns in a revision, and include the additional experimental support in e.g. Supplementary Materials.
Decision: Accept (Poster)