Title: Preferential Normalizing Flows
Venue: NeurIPS 2024
Accepted: yes
Keywords: normalizing flow

elicitation

random utility models

prior distribution
Abstract: Eliciting a high-dimensional probability distribution from an expert via noisy judgments is notoriously challenging, yet useful for many applications, such as prior elicitation and reward modeling. We introduce a method for eliciting the expert's belief density as a normalizing flow based solely on preferential questions such as comparing or ranking alternatives. This allows eliciting in principle arbitrarily flexible densities, but flow estimation is susceptible to the challenge of collapsing or diverging probability mass that makes it difficult in practice. We tackle this problem by introducing a novel functional prior for the flow, motivated by a decision-theoretic argument, and show empirically that the belief density can be inferred as the function-space maximum a posteriori estimate. We demonstrate our method by eliciting multivariate belief densities of simulated experts, including the prior belief of a general-purpose large language model over a real-world dataset.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the problem of eliciting a complex multivariate probability density from an expert. Existing works mainly use simple distributions. This paper proposes to model the belief density with a flow based model. To apply normalizing flows to this problem, we will need to address a few challenges. (1) We want to train flows with a small set of samples, which may result in collapsing and diverging probability mass. (2) The samples presented for the expert might be drawn from an independent and known distribution.\n\nThis paper proposes to use Bayesian inference to address the challenges of collapsing and diverging probability mass. That is, we are interested in representing the belief density $p_{}(x)$ with a flow model. We then optimize the parameters of $p_{*}(x)$ with MAP estimation. The likelihood of samples, i.e., $p(D|f)$ can be computed with Proposition 3.5 of the paper. The prior is approximated by the probability of the k-wise winners i.e., $p(D_{k}^{\\succ})$. \n\nThe experiments on synthetic and real datasets show the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. This paper applies normalizing flows to the problem of expert knowledge elicitation. I think this is a promising application of flow-based models. The idea is novel. \n\n2. The proposed method is effective, and has theoretical guarantees.\n\nWEAKNESSES:\nSome definitions and formulas are not very clear to me.\n\n1. In Proposition 2.1, it is not very clear what is the relationship between W and the limit $\\lim_{\\beta \\rightarrow 0} p(\\mathcal{D}) = 1$.\n\n2. The symbol $\\mathcal{D}$ is used in two places, i.e., Proposition 2.1 and Section 4.1\n\n3. In Assumption 3, what is $Exp(s)$.\n\n4. In Proposition 3.5, is $s()$ a function or a constant?\n\n5. In Eq.4, Is the numerator $\\exp{(sf(x)\\lambda(x))}$?\n\n6. The symbol $\\mathbb{x}$ is used in several places. Sometimes, it represents a random variable, sometimes, it represents a sample.\n\n7. In Eq. 6, is $f_i = f(x_i)$？\n\nQUESTIONS:\n1. I am not an expert of expert knowledge elicitation, so I may misunderstand something. Based on Eq. 1, the $f$ is a continuous function. Based on Eq. 6, $p(f) \\propto \\prod \\exp{f_i}$. Therefore, why do we need normalizing flows to represent $f$. That is, it seems that we can use an arbitrary neural network to represent $f$.\n\n2. What’s the meaning of Theorem 3.6. I may have missed something, but I did not see any discussion or analysis on this theorem.\n\n3. The paper mentions that one challenge of this task is samples presented for the expert might be drawn from an independent and known distribution. How this challenge is addressed by the proposed method?\n\n4. The experiments actually not very convincing, to improve the experiments,\n    \n    (1) Are there any existing methods that can be used as baselines?\n    \n    (2) Do we have any quantitative metrics to evaluate the proposed method?\n\nLIMITATIONS:\nIn my opinion, the proposed method does not have potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to learn so-called belief densities based on k-wise rankings or comparisons of alternatives. This belief density is learned by combining function-space Bayesian inference and normalizing flows, which allows for the learning of complex (multivariate) probability densities. The authors evaluate their method empirically by studying synthetic scenarios, a regression task and a “real” use-case by querying an LLM and evaluating how close the resulting flow is to the ground truth density by three metrics.\n\nSTRENGTHS:\nThe paper addresses an interesting and very relevant problem, especially as human feedback has recently been used a lot in the training of LLMs. The paper has a strong theoretical contribution proposing a method how to learn the belief density using normalizing flows from rankings while preventing failure modes by using a functional prior.  The paper is well written and the authors clearly disentangles their contribution from the work in related works.\n\nWEAKNESSES:\nThe experiments could have been more varied. At the moment all experiments use k = 5 and it would be interesting to see how the results vary with different values of k. Specifically, k = 2 is a very relevant real-life use case as this type of feedback is given a lot in e.g. chatbots. Similarly, it would be good to get some idea of how the flows converge to the ground truth density as a function of the number of preferential samples.\n\nMinor points and typos:\nFig. 1 Can you make the numbers larger? They are hard to read even when zooming in and without zooming in it’s not clear that they are numbers\nl. 78 “Diverging” -> “diverging”\nl. 94 “Mollester” -> “Mosteller”\nl. 201 “to Bayesian inference” -> “to use/do, etc. Bayesian inference”\nl. 201 “given a preferential” -> “given preferential”\nl. 405 citation ends abruptly \nl. 458 “probality” -> “probability”\n\nQUESTIONS:\nIs assumption 3 satisfied in the LLM experiment? If not, how does this affect the results?\n\nLIMITATIONS:\nThe paper mentions fair limitations in the discussion section.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an approach for expressing an expert's belief density using a normalizing flow. Interestingly, the flow is trained solely on preferential questions (e.g., comparing and ranking) which follow a random utility model (RUM). The approach avoids several optimization issues that could occur when trying to model the more complex underlying data. The authors method relies on an approach that defines a functional prior over the preference data. The authors demonstrate their method on both simulated and real-world data, showing a very promising method for representing the belief density of expert opinions.\n\nSTRENGTHS:\nThe authors overcome a unique challenge in modeling expert's beliefs with flows, namely that the true data distribution of experts beliefs $p^*$ cannot be directly sampled from. Instead, they sample from preference data $D$ and choose a specific function-space prior for the flow in order to model the likelihoods of the k-wise comparisons. This unique solution results in a flow that properly places mass on the winner points following the underlying decision model of the data. The experimental results confirm their hypothesis, showing much better results as a result of their prior.\n\nAdditionally, The use of LLMs as experts in their experiments is clever. The approach is reasonable and replicable, while also demonstrating and more end-to-end application that would be of interest to the broader community.\n\nWEAKNESSES:\nOne primary advantage of normalizing flows is that they offer exact likelihood estimation. Because of the underlying data model for preference data (eq. 3), I suspect that the method here is actually optimizing a lower bound on the log likelihood. This may explain the optimization challenges and need for fixing the prior's precision $s=1$ described on lines 228-230. Specifically, their objective may be similar to the Max surjective flow described in Nielsen et. al. [2020, https://arxiv.org/abs/2007.02731]. In either case, the authors may want to clarify if their objective function is a lower bound or not, since adapting flows for non-bijective mappings is also a topic of interest to many.\n\nQUESTIONS:\nThe Appendix explores various settings of the parameter $s$ when the true value of s varies, which shows that their choice to fix $s=1$ is reasonable. However, the authors almost mention using using $\\lambda \\propto 1$ as well to maintain tractability. The implications of this choice are unclear, did the authors run any validity checks to test the implications of their choice?\n\nLIMITATIONS:\nIn practice collecting an training dataset of expert feedback is laborious, and often many experts are needed in order to collect enough data. The paper only considers the case of a single expert, with a flow model to that experts belief density. The authors may wish to comment on future work in this area — specifically, if their approach could be adapted to a multi-expert setting, if a single flow model is still sufficient or if a flow mixture model is preferred.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}