Elicitation Inference Optimization for Multi-Principal-Agent Alignment
Authors: Andrew Konya
Yeping Lina Qiu
Michael P Varga
Aviv Ovadya
Venue: Submitted to ICLR 2023 (2023)
Keywords: alignment
large language models
LLMs
NLP
transfer learning
human-centered AI
LLMs
preference modeling

Abstract: In multi-principal-agent alignment scenarios spanning governance, markets, diplomacy, and AI, it is infeasible to elicit every principal's view on all perspectives relevant to agent decisions. Elicitation inference optimization (EIO) aims to minimize the $n$ elicitations needed to approximate $N$ principal's views across $K$ perspectives. In this work, we demonstrate an EIO approach where data efficiency ($NK/n$) increases with scale. We introduce STUMP: an elicitation inference model which integrates an LLM with a latent factor model to enable learning transfer across samples, contexts, and languages.  Then, we characterize STUMP's performance on a set of elicitation primitives from which scalable elicitation (sampling) protocols can be constructed. Building from these results, we design and demonstrate two scalable elicitation protocols for STUMP where data efficiency grows boundlessly, scaling like $O(n)$ in the number of elicitations $n$. This makes it possible to obtain complex, high-dimensional preference signals spanning principal populations at any scale.

Under review as a conference paper at ICLR 2023
ELICITATION INFERENCE OPTIMIZATION FOR ALIGNMENT
Anonymous authors Paper under double-blind review
ABSTRACT
In multi-principal-aa principal-agent alignment scenarios including governance, markets, con- flict resolution, and AI decision maaiin,, is infeasible to elicit every principal's view on all perspectives relevant to an agent's decisions. Elicitation inference optimization (FIO) aims to minimize the n elicitations needed to approximate N principal's ss views across K' perspectives. In this work, we demonstrate an EIO ap- proach where data efficiency (N K/ /n) increases with scale. We introduce STUMP: elicitation inference model which integrates large language model with latent factor model to enable learning transfer across samples, contexts, and languages. We characterize STUMP's performance which scalable elicitation (sampling) protocols can be constructed. Building from these results, design and demonstrate two elicitation protocols for STUMP where, surprisingly, data efficiency scales like (()) in the number of elicitations n. In other words, the number of elicitations needed per principal remains constant even as the number of perspectives and principals grows. This makes it possible to approximate complex, digg--iimesional preference signals spanning principal populations at scale.
set of elicitation primitives from
INTRODUCTION
The principal-agent problem involves aligning agent decisions with principal interests. A challenge creating situations where agent choices are sufficiently influenced by signals containing principal preferences. With a single principal, high-complexity preference signals can be elicited directly via open-ended interaction. and powerful agents such as: governments & citizens Giger and Lefkofridi (2014); Gabriel (2020).
malti-princippl--agett scenarios can involve large populations of principals
firms customers Roberts and Grover (2012), peacekeepers & conflict parties United Nations (2012), existing Al systems & impacted populations Prabhakaran et al. (2022). and potentially even transformative AI & humanity Russell et al. (015); Christiano et al. (2017). As the number of principals grows, and the domain of agent decisions becomes open-ender directly eliciting the preference of all principals on all relevant perspectives becomes unfeasible [A.]]. As result, lower- complexity forms of elicitation like ballot voting (ie. for governments) and price signals (ie. for firms) are used to learn preferences. While clearly effective-a basis of democracy and the economy-tiie approaches drastically simplify real preferences. For example, they do not allow citizens (principals) to express what they would like government (agent) to do, or why, only if they support predefined options.
Elicitation inference optimization (EIO) aims to decrease the amount of direct elicitation needed to recover preference signal (enabling the use of more complex, higher-dimension preferences; e.g. in natural language). Consider an N x K matrix O where rows correspond to N principals, columns correspond to perspectives, and every element captures a cenecpal--prspeccti goal of elicitation inference optimization is to obtain a sufficient approximation of @ with minimal elicitation budget by directly sampling some elements and inferring the rest. Thus, EIO involves combining a) sparse elicitation (sampling) protocol with, b) elicitation inference model.
relationship. The
Closed-ended surveys simplify EIO by constraining the set of relevant perspectives to a pre- defined set typically, with K << N. Matrix sampling techniques elicit responses from each participant on a subset of perspectives selected randomly (hhemmker (1973). heuristically Raghu nathan and Grizzle (1995), or dynamically such that inference accuracy adaptively optimized

---

Under review as a conference paper at ICLR 2023
Gonzalez and Eltinge (2008): Zhang al. (2020). Inference models exploit the low-rank-ness, of +
via matrix factorization and other collaborative filtering techniques Zhang et al. (2020); Senguptal
et al. (021); Oliveira et al. (202I).
Collective response systems (CRS) enable high-complexity signals by allowing the set of relevant perspectives to be open-ended: neither limited in number nor requiring pre-definitio Ovadya (2022). In a CRS, participants contribute open-ended perspectives in the context of a question, prompt, or conversation and respond to subsets of perspectives contributed by others - typically, yielding K ~ O(N). Inspired by preference models Thurstone 1927)) and conjoint analysis Green and Srinivasan 1978). one class of approaches elicits randomized pair-choice votes Konya and Slodov (2015) and uses hierarchical probit-like models for inference Salganik and Levy (2013). Other approaches elicit agreement votes to learn absolute
relationships. Polis samples agreement votes semi-randomly, prioritizes votes that improve clustering, and uses mean imputation to support dimensionality reduction (Small al. (202T). Additional work bridges these approaches by randomly sampling both pair-choice and agreement votes; inferring missing votes by modeling both vote types with single utility matrix learned via regularized matrix completion Bilich et al. (2019).
Previous approaches to EIO for CRS have been limited in scope to single samples where the set of relevant perspectives all correspond to the same context (ic. the same question or prompt) Salganik and Levy (2013); Konya and Slodov (2013); (Smal et al. 2); Bilich et al. (2119). This means votes must be elicited from every person for every new context. As a result, to approximate some O spanning proportional to the number of elements in e (A.2) data efficiency does not increase with scale. However, previous approaches do not fully leverage all available data: learning from one context not transferred to support inference in other contexts, and the information contained in the perspective text is not used at all. Can an EIO approach that better leverages available data, enable increasing data efficiency with scale?
arbitrary number of contexts, the number of direct elicitations needed grows
In this work we introduce an approach EIO for CRS which becomes increasingly data efficient as the amount of data elicited grows. First, we introduce a novel elicitation inference model - STUMP- which better leverages available data by integrating pre-trained LLM with latent factor model. Second, we characterize STUMP's performance across a range of elicitation primitives from which arbitrarily scalable elicitation (vote sampling) protocols can be constructed. Finally, building on these results, design and demonstrate two scalable elicitation protocols where STUMP can infer with meaningful accuracy while data efficiency increases linearly with scale.
2 PROBLEM SETUP
Elicitation form. In a collective response system (CRS) participants respond to prompts with open-ended perspectives and vote on perspectives submitted by others. Let // be the set of N human participants, P be the set of K perspectives, and θ be the corresponding N x K human- perspective matrix. Both agreement and pair-choice votes may be elicited on perspectives. In agreement exercise eij. participant i is asked if they agree with perspective j. In pair choice exercise Ly participant is asked if they prefer perspective j or We denote set of exercise data E = (c eijk i Hij,kE E P}.
Elicitation inference involves training a model on available data and predicting all missing elements in target O. We choose binary agreement as the human-pent perspective relationship for our target O Inference performance probed experimentally by training on a subset of exercises including both vote types E,, and computing prediction accuracy on a validation set of only agreement votes Eg.
Data efficiency in EIO can be quantified the ratio of a) the number of votes contained in the target θ and b) the number of votes directly elicited via a sampling protocol which model needs to be trained on to meaningfully approximate O We denote the data efficiency to achieve inference accuracy ace as:
Baoe n(O) n(E,)
n(H) X xn(P) NK n(Ee)
2

---

Under review as a conference paper at ICLR 2023
Pj
Pre-trained LLM
F(p/lo)
Single Contoxt
Multipie Contert
e
Sample
Transformations to latent space
tonh(MMee by b²)
tanh(M e. bD)
Type! Typel
"joet
mnn
No Sample Overlap
Zi
Zi
Utility matrix
Figure 1: Diagram of STUMP. Universal model elements shown in grey. Perspectives and context- specific model elements colored by context.
Figure 2: Elicitation primitives. Color boxes indicate sparse vote data available for training. Colors denote context.
where n(X) denotes the size of set X and n, = (EE). We refer to the rate data efficiency changes with more elicited votes (0face/Oni as data leverage. EIO becomes more data efficient with scale data leverage is positive for arbitrarily large above a sufficient threshold of accuracy. To achieve this, the challenge is to combine: a) scalable elicitation (sampling) protocol where N K grows faster than ne, and b) inference model that sustains meaningful inference accuracy at scale using the data the protocol generates.
ELICITATION INFERENCE MODEL
Elicitation inference (EI) is analogous to collaborative filtering (CF) in recommenders, except instead of items people may prefer, there are perspectives people may agree with. A standard approach to CFis is to learn a low-rank approximation a user-item preference matrix by optimizing an objective function against observed preferences (Johnson) 2014). Bilich et al. (2019) demonstrated similar approach for on CRS data. They use regulanzed matrix completion with low-rank low-rank-promoting nuclear norm prior similar to Fazel et al. (2001) to learn utility matrix that predicts votes. We refer to this as the nuclear norm model (NN), and adopt it as a baseline (see A.3 for technical details).
Latent factor models (LFMs)) learn vectors for each user z, and item w, which embed them in the same latent space, and describe the preference of user for item j as a function of Mij = Z; W [Johnson] 2014). Known item features (i.c., director of movie) can be used improve performance by projecting features into latent space as a component of item embeddings (Zhang et al. 2021). Relevant features can also be extracted from item content (i.e., song's time sequence) by link to latent space embeddings (van den Oord et al. 2013).
We consider EI on text-based perspectives. Collaborative Topic Ranking (CTR) uses information in text by learning topic model on item texts and equating topic feature vectors to latent space embeddings (Xin al. 2015). Pre-trainable large language models (LLMs)) (Cer et al. 2018 Devlin
et al. 2018: Yang et al. (019) learn to encode information from text into embeddings suitable for
wide range of downstream tasks (Men) Hao et al.) 2020 Schick and Schütze 2021). Our approach
to using the information in perspective texts is to start with pre-trained LLM, then similar to
Zhang et al. (2021), learn a transformation text embeddings into latent space. A key challenge in
CF is maintaining alignment between latent spaces learned from overlaappg data (Cremonesi and Quadrana] 2014). Sharing the same embedding model and transformation across perspectives enforces alignment even in the absence of overlapping data (see section 4.3).
3

---

Under review as a conference paper at ICLR 2023
Cross-Domail recommendation (CDR) involves aligning latent spaces across items in different domains like movies and songs (Zang et al., (022). One class of approaches uses information like tags that can be shared across domains (Sahu al. 2018). Others focus aligning latent space distributional properties (Liu et al. 2021).
In EI the analog to CDR is multi-context inference. How votes perspective depends the context was elicited in. Take the perspective feel happy:" Consider two scenarios, where was elicited by asking "What would you say if you saw your parents smile?" "What wouldn you say if you saw your parents smile?". One would expect nearly opposite voting behavior. But, if the same embedding model and transformation are shared in both contexts, then predictions will be the same. Enabling multi-conter inference requires relaxing this condition. In the spirit of Sahu et al. (2018) our approach is to share the same I.M across contexts, but learn a unique transformation into latent space for each context.
Our model. We formally introduce our model, the Semantic Transfer Utility Model of Participants: STUMP. We denote the event that participant reports agreement with perspective j by daj. and disagreement by dij. We denote the event that participant reports preference for perspective over perspective & by Ciyk- We assume a logit utility model of choice similar to Johnson (2014); Bilich et al. (2019) where participant ? has a latent utility Wy corresponding to perspective j and voting probabilities are given by p(a,y (un) o Utg)), ), p(d,) (uy)) o M4y and p(cggs (uy)) a (Uij- - Mzze,, where denotes the logistic function. The utility Viy for participant i and perspective j is constructed
LFM:
Kij z; W
where z, is a latent embedding for participant i, and w, is a latent embedding for perspective j. To use information in perspective text, employ a pre-trained LLM generate text embeddings. We denote the embedding of the jth perspective text Pj by model F with parameters Φ, as:
e, =(p)oo))
Text embeddings are mapped into latent space by context transformations:
w tanh(M + b²)
where M and ba are the transformation weight and bias for context (see Figure I. Finally, let = {1.jaag = (i. jljjd ), and C = k(cikk The log loss defined by:
L -= Σ
log(o - ux)) + Σ log() + Σ log(11 - oug)))) + Am | KJ.kEC
AJEA
IJED
where ||-1|= denotes the Frobenius norm of matrix, and Am is a hyper-parameter controlling regularization strength. The participant embedding vectors z and the set of transformation parameters for each context (M", b") are learned by minimizing the log-loss for observed data. Optionally, the LLM parameters á may be fine-tuned through the same minimization.
4 ELICITATION PRIMITIVES
Our strategy for elicitation protocol design begins with characterizing STUMP's performance across a range of elicitation primitives from which arbitrarily scalable protocols can be constructed. To this end, we explore four types of elicitation primitives arising from two orthogonal data set properties: context homogeneity and sample overlap.
Context homogeneity. Perspectives and votes are elicited under some context for example, in response to a question. Thus, a data set (or θ) may correspond to one or more contexts. We denote the set of perspectives corresponding to the 9th context P C P.
Sample overlap. Vote data E may come from one or more samples, where each th sample involves subset of humans H. C H and subset of perspectives P, C P. Two samples are said to have no overlap when they share neither humans nor perspectives, ie H, n H2 = @ and P1 n Pg = 0. A sample with overlap may correspond to data set with only a single sample, with multiple samples where either humans or perspectives are shared, ie H, 0 H2 + 0 or Pi n P2 o 0.
Permuting these properties gives four types of elicitation primitives (Figure Table 1
4

---

Under review as a conference paper at ICLR 2023
Table 1: Properties elicitation primitives.
Type
Context
Sample
Definition
I
Single
Overlap
Ps P and H1 n H2 * or Pr #
II
Single
No overlap
p = P and H1 0 H2 P P3 =
III
Multiple
Overlap
ps 7 P and H1 H2 or P #
IV
Multiple
No overlap
Ps + P and Hi n H2 Pin
MC STM
NC ASYN
NN. SYN
NN ASYS
STUMP
STUNP
80
80
zhtw
en zh
accuracy (%)
70
accuracy (%)
en pt
en nl
en
60
60
en_ir
en es
STUMP
en en
50
50
CS-MI
en de
10
30
40
0.5
50
70
80
votes per participant
sample overlap
accuracy (%)
Figure 3: Type I inference accuracy vs votes per par- ticipant.
Figure 4: Type I/II infer- ence accuracy vs sample overlap.
Figure 5: Type II inference
curacy by language pair.
4.1 ELICITATION PRIMITIVE EXPERIMENTS
Baselines. We use four prediction baselines contextualize STUMP's performance: a) a Random baseline where votes are predicted randomly with uniform probability, b) Most Common (MC) baseline where votes are predicted to the most common choice observed each perspective, c) the nuclear norm (NN) model from Bilich al (Bilich et al.) 2019) A.3 which represents standard approach to CF and is the only prior work we are aware of focused on single-vote-lew inference for
CRS, and d) the classic matrix factorization (CF-MF) model widely used in recommendation systems (Koren et al. 2009).
Data and implementation. Data used for experiments collected as part of regular load testing for an existing CRS product and de-identified prior to use in this study. Data came from 534 participants solicited through online research panels. We data corresponding seven questions [A.4] asked during the CRS process, chosen for their nuance and presumed mard-to-predict nature. Fach question treated unique context, meaning only perspectives and votes elicited in response to the same question share the same context transformation weights. The full dataset used consists of 172,242 votes across 3,703 perspectives made by 534 humans. Experiments are constructed by partitioning exercise data E into training and validation sets (E, and E,). In each of the primitives described below, the inference model is trained with E, and evaluated with E., Data and training details are described in A.5
4.2 TYPE SINGLE CONTEXT, OVERLAPPING SAMPLE
Data elicited from one question during a collective response (CR) represents a type primitive. We test type I model performance function of vote per participant (VPP) for two modes of CR; asynchronous and synchronous. an asynchronous CR, participation happens on a rolling basis, so participants only vote perspectives submitted prior their participation. In synchronous CR, all
5

---

Under review as a conference paper at ICLR 2023
75
70
accuracy (%)
70
accuracy (%)
accuracy (%)
70
60
60
65
50
50
60
0 0.2 0.4 0.6 0.8
2 3
4
6
0
0.2 0.4 0.6 0.8
participant fraction
questions in pre-train
sample overlap
Figure 6: Type III inference accuracy vs participant frac- tion r.
Figure 7: Type III inference accuracy questions in pre- training data.
Figure 8: Type IIIIVV in- ference accuracy vs sample overlap.
participation happens in the same minute-scale window of time, so any perspective may be voted on by any participant.
Type I as a baseline. We simulate both cases while varying VPP by partitioning the appropriate votes into training and validation sets. To simulate the asynchronous CRS scenario, participants votes pertaining to perspectives submitted after their own become the validation set E., and votes pertaining to perspectives submitted before their own become the training pool. We modulate vote density by adjusting the fraction votes from the training pool which is randomly partitioned into the training Fg. To simulate the synchronous CRS scenario, we randomly partition votes such that the training set Fg and validation set E.g. are the same size as those in the asynchronous scenarios for each experiment. For each experiment, we train the models for each question and compute the corresponding validation accuracy.
STUMP outperforms all baselines < 30 VPP and achieves level of accuracy only 10 VPP which the NN baseline only achieves with 3x the amount of data (Figure As the simplest elicitation primitive. Type performance across simple baselines informs what constitutes meaningful accuracy in general. The Random baseline achieves 50% accuracy (not shown) and the Most Common baseline saturates at an average of ~ 64% accuracy with high degree of variance. A reasonable conclusion is that 50- 64% accuracy is not meaningful, 65 69% is arguably meaningful, and 70% more is meaningful.
4.3 TYPE H: SINGLE CONTEXT, NON-OVERLAPPING SAMPLE
Type to Type II transition. We simulate EI within single-question context as the overlap between two samples decreases to zero. For each question, we randomly split the participants and perspectives into two equal sized samples: {H,1, P.)) and (H... P=2]. Agreement votes by participants in Ha perspectives in P.2 become the validation set E, (eiy, Ejj, E Hat,j, k E P.2). The remaining votes contained in the question become the Type training set E,, where sample overlap is maximum, E, {ei,. Gi
H. E Ps1] {eey E | He2.j, P.2). Sample overlap decreased by removing votes on perspectives in P.i by participants in Hl. until none remain, giving Type II training set, E. = {E Eija i j, U (e)) e | i H22. k E P.2} (more experimental details are described [.66I D. We define sample overlap p as the fraction of participants in Ha² with training votes in P.1. and compute validation accuracy for STUMP, NN and CF-MF baseline as p varies (Figure 4. is shown that NN and CF-MF baselines are sensitive to sample overlap. In contrast, STUMP's performance remains stable even at O; evidence of STUMP's ability to maintain latent-spuce alignment across non-overlapping samples.
Type II with Different languages. We investigate STUMP's inference capability across two non- overlapping samples in different languages. Perspectives one sample are translated into one of nine different languages using Google Translate API: German (de), Spanish (es), French (fr), Italian (it), Japanese (ja), Dutch (nl), Portuguese (pt). simplified Chinese (zh) and traditional Chinese (zhtw) [.. We compute STUMP's Type II accuracy for each language pair and compare it to an English English baseline (Figure 5). Performance across all language pairs is on par with the baseline.

---

Under review as a conference paper at ICLR 2023
Figure 9: Case sampling protocol, where shaded regions correspond sampled votes and color corresponds to context.
Figure 10: Case 2 sampling protocol, where shaded regions correspond to sampled votes and color corresponds to context.
4.4 TYPE III: MULTIPLE CONTEXT, OVERLAPPING SAMPLE
Transfer learning. To test STUMP's capability for transfer learning, we pre-train participant embeddings in one context and use them for inference in a new context. Given pair of contexts Q1 and Q2, we denote all perspectives in Q1 as pl and all perspectives in Q2 as p2. We split the participants who participated pre-training set contains votes by all participants on perspectives in context Q1, E1 {ee- Sik i H, j, k E Pl], and is used pre-train participant embeddings. The second training set contains votes for a fraction r of participants on perspectives in a new context Q2. E2 = (c Git E H,2,j, k E p2], and r To ((HH)))(((L)))) ((((H)))nn((H))) The pre-trained embeddings are frozen while the transformation to latent space for the new context is learned. Votes on perspectives in Q2 by participants not in the second training comprise the type IIII validation set E. fel. Eis
both contexts randomly into two groups H₂1 and H22. The
H, & E P2] (more details are deseribed in (A.6.2) ) We vary the participant fraction from 0.05 to 0.9, and compute validation accuracy for STUMP and the CF-MF and NN baselines (Figure 6 STUMP performance increases with participant fraction when
small, leveling off at r 0.2, while the NN baseline only reaches similar performance around r 0.8. The CF-MF baseline saturates at lower accuracy. other words, STUMP can leverage data from different context via transfer learning to achieve the same performance the NN baseline with 75% fewer data.
Increased pre-training data. Next, we investigate how more pre-training data from more contexts affects STUMP's transfer learning performance. We increase the number of question contexts whose votes are included in the pre-training set [A.6.2] and compute type III validation accuracy at different participant fractions (Figure 7 STUMP's transfer learning performance increases as the number of question contexts included in pre-training increases.
4.5 TYPE IV: MULTIPLE CONTEXT, NON-OVERIAPPINN SAMPLE
Type III to Type IV transition. Finally, we explore transfer learning across contexts sample overlap decreases. Let sample overlap be the fraction of participants in the training sample (context Q2) who are also in the pre-training sample (context Q1). Moving from Type IIII to Type IV means decreasing p from to O. To simulate this, we start with Type III setup and iteratively remove participants from the pre-training sample who are also in the training sample (more experimental details are described [A.63)). ) We use the training procedure and validation set from (4.4) and compute accuracy as p varies (Figure 8). STUMP performance arguably meaningful when sample
overlap is larger than 50% but drops below that.
5 INCREASING DATA EFFICIENCY WITH SCALE
Based on STUMP's performance characteristics across elicitation primitives, we design two arbitrarily scalable elicitation protocols where: a) one may expect STUMP to sustain meaningful inference accuracy, and b) data leverage is positive, ie. ag
Sn. 0.

---

Under review as a conference paper at ICLR 2023
80
data efficiency Pro
30
25
70
20
accuracy
15
60
10
Case
Case II
5
Case II
50
2
3
3
samples
n// 10
Figure 11: Inference accuracy as function of the number of local samples.
Figure 12: Data leverage as function of the number of votes in the training set.
Case I. STUMP shows meaningful type III inference for low participant fractions (Figure 6). This means if participants all vote on perspectives from shared context, only small fraction need to vote on perspectives from new context to meaningfully infer the rest. Thus, the following elicitation protocol is viable: Elicit an initial sample where participants each give 1 new perspective and vote on e other perspectives for a single context. Let each sample after the first elicited from m new participants where each a) provides I new perspective and votes on other perspectives for new context, and b) votes on perspectives for the context shared by all samples (Figure 9 This protocol results in data efficiency of B n+/(4c) m/(2c) m/(4n,) O(n,). and data leverage
ag One.
1/(4c²) 0. A.7.1
Case H. STUMP shows arguably meaningful type III/V inference once sample overlap approaches ~ 0.5 (Figure 8). In other words, inference for population of participants across two contexts requires around half of them vote on perspectives from both contexts. This implies such overlap supports latent space alignment between contexts. To maintain latent space alignment across an arbitrary number of contexts, protocol: Elicit an initial sample where m participants each provide new perspective and vote other perspectives for single context. Let each sample after the first be elicited from m participants where m/2 participants are new, and m/2 are from the previous sample. Let each participant provide new perspective and vote on other perspectives for new context (Figure TT). This protocol results in data efficiency of 3 = n,((2c)) + m/ ~ (ne)) and Zin 1/ 0. A.7.2
chain successive overlaps together with the following elicitation
We simulate EI under both protocols for = 2,3, samples by partitioning the appropriate perspectives, participants, and votes into training and validation sets A.8) In both cases STUMP sustains meaningful accuracy of 70% as g increases (Figure 11.. approximating O(n?) votes using only (nz). Thus, the combination of either clicitation protocol with STUMP achieves 870 (mz) (Figure 12 giving positive data leverage: 807o On.
const 0. In other words, data
efficiency increases linearly with scale.
DISCUSSION
Improving inference accuracy. This work prioritized data efficiency with scale over maximizing inference accuracy. We do not expect STUMP achieve SOTA accuracy in its current form. Neural collaborative filtering (NCF), which replaces the inner product between latent space embeddings with neural architecture able to learn a more general function, has been shown to improve performance in recommenders He et al. (2017). and is likely to improve accuracy in this setting as well (preliminary experiments have confirmed this). Experiments pre-training data can improve accuracy. STUMP's use of information in the perspective text is limited by the LLM it employs and we expect SOTA LLM Chowdhery et al. (2022): Zhang et al. (2022); Mitchell et al. (2022) can improve accuracy. Other directions to improve accuracy may include: using participant side information like demographics, extending STUMP to use information in prompts, using active sampling to elicit votes expected to give the most information Gonzalez and Eltinge (2008): Zhang et al, (2020), and fine-tuning LLM weights while training STUMP. These will be our future work directions.
(Section 4.4) indicate more, and more diverse,
8

---

Under review as a conference paper at ICLR 2023
Limitations. STUMP fails to align latent spaces across non-overlapping samples from different contexts. This may require an embedding distribution alignment module and Stien Path training Liu et al. (20II. STUMP makes assumption of low-rank structure and and does not model non- linearittt Our work does not quantify confidence in aggregate vote statistics, though Bilich et al. (2019) shows stochastic weight averaging Maddox waeddoingMMaddox et al. (2019) estimate the posterior variance in this setting. We only use text perspectives. Handling perspectives spanning text, images, audio, and video requires swapping STUMP's LLM for a multi-modal foundation model able to generate aligned embeddings across media types Jabbar et al. (2022); Radford et al. ); Bommasani et al. (2021) y [Song et al. (2022); Reed et al. (2022). Our experiments were limited by data which was available, and is possible that some kinds of contexts, perspectives, or participants are materially different in ways that make our results generalize less than expected. STUMP can only represent changing participant views by treating different times as different contexts, while recommenders often model it explicitly [Chen] ict al. 2021a). This work does not show how to make results understandable for human agents, though clustering latent space is promising direction [mall et al. (202I). We don't show how learned preference signals can directly control other ML models, but jury learning for classifiers Gordon et al. (2022) and reward learning for RL agents Ziegler et al. (019))
encouraging directions. STUMP is not differentially private Dwork (2008) but approaches developed for LFMs in recommenders may be applicable Berlioz al. (2015). Finally, elicitation inference is just one of the components helpful for achieving the underlying goals of multi multi- principal- agget alignment.
Risks. In some biases common to recommenders Chen al. (2020) are less significant due to the structure of the task. In particular, if vote exercises are random and can't 't be skipped, while users are anonymous and can't 't see results, then biases spanning selection, exposure, conformity, position, and popularity are minimized. However, participants who respond and vote more slowly may still be under-represented in the data, bad elicitation prompts can still bias participants [[ttwak] (1956), and malicious actors may still attempt to influence results through coordinated participation. Biases in pre-trained LLMs Schramowski et al. (2022); Nadeem et al. (20TT); [Lu et al. (2022) may also propagate through the model to effect inference in unknown ways. We avoid active sampling Zhang et al. (2020) due to the risk of compounding small model bias into large inference bias. Further, non-representative participant populations can arise from biased solicitation, or unequal willingness or ability to participate. As such systems become used for important decision-making. they risk neglecting views of already marginalized communities due limited technology access and capacity Roessler (2018) unless specifically mitigated for, e.g. using existing inclusive voting infrastructure, structurally increasing digital equity, and/or sortition with high touch support. Even if the data collected is unbiased and representative. there are still risks that unexpected inference errors can corrupt results, or that even high-quality results may be misinterpreted or misused due to miscalibrated confidence, missing context, or malice. Elicitation inference more generally enables a different set of tradeoffs than traditional approaches to vote collection and aggregation. Its efficiency gains can enable increased access to agenda-setting power compared to traditional vote-with the tradcoff that this involves accepting approximation and error bars in vote outcomes.
Conclusion. We have demonstrated EIO for collective response systems where data efficiency increases linearly with the size of the elicited data set. As a result, the number of elicitations needed per participant remains constant even as the number of perspectives and participants grows. Our experiments suggest this scaling relationship is likely to hold for data sets spanning contexts, languages, and population samples. If true, that would make it feasible to approximate high- complexity preference signals for massive principal populations spanning open-ended perspectives across an agent's decision domain. Such signals may be used to steer complex objective and evaluation functions for mulli-princippl-agee and AI system alignment. For example, to support collective response for peacebuilding Alavi et al.] (2022) and constructive democratic governance of global platforms Ovadya (in combination with other processes and tools). More speculatively, EIO may help ensure that the quality and complexity of human input driving the actions of powerful automated systems increases with the capabilities of those systems A.9
scenarios spanning governance, markets, diplomacy,
REFERENCES
Nathalic Giger and Zoe Lefkofridi. Salience-based congruence between parties & their voters: The swiss case. Swiss polifical science review, 20(2):287-304, 2014.

---

Under review as a conference paper at ICLR 2023
lason Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411 -437, 2020.
Nicholas Roberts and Varun Grover. Investigating firm's 's customer agility and firm performance: The importance of aligning sense and respond capabilities. Journal of Business Research, 65(5): 579-585, 2012.
United Nations. Strengthening the role of mediation in the peaceful settlement of disputes, conflict prevention and resolution. sixty session, Agenda item 34 (a), Prevention of armed conflict, A/66/811, 2012. URLptttp:///peceeaaee:: wttp://wwww.aa..eeooeeeee/eeaaaa Salls//GReport..o
.n. org/
Vinodkumar Prabhakaran, Margaret Mitchell, Timnit Gebru, and lason Gabriel. A human rights-based http://awwvv.agaaaaaaa/ooo abs/ 2210.. 02667
approach responsible ai, 2022. URL
Stuart Russell, Daniel Dewey, and Max Tegmark. Research priorities for robust and beneficial artificial intelligence. Ai Magazine, 36(4):105- 2015.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep rein- forcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. httLhhtts:///rooooe Pwper.
meurips
David M Shoemaker. Principles and procedures of multiple matrix sampling. Ballinger, 1973.
Trivellore E Raghunathan and James E Grizzle. A split questionnaire survey design. Journal of the American Statistical Association, 90(429):55 1995.
Jeffrey M Gonzalez and John L Eltinge. Adaptive matrix sampling for the consumer expenditure quarterly interview survey. In Proceedings the Section on Survey Research Methods, American Statistical Association, pages 2081-20888 2008
Chelsea Zhang, Sean J Taylor, Curtiss Cobb, and Jasjeet Sekhon. Active matrix factorization for surveys. The Annals of Applied Statisticx, 14(3):1182-122) 2020.
Nandana Sengupta, Nati Srebro, and James Evans. Simple surveys: Response retrieval inspired by recommendation systems. Social Science Computer Review, 39(1): 109-)9105-129, 2021.
Markos FBG Oliveira, Myriam Delgado, and Ricardo Lüders. Comparative analysis of collaborative filtering-basee -baend predictors of scores in surveys of a large company. In Anais do XVIII Encontro Nacional de Inteligência Artificial e Computacionall pages 739-750. SBC, 2021.
Aviv Ovadya. Collective response systems, 2022.
Louis L Thurstone. The method of paired comparisons for social values. The Journal of Abnormal and Social Psychology. 21(4))384, 1927.
Paul E Green and Venkatachary Srinivasan. Conjoint analysis in consumer research: issues and outlook. Journal of consumer research, 5(2):103-23,, 1978.
Andrew Konya and Aaron Slodov. Collective speech. In Collective Intelligence, 2015.
Matthew J Salganík and Karen EC Levy. Wiki surveys: Open and quantifiable social data collection. PloS one, 1013))38 2015.
Christopher Small, Michael Bjorkegren, Timo Erkkilä, Lynette Shaw, and Colin Megill. Polis:: Scaling deliberation by mapping high dimensional opinion spaces. Recerca. Revista de Pensament i Andlisi, 26(2):1-26, 2021.
Jordan Bilich, Michael Varga, Daanish Masood, and Andrew Konya. Faster peace via inclusivity: An efficient paradigm to understand populations in conflict zones. 2019.
Christopher C. Johnson. Logistic matrix factorization for implicit feedback data. 2014.
10

---

Under review as a conference paper at ICLR 2023
M. Fazel, H. Hindi, and S. P. Boyd. A rank minimization heuristic with application to minimum order system approximation. Proceedings of the 2001 American Control Conference, volume 6, pages 4734-4739. IEEE, 2001.
Haiyang Zhang, Ivan Ganchev, Nikola S. Nikolov, Zhanlin Ji, and Máirtín O'Droma. Featuremf: An item feature enriched matrix factorization model for item recommendation. IEEE Access, 65266-65276, 2021. doi: 10.1109/ACCESS 202.33333
Aaron van den Oord, Sander Dieleman, and Benjamin Schrauwen. Deep content-base based music recommendation. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Wein- berger, editors, Advances in Neural Information Processing Systems, volume 26. Curran As- sociates, Inc., 2013. URL htt.tpppp//pooceeeinnngo
.df
Xin Xin, Zhirun Liu, Chin- Yew Lin, Heyan Huang, Xiaochi Wei, and Ping Guo. Cross-domain collaborative filtering with review text. 07 2015.
Daniel Cer, Yinfei Yang. Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedee Steve Yuan, Chris Tar, et al. Universal sentence encoder. arXiv preprint arXiv: 175, 2018.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernández Ábrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Multilingual universal sentence encoder for semantic retrieval. CoRR, abs/1907.04307 2019. URL http:
//arxiv. o/arxiv..rc/abs/11001.11
Boran Hao, Henghui Zhu, and Ioannis Paschalidis. Enhancing clinical BERT embedding using biomedical knowledge base. In Proceedings of the 28th International Conference on Com- putational Linguistics, pages 657-661, Barcelona, Spain (Online), December 2020. Interna- tional Committee on Computational Linguistics. doi: 10. (tt6://////200.cooogggoooo httpps//.aaaantoooooo..or/00....
URL
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In EACL. 2021.
Paolo Cremonesi and Massimo Quadrana. Cross-domain recommendations without overlapping data: Myth or reality? Proceedings of the 8th ACM Conference on Recommender systems, pages 297-300, 2014.
Tianzi Zang, Yanmin Zhu, Haobing Liu, Ruohan Zhang, and Jiadi Yu. A survey on cross-domain recommendation: Taxonomies, methods, and future directions. ACM Transactions on Information Systems, 07 2022. doi: 10. 10.1 155/3348448
Ashish K Sahu, Pragya Dwivedi, and Vibhor Kant. Tags and item features as a bridge for cross-domain recommender systems. Procedia Computer Science, 125:624-631, 2018.
Weiming Liu, Jiajie Su. Chaochao Chen, and Xiaolin Zheng. Leveraging distribution alignment via stein path for cross-domain cold-start recommendation. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 19223-19234, Curran Asso- ciates, Inc., 2021. URL httttt/ss:/ppppceeeessttt werpiiinnnnnn.uuupeepereme neurips /ctpp//pe///22///lllee 30448c866c332225623777711
Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-37, 2009.
11

---

Under review as a conference paper at ICLR 2023
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th International Conference on World Wide Web, WWW '17. page 173-182, Republic and Canton of Geneva, CHE, 2017. International World Wide Web Conferences Steering Committee. ISBN 978 07884450349199 30. 0et.e1115333388 3052569
doi: 10.1 10.1145/3339112330055
URL httphhttss///dii.rrgggg.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha T'vyashchenko, Joshua Maynez, Abhishek B Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garefa, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Oliveira Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Mcier- Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. ArXiv, abs/2204.02311 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer. Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. URL /ttpp::/aaxxv..orrrrt/ 2205 01068
/ttp/viivvvv.////v/ www.wwwwww...v...oooaoooooaaaaaa abs/
Margaret Mitchell, Giada Pistilli, Yacine Jernite, Ezinwanne Ozoani, Marissa Gerehick, Nazneen Ra- jani, Sasha Luccioni, Irene Solaiman, Maraim Masoud, Somaieh Nikpoor, Carlos Muñoz Ferrandis, Stas Bekman, Christopher Akiki, Danish Contractor, David Lansky, Angelina McMillan-Major Tristan Thrush, Suzana Iliè, Gérard Dupont, Shayne Longpre, Manan Dey, Stella Biderman, Douwe Kiela, Emi Baylor, Teven Le Scao, Aaron Gokaslan, Julien Launay, and Niklas Muennighoff. Bloom. 2022.
http://w.........eo..eeeeeee/eee wtt.:ee/oogiicceeeeee www.iiienneeeoooonoooooo bloom
Wesley J. Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson. A Simple Baseline for Bayesian Uncertainty in Deep Learning. Curran Associates Inc., Red Hook, NY, USA, 2019.
Muhammad Shahid Jabbar, Jitae Shin, and Jun-Dong Cho. Ai ekphrasis: Multi-modal learning with foundation models for fineinrained poetry retrieval. Electronics, 11(8), 2022. ISSN 2079-9292. doi: 10.3390/electronicc/ 1081275 URL htttstt::: /www www.w/ww..cc.////.....//.. /ttpa/c////000022222/1/111000010
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https
//axxiv.org/ab//222/
00020
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson,, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespic, Karan Goel, Noah Goodman, Shelby Grossman. Neel Guha Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa,
12

---

Under review as a conference paper at ICLR 2023
Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, 2021. (tRl:httts://// //rx: wwg. wwtgg/ab///lii http:pp:/aa....orgrg/bbb////////
07258
Ruihua Song, Xin Gao, Tao Xiang, Hao Sun, and Ji-Rong Wen. Towards artificial general intel- ligence via multimodal foundation model. Nature Communications, 13(13), 2022. ISSN 2041-1723. doi: 541467-022-30761-2
URL
10338/
Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards. Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. generalist agent, 2022. (tthss:
7/ a //arxiv. wttp:vvorr//aaao abs 2205 06175
Yi-Cheng Chen, Lin Hui, and Tipajin Thaipisutikul. A collaborative filtering recommendation system with dynamic time decay. The Journal of Supercomputing, 77, January 2021a. doi:
1221000011277--200000223
Mitchell I.. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeffrey T. Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. Jury learning: Integrating dissenting voices into machine learning models. CoRR, abs/2202.02950, 2022. URL https::/aaxii..o/////////
02950
Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv, abs/1900.08599. 2019.
Cynthia Dwork.
Differential privacy: A survey of results. In International conference on theory and applications of models of computation, pages 1-19. Springer, 2008.
Arnaud Berlioz, Arik Friedman, Mohamed Ali Kaafar, Roksana Boreli, and Shlomo Berkovsky. Applying differential privacy to matrix factorization. In Proceedings of the 9th ACM Conference on Recommender Systems, pages 107-114, 14, 2015.
Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng. Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. ArXiv, abs/2010.03240, 2020.
Eugene Litwak. classification biased questions. American Jouriic Journal of Sociology, 62(2):182-186 1956. doi: 10.1086/221960. /
Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin Rothkopf, and Kristian Kersting. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nature Machine Intelligence, 4:258-263 03 2022. doi: 10. 10.1038/s42256- 0210044222260022-000000 022-00458-8.
Moin Nadcem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pre- trained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference Natural Lan- guage Processing (Volume 1: Long Papers), pages 5356-5371, Online, August 2021. Asso- URL https
ciation for Computational Linguistics. doi: 10. 1e..1866//vl 1/202/1122111 a
//aclanthology: org ww1.:2211111c1oooo 416 416.
Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, and Soroush Vosoughi. Quantifying and alleviat- ing political bias in language models. Artificial Intelligence, 304: 103650, 2022. ISSN 0004-3702. doi: https://doi.org/100/ 10tp:///1..111/.rrriiiiio com/ http://ccaeeeerreeeeeee//////2op
103654. URL https: www. sci wwtp://wwwww..ww..w.ctwc....c.ce rect,
Philip Roessler. The mobile phone revolution and digital inequality: Scope, determinants and
consequences. Prosperity Commission Background Paper Series, (15), 2018.
13

---

Under review as a conference paper at ICLR 2023
Daanish Masood Alavi, Martin Wählisch, Colin Irwin, and Andrew Konya. Using artificial intelli- gence for peacebuilding. Journal of Peacebuilding & Development, 17(2):239 243, 2022. doi: 10. 0177/15423116221107777 /
Aviv Ovadya. Towards platform democracy: Policymaking beyond corporate CEOs and partisan pressure.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 15084 15097. Cur- ran Associates, Inc., 2021b. https:stss//p/ www.:iin......uuueueeeeeee cc/ cc/paper/2221// Paper...pppppppppp.....
Qinging Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, edi- tors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 27042-27059. PMIR, 17-23 Jul 2022. URL. https: http::://proceedinngg
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 1273- 1286. Curran Associates, Inc., 2021. URL http:tttss://// /ttpo/////uttttt///r//eeooooooe newriiiss neurips ccppsscccpppe cctpppprrcccppprr / FD0/F122005560044222334445333320
A APPENDIX
A.1 ELICITATION FEASIBILITY
Assume a scenario where there are 10,000 principals and agent with an open-ended decision space such that each principal may provide a unique perspective related to decisions the agent could make in given month. Consider 10,000 perspectives, and assume that directly eliciting principal's is view on a perspective involves casting a single vote which takes 10 seconds each. Direct elicitation of all votes equates to 28 hours spent voting per principal per month 2280,000 hours per month in total), This is unreasonable but theoretically possible. However, increasing this to just 100,000 principals equates 280 hours per principal per month more than a full-time job and increasing it to 1, 1,000,000 principals equates to 2,800 hours per principal per month which is more than the number of hours in month. There more than 300,000,000 principals (US citizens) in the US govermment agent.
A.2 SCALING BEHAVIOR OF EIO FOR CRS BASED ON PREVIOUS APPROACHES
Consider numaa-verspective matrix @ corresponding to N humans and K perspectives spanning Q different contexts (ie. elicited by Q different questions or prompts). Assume the corresponding CRS generates a finite number of k perspectives for each context K Qk. The number of elements in is NQK (NN)). Bilich et. al. demonstrate an EIO approach to CRS in (Bilich et al. 2019) where only finite number of votes per person, is needed to recover an approximation of O for a single context. This means the number of votes which need to be sampled for a θ spanning Q contexts is NQc ~ O(NQ).
A.3 NUCLEAR NORM MODEL DETAILS
The nuclear norm model likelihood is given by:
p( (A,,
II ( II
(1 (wi)))
II k..
amii m,k)
JJEA
IEED
14

---

Under review as a conference paper at ICLR 2023
where A {i.jlan), {i.jldiz), ), and C klcykk) The NN model posterior is given by
p(u A, C. = z D,(A,,,C1 H|al|esr
7 Bexx
MA,
where 1 Julle<T denotes uniform prior over the nuclear norm ball for u of radius Leaning happens through optimization via SGD of the MAP estimator.
A.4 DATA SOURCE
Data used for this research comes from periodic load testing done by private company for internal quality assurance, and was fully de-identified prior to use in the research. The researchers identified that portions of could be interesting for this work due to the scale the data and the predictive challenges associated with the nuance and ambiguity of certain questions that were asked. Data made available for this research came from the following open-ended questions asked during the collective response load test:
During the pandemic, what fun experiences have you newly discovered? These can be any activities, hobbies, or events you consider fun.
Please tell me more: what more important you now?
And conversely, what is less important you now?
What new routines or behaviors have you started, if any, based on what you now think is important?
How or where did you learn the new routine or behavior?
Tell me about a product or service you have used MORE than ever in the past year. Why did you use more?
Please tell me more how is this impacting your decisions, at all?
A.5 DATA AND TRAINING DETAILS
For each question in the dataset, there are an average of 529 perspectives, 14455 agreement exercises, and 10151 pair-choice exercises. The full dataset consists of 172,242 votes across 3,703 perspectives made by 534 humans.
In training the STUMP model, we use L2 regularization on the transformation weight and apply dropout on the universal embedding. We use a pre-trained multilingual universal sentence encoder (Apache-2.0 license) to generate perspective embeddings F(p:(Ø). Hyper-parameter are tuned by grid search, and the following values are used in our experiments: regularization rate 0.01, dropout rate = 0.3, latent participant embedding dimensions 50, and learning rate 0.001 with Adam optimizer. The models are implemented with PyTorch, and trained PC with 4-core Intel 17-1165G7 processor.
A.6 ELICITATION PRIMITIVE EXPERIMENT DETAILS
The exercises in the training sets consist mix of agreement and pair-choice exercises where agreement exercises account for 60%-75% of the total exercises.
A.6.1 TYPE II: SINGLE CONTEXT, NON-OVERLAPPING SAMPLE
Type I to Type II transition. For each question, we randomly split the participants into two equal-sized groups Ha and H.2. and their corresponding perspectives P.1 and Pa2. We select votes on perspectives in P.2 by participants in H.1 the validation set, E, ef Her.j,k ke P.2). The train set consists of the following three pieces: votes perspectives in P. by participants in H&1 votes on perspectives in P.2 by participants in H and votes perspectives in Pa submitted by fraction p of participants in H where
[0, 1). F. (c%, Ey
15

---

Under review as a conference paper at ICLR 2023
Hai.j,k E Pa) Pa) feg Gikk lie H,k U fei lie Hr,2.j.k Pai. H1 C Hz2, | //122////HH.... contains max number of participants with votes on perspectives in both Ps1 and Ps2. When p = 0, the train data has zero participant overlap, i.e., it contains with votes on perspectives in both Pai and P.2 (Figure 13a).
= p). When p the train data has maximum participant overlap, i.e., it
Type II with Different languages. For each question, we randomly split the participants into two groups H.jand Ha2. and their corresponding perspectives P.1 and P.2. The train set consists of exercises pertaining to perspectives in P&1 submitted by participants in He1. and exercises pertaining to perspectives in Ps2 submitted by participants in Hs2. E, = { Siz (ely, eis | Hs2, j, k }.2). The validation set consists of exercises pertaining to perspectives in Psi submitted by participants in Hs2, and exercises pertaining to perspectives in Ps2 submitted by participants in Hs1, E. {etz> Cijk k E H.2. The second group perspectives are translated to nine languages respectively: German (de), Spanish (es), French (fr), Italian (it), Japanese (ja), Dutch (nl), Portuguese (pt), simplified Chinese (zh) and traditional Chinese (zhtw).
H,,,J, k E Pa U
P.1] U feiy.ei I | E H. k P=2)-
A.6.2 TYPE III: MULTIPLE CONTEXT, OVERLAPPING SAMPLE
Transfer learning. Given a pair of contexts Q1 and Q2 we denote all perspectives in Q1 as Pl and all perspectives in Q2 as pa We split the participants who participated both contexts randomly into two groups H,1 and H.2. The pre-train set contains exercises pertaining to perspectives in pl submitted by all participants, Fu1 = {el:: Cigk E H. k E P!.. The second train set contains exercises pertaining to perspectives in p2 submitted by participants in Hs2, E2 {efj Gyk i H, E p2]. The validation set contains exercises pertaining to perspectives in pa submitted by participants in Hs1, Eg = {ee,. Cijx l H,i,j, k E P2). Participant fraction r is defined as r r= n== (Ha)) (++n(HH))) (Figure 13
The model is pre-trained using the pre-train set FA,1, and the learned participant embeddings transferred to the next training cycle using the second train set Eg2. In the second training cycle, z is fixed while the new context transformation M is learned. Then the participant embeddings learned from pre-training and context transformation M learned from the second training are together used to make predictions for exercises in the validation set.
Increased pre-training data. Given a target context Qt and its perspectives Pt, we select a set of N contexts that do not contain Qe as the pre-train contexts. We split the participants who participated in all of the selected questions randomly into two equal-sized groups H, and Hs2- The pre-train set contains exercises pertaining to perspectives the pre-train contexts submitted by all participants, En = Up= ( J Eij | H, k E p"]. The second train set contains exercises pertaining perspectives in Pt submitted by participants in H.2, Eez {ey, Sifk j,k E P']. The validation set contains exercises pertaining perspectives in pt submitted by participants in H,,, E, {elz, Gis Hat,j,k k E P'].
A.6.3 TYPE IV: MULTIPLE CONTEXT, NON-OVERLAPPING SAMPLE
Type III to Type IV transition. Given a pair of contexts Q1 and Q2 and their corresponding perspectives pl and p2, we split the participants who participated in both contexts randomly into two equal-sized groups H,1 and H.2. The pre-train set contains exercises pertaining to perspectives in pl submitted by participants in H,1 and exercises pertaining to perspectives in pl submitted by a fraction p of participants in Ha. where p E 1], E. = {ee) cf f i H,,, k E pT} U { Syy E H1 e pa H',² C Hs2, 1//l2////Ha = p}. The second train set contains exercises pertaining to perspectives in p2 submitted by participants in Ha. Er = {e E | i H k E p2). The validation set contains exercises pertaining perspectives in p2 submitted by participants in Ha1. E, { eg i E H,I.j. l E P2). When p = 1, En and E,2 have maximum participant overlap. When p 0, they have zero participant overlap (Figure 13b). T3b).
16

---

Under review as a conference paper at ICLR 2023
Figure 13: Elicitation primitive experimental setup
Mappe wellap
passejann
comple coodap
Type
Type
0.5
Typetrrte
0.5
62
Type
Typel
Type
10 Pype to Type M transiiion
A.7 DATA EFFICIENCY SCALING FOR ELICITATION PROTOCOLS
A.7.1 CASE
The first sample elicits me votes and initializes with N m participants and = m perspectives. participants and new
Each additional sample elicits additional 2mc votes and adds perspectives to O After q samples, 72 = mc(2g- - 1) votes have been elicited, and θ is comprised of K mg perspectives and N mq humans resulting in n(O) m² 2 Noting the relation q = (ni + mc) (2mc), then n()) = n2 ( + (2cc) + m/4 ~ On,)), data leverage is Sm, 2 1/(4c²) for n, >> mm/2.
3 (4e(2 + m/(2c) + m/(4m,) ) ~ (m.)) and as
A.7.2 CASE II
The first sample elicits me votes and initializes O with N = m participants and K = perspectives. Each additional sample elicits an additional me votes and adds m/2 new participants and m new perspectives to θ. After q samples, Tip = meg votes have been elicited, and θ is comprised of (m/2) + 1) humans resulting in n(@) (m²2² + m²²g /2. Noting the relation q nu/ (mc), then n ( n. n2c(22") + (nem)/(2c) ~ O(n)), data leverage is (2c2) m2cc) On,), and ag 1/ 12c²2
K = my perspectives and
Das
A.8 PROTOCOL SIMULATION EXPERIMENTS
Case I Given a set of perspectives corresponding Q differrnt contexts Pl...P and Q equal sized non-overlapping sets of participants H.....l.q. we construct the following experiment for each q 2...Q: let O be the HPM corresponding to - and - (P"). Partition the
training set in each case to correspond to local samples denoted by the shaded areas in Figure 9 such
that E 1 - (et-. Egk 1 H.n.j. j. is p"] U {ee, eis validation set E. contain all other exercises O9, denoted by the white areas in Figure
U
& p'} Let the
Case II Given a set of perspectives corresponding to Q contexts p1 PQ... and Q equal sized partially overlapping
of participants H.I...H.q with adjacent overlap p defined by [/ H
2.. p* H sil we construct the following experiment for each 2...Q:let @ be the HPM corresponding 0 ( and U {P*}. the train is denoted by the shaded areas in Figure 10 E Hsn, j, k E pa) and the validation set E,, contains all other exercises in O4
U U (eg E
denoted by the white areas Figure 10)
A.9 SPECULATION ON FOUNDATION MODEL ALIGNMENT SCENARIO
As foundation model (FM) capabilities increasingly span LLMs and RL agents Bommasani et al.) (202T): Reed et al. (002);; Chen et al. (2021b): Zheng et al. (2022): Janner et al. (2021), sharing an FM between an EIO model and an RT. agent may enable scenario where the quality and nuance of
17

---

Under review as a conference paper at ICLR 2023
normative alignment signals improves as the agent learns. As an example, this could happen in scenario where the following are true:
1. The quality of perspective representations (e.g. embeddings) produced by the shared FM, and used by the EIO model, improves as the RI. agent learns.
2. The performance (e.g. inference accuracy or data efficiency) of EIO increases as the quality of perspective representations improves.
A.10 PARTICIPANT PAYMENT AND CONSENT
The 534 participants who joined the load testing exercise had been solicited through online panel providers for an incentive approximately $15/hou USD, and consented the policy shown in Figure 14 While this consent form does mention PII, this provider and no PII was known to contained in the data set made available for this research. The researchers were structurally limited from accessing any directly collected PII due to system design, and none of the raw outputs that were inspected directly during the research included any PII.
general consent form for the data
18

---

Under review as a conference paper at ICLR 2023
Figure 14: Screenshot of consent policy presented all participants as part of the load test.
Conversation Terms
By clicking "accept" you agree to the Conversation Terms (including as to confidentiality) and to participate the conversation subject to such terms.
Pi Policy
By clicking "accept," you agree that collecting/poceessing personal information/personall data ('PF' (such demographic, employment related, consumer behaviours, etc.) on behalf of entities sponsoring this research activity ("Sponsors" - per Conversation Terms) which is used by the Sponsors to gather research insights. We may also share details of your participation with any sample provider who recruited you. For more information about how your Pl is processed, your rights and details, see our Privacy Policy; and with respect to a third party sample provider (if you were recruited by a third party), such third party sample provider's privacy policy applies. Your Pl is collected and stored on USA (hosted by Google as subprocessor) which perform data analytics to gather
contact
servers located in the
research insights, such data made available to the Sponsors and may be transferred by the Sponsors worldwide
Show Less
Sensitive Data Processing
By clicking "accept," you agree where the Conversation seeks 'special category' 'sensitive' Pl, such your race, religion, ethnicity, political opinions, health sexual orientation, you agree allow for all such Pl to be collected and used to gather research insights for our Sponsors and details of your participation may be provided to sample provider. Your Pi is collected and stored servers located in the USA (hosted by Google as subprocessor) and may be transferred by the Sponserwerldwidee
You may withdraw your consent any time, but this will not affect any use of your information before consent has been withdrawn. All Pl will be held until the Sponsor requests deletion or until you withdraw consent.
Show Less A
Cookies
We cookies improve your experience analyze website traffic, and may share your site usage data with our analytics partners service providers, outlined Privacy Policy. By clicking "accept," you agree cookies be the
stored your device. You change your cookie settings any time clicking
top left
of the Conversation.
Strictly Necessary Cookies (Always Active)
These cookies are necessary for the website to function and cannot be switched off in our systems. You can set browser to block or alert you about these cookies, but then some parts of the site may not work. These cookies do not store any personally identifiable information.
Statistical Cookies
These cookies allow us to count visits and traffic sources so we can measure and pages are
improve performance of our They help to know most and least popular and see how visitors move around the site. you do not allow these cookies we not know when you have visited our site, and not monitor its performance.
Selections marked with are required
Select All
agrree the Conversation Terms
agree the Pi Policy
agree to the Sensitive Data Processing
consent to use of all cookies (optional)
Accept
19


Reviews:
[{"review_id": "u9ivWRT35D", "reviewer": "Reviewer_pwkh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of Elicitation inference optimization where the goal is to get the perspectives of N participants on K different perspectives by minimizing the total number of elicitations. Elicitation could be direct: does participant i agree with perspective j or comparative, does participant i agree with perspective i over j. \n\nAuthors study the problem in the context of collective dialogue systems where the set of perspectives is an open-world meaning that super-set of possible perspectives is unknown apriori. The perspectives here are text-based; the paper proposes an algorithm that leverages logistic utility models with large-language models. In particular, they use a factored model where they learn a representation z_i for the i-th principal and w_j for the j-th topic. The utility of principal on perspective is u_ij = z_i w_j  and the response are assumed to follow sigmoid of these utility functions. In order to learn the representation of each perspective, the authors use a pre-trained LLM to generate text embedding e_i and then w_j is set to tanh activated affine transformation with parameter M. The matrix M and the parameter z_i are learned by maximizing a likelihood function on the observed data.\n\nThey empirically evaluate the performance of their proposed method (STUMP) against existing baselines, across a range of elicitation primitives arising from two data set properties: context homogeneity and sample overlap. Baselines used were random baseline, most-common baseline and nuclear norm baseline, on datasets with two types of contexts and different overlap assumptions. The proposed method is claimed to become increasingly data-efficient as the amount of data elicited grows. They first introduce a model that integrates a pre-trained LLM with a latent factor model. Finally, based on the evaluation, they design two elicitation protocols that guarantee meaningful accuracy while data efficiency increases linearly with scale.", "summary_of_the_review": "The problem setting could be interesting but already well-studied in the collaborative filtering/ matrix completion + representation learning  based literature. It is hard to appreciate the technical novelty of the paper. Moreover, the paper is very hard to parse and uses too many acronyms, and incomprehensible at time (some expt setup, theoretical novelty again existing baselines, etc). Repositioning the paper highlighting its key novelties (new contributions of this work), emphasizing on the reproducibility of the experiments, would greatly benefit the paper.   \n  ", "strength_and_weaknesses": "Strength -  \n\n- Interesting and practically relevant problem setup. The problem of matrix completion or collaborative filtering from preference or reward feedback has many applications.\n\n-  The idea of using pre-trained LLM for generating task embeddings might be interesting. \n\n- Extensive evaluations\n\nWeakness - \n\n- The paper is extremely hard to read, and uses too many acronyms (which I found at times unnecessary, say EIO, CRM etc) and hinders the flow of reading/ affects the comprehensibility of the paper. \n\n- Clarity: Notations are not well defined always:, e.g. E_t, E_v which are essentially the training and validation component of the main dataset E. I even found it hard to understand the problem setting and objective (beta_acc in Pg2) - it essentially is a fraction of total possible query to #required queries, so higher the accuracy better the performance, but it is written in a convoluted way in my opinion. Even the algorithm, embedding technique is not very well defined, I struggled to figure out the section/ paragraph which is dedicated for these details. \n\n- Novelty of the paper is not clear. On a high level, the work used a pre-trained LLM to generate text embedding and used Frobenius norm-based likelihood maximization to fit the model parameters -- such techniques ar fairly well studied. I fail to see if there is any new key idea offered in the current methods. \n\n- The paper did not discuss the theoretical performance of the suggested method, or compare that with any of the existing baselines.  \n\n- Experiments: the paper studies different experimental setups, but they do not seem to be reproducible since the settings are poorly defined, the used baselines (random, most-common, nuclear-norm-based optimization) also seem to be restricted as the literature of collaborative filtering is diverse and many recent works used more sophisticated techniques, which should have been compared against. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tItUX6xnX-f", "reviewer": "Reviewer_p4dW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of eliciting $N$ principals views across $K$ topics. In particular, the goal is to ask significantly less than $NK$ questions and infer the remaining entries from such questions. The authors use a factored model where they learn a representation $z_i$ for the $i$-th principal and $w_j$ for the $j$-th topic. The utility of principal $i$ on perspective $j$ is $u_{ij} = z_i \\cdot w_j$ and the response are assumed to follow sigmoid of these utility functions. Moreover, in order to learn the representation of each perspective, the authors use a pre-trained LLM to generate text embedding $e_j$ and then $w_j$ is set to tanh activated affine transformation with parameter $M$. The matrix $M$ and the parameter $\\{z_i\\}$ are learned by maximizing a likelihood function on the observed data.\n\nThe experiments compare the proposed method (STUMP) with a nuclear norm based method on datasets with two types of contexts and different overlap assumptions. The authors found that the proposed method performs well across all language pairs for the single context setting. For the multiple context setting, the authors showed that STUMP can reduce required amounts of data by leveraging data from a different context.", "summary_of_the_review": "The paper introduces an interesting idea of using pre-trained LLM to generate embeddings for different perspectives which are then used to generate opinions of experts on missing entries. However, the choice of the utility function is simple, and the method is also not compared against a set of different alternative algorithms.", "strength_and_weaknesses": "Strength: I think the idea of using pre-trained LLM for generating task embeddings is quite interesting. Additionally, the results on transfer learning looks promising as it can leverage samples across different contexts.\n\nWeaknesses:\n- The authors implicitly makes assumption of a low-rank structure. In particular the formulation of utility $u_{ij} = z_i \\cdot w_j$ implies that the proposed method needs to only learn $O(N+K)$ parameters. In that respect the proposed method is not very different than matrix factorization based methods.\n- The second drawback about the model is that the utility model assumes a specific structure ($P(a_{ij} | u_{ij}) = \\sigma(u_{ij})$) which is quite restricted in my opinion. There are more general models of preferences e.g. Thurston-Mosteller or Random utility model. The authors also did not motivate the choice of the particular utility model.\n- Perhaps the biggest weakness of the paper is the benchmark against which it is compared. Since the proposed method essentially learns low-dimensional embeddings for each user and perspective, the approach is similar to low-rank matrix factorization, and methods such as deep matrix factorization should have been compared. Additionally, both the proposed method and alternative nuclear norm based method are optimizing similar log-likelihood functions. This makes me think that STUMP does better only because it learns embeddings from pre-trained LLM.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F8XaaSQiPR", "reviewer": "Reviewer_wpVc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of Elicitation inference optimization. At a high-level, in this problem the goal is to get the perspectives of N participants on K different perspectives by minimizing the total number of elicitations. Elicitation could be direct: does participant i agree with perspective j or comparative, does participant i agree with perspective i over j. In this work they study this in the context of collective dialogue systems where the set of perspectives is an \"open-world\" meaning that super-set of possible perspectives is unknown apriori. The perspectives here are text-based; the paper proposes an algorithm that leverages logistic utility models with large-language models. They run sample experiments to show that this model can outperform random baseline, most-common baseline and nuclear norm baseline from prior work.", "summary_of_the_review": "Overall, this paper has enough interesting aspects to get my interest. I have read this as carefully as possible, but still overall have a few things unanswered. In particular, the ones I have mentioned in the weakness. I believe this paper needs a major rewriting which would be needed for me to also understand the paper even better to judge the paper's correctness. At this point, I made some best guess judgement on how the experiment is setup. I also think, the authors should add some stronger baselines from collaborative filtering line of work and make the comparison to CF grounded in data.", "strength_and_weaknesses": "Strength\n\n+ I personally found the problem setting to be very interesting. The paper was a bit hard to parse, but an example provided in the paper helps. The goal of the problem is to predict whether a user will agree with a perspective in dialogue systems. An example is when a user is asked \"What would you say if you saw your parents smile?\" would they agree with the perspective \"\"I feel happy\". These questions are very important for designing interactive real-world systems and thus, designing good algorithms that are robust in a wide variety of datasets is important.\n\n+ The paper overall does extensive empirical validation and performs quite competitively against the one non-trivial baseline from prior work in a number of setting. To that end, it seems to show good results. However, I am a bit worried that the other two chosen baselines are pretty weak and not a good comparision except that it provides a good sanity check.\n\n\nWeakness\n\n+ My primary qualm with the paper is that its pretty poorly written. The paper introduces a number of concepts in section 2 and then barely uses them in the paper. I think its better to introduce the terms at the location where they are used, if they are going to be used only once or twice. Another worry is that the empirical setup is not well explained. Its not clear to me what the model was trained on, what was the dataset, what were the inputs, how was the training and evaluation done, etc. The paper directly jumps to talking about the results. The empirical section is very hard to read. I believe more details needs to be added in that section so that the reader can verify the setup and details. From a reproducibility stand-point, its unclear how to set this experiments up if someone chose to.\n\n+ Overall, the paper also needs to add more motivations for the problem. Although the problem is interesting technically, adding a concrete application would greatly benefit the reader. Since the authors compare this problem to collaborative filtering, I am surprised that the baselines chosen in the experiments do not have anything from the recommendations literature. Since there is a 1:1 correspondence between the two problems, is there a possibility of comparing against stronger baselines from the rec-sys literature?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8VOfeMCjNEk", "reviewer": "Reviewer_Pf2b", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of approximating N principal’s views across K perspectives (which can be represented as a N*K matrix) with a minimum number of direct elicitation. They improve the data efficiency (N*K/# of direct elicitation) of the existing approaches by transferring learning from one context to support inference in other contexts and leveraging the information in the perspective text. They propose a method that becomes increasingly data-efficient as the amount of data elicited grows. They first introduce a model that integrates a pre-trained LLM with a latent factor model. They empirically evaluate the performance of their model across a range of elicitation primitives arising from two data set properties: context homogeneity and sample overlap. Finally, based on the evaluation, they design two elicitation protocols that guarantee meaningful accuracy while data efficiency increases linearly with scale. ", "summary_of_the_review": "Overall, I think this is a good paper with meaningful conceptual and methodological contributions, as well as insightful empirical evaluations. They innovatively improve the existing method by adopting LLM technologies. They also provide extensive empirical evaluations that give insights into the performance of their methods.", "strength_and_weaknesses": "Strengths:\nThe paper proposes an unconventional method that improves the data efficiency of the existing EIO methods. The idea of integrating pre-trained LLM is novel and meaningful. Their model allows context-specific learning with a shared LLM across different contexts, which transfers learning from one context to other contexts. The model is evaluated comprehensively in four primitives, which gives insights into the performance of their approach. They also provide two novel elicitation protocols based on their empirical evaluation results,  and they show by both theoretical analysis and empirical simulations that their data efficiency increases linearly with scale. \n\nWeaknesses:\nI'm not super familiar with the literature and I find no glaring weakness in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject