ThinkSum: Probabilistic reasoning over sets using large language models
Authors: Batu Ozturkler
Nikolay Malkin
Zhen Wang
Nebojsa Jojic
Venue: Submitted to ICLR 2023 (2023)
Keywords: NLP
language models
prompting
zero-shot learning

Abstract: Large language models (LLMs) have a substantial capacity for high-level analogical reasoning: reproducing patterns in linear text that occur in their training data (zero-shot evaluation) or in the provided context (few-shot in-context learning). However, recent studies show that even the largest LLMs fail in scenarios that require reasoning over multiple objects or facts or making sequences of logical deductions. We propose a two-stage probabilistic inference paradigm, ThinkSum, that reasons over sets of objects or facts in a structured manner. In the first stage (Think -- 'fast' retrieval of associations), a LLM is queried in parallel over a set of phrases extracted from the prompt or an auxiliary model call. In the second stage (Sum -- 'slow' probabilistic inference or reasoning), the results of these queries are aggregated to make the final prediction. We demonstrate the advantages of ThinkSum on the BIG-bench suite of evaluation tasks, achieving improvements over the state of the art using GPT-family models on ten difficult tasks, often with far smaller model variants. We compare and contrast ThinkSum with other proposed modifications to direct prompting of LLMs, such as variants of chain-of-thought prompting. We argue that because the probabilistic inference in ThinkSum is performed outside of calls to the LLM, ThinkSum is less sensitive to prompt design, yields more interpretable predictions, and can be flexibly combined with latent variable models to extract structured knowledge from LLMs.

THINKSUM: PROBABILISTIC REASONING OVER SETS USING LARGE LANGUAGE MODELS
Anonymous authors Paper under double-bling review
ABSTRACT
Large language models (LLMs) have substantial capacity for high-level ana- logical reasoning: reproducing patterns in linear text that occur in their train- ing data (zero-shot evaluation) or in the provided context (few-shot in-context learning). However, recent studies show that even the largest LLMs fail in sce- narios that require reasoning over multiple objects or facts or making sequences of logical deductions. We propose two-stage probabilistic inference paradigm, THINKSUM, that reasons over of objects or facts in a structured manner. In the first stage (THINK 'fast' retrieval of associations). L.M is queried in parallel over set of phrases extracted from the prompt or auxiliary model call. In the second stage (SUM 'slow" probabilistic inference or reasoning), the results of these queries are aggregated to make the final prediction. We demonstrate the advantages of THINKSUM on the BIG-bench suite of evaluation tasks, achieving improvements over the state of the art using GPT-family models on ten difficult tasks, often with far smaller model variants. We compare and contrast THINKSUM with other proposed modifications to direct prompting of LLMs, such as variants of chain-of-thought prompting. We argue that because the probabilistic inference in THINKSUM performed outside of calls to the LLM, THINKSUM less sensi- tive to prompt design, yields more interpretable predictions, and can be flexibly combined with latent variable models extract structured knowledge from LLMs.
INTRODUCTION
Large language models (LLMs) (Brown et al., 2020 Rae et al. 2021 Chowdhery et al. 2022 can recall broad range of basic facts, recognize and mimic various forms language, and efficiently extrapolate analogies in structure and meaning. These abilities allow LLMs to excel in zero-shot and few-shot tasks that are formulated generation selection of a likely completion of prompt. This formulation requires LLMs to perform associative fast thinking, in which each token of text in the sequence making up the answer is generated or scored in one pass through the model and, other than that, no intermediate information is created retained. Fast thinking is made possible by the compression in the LLM weights of information that is repeated in a variety of ways in large training datasets.
However, is increasingly evident that when reasoning, or slow thinking. is required, failure modes of LLMs are revealed. In our usage, reasoning is sequential manipulation of concepts that can be expressed in language. Tasks that require iterative retrieval of rarely stated knowledge, uncertain- ties over multiple objects or facts, or multiple steps of deduction are difficult even for the most advanced LLMs. In recently designed suite of evaluations, BIG-bench (Srivastava et al.) 2022), some of the tasks where the gap between machine and human performance large involve infer- ence sequences with nested counterfactuals (LOGICAL DEDUCTION), concepts introduced though definitions (CONCEPTUAL COMBINATIONS), etc. (see Fig. A.I.. These are tasks where human solver's ' intuitive feeling "(in)coherence' is not sufficient to produce the right answer: the solution is obtained by sequence of thoughts that can be explained in words and may even require writing down intermediate results working memory is insufficient.
We show on several examples BIG-bench that such problems can be addressed by a two- component mechanism, which name THINKSUM:
THiNk (fast thinking association / knowledge retrieval step): creating an association of spans of text with sets of strings. This process may involve generation from a language model, as is the case in Fig. [ where the novel word 'hinne" is associated with the set of strings ('cat', mink'....)

---

DIRECT PROMPTING
A binne is any furry four-legged creature, and a bam simple dwelling.
A binne bam is place for people (55%) animals (44%) birds (0.87%) researchers (0.022%)
CHAIN OF THOUGHT / AUXILIARY KNOWLEDGE
A binne is any furry fourlegggd creature, and a bam is a simple dwelling.
Examples of binnes: cat, mink, ferret, guinea pig, rabbit.
Examples of bams: hut, cabln, cottage, shelter, shack.
A binne bam is place for people (51%) animals (48%) birds (0.76%) researchers (0.011%)
THINKSUM
A binne is any furry four-legged creature, and a bam is a simple dwelling. binne = (cat, mink, ferret, guinea pig, rabbit}
THINK (auxiliary LM calls define sets)
bam = (hut, cabin, cottage, shelter, shack)
A cat cottage is a place for A rabbitt cabin place for mink shelter is a place for
SuM (aggregate LM likellhoods)
binne bam is a place for animals (65%) people (34%) birds (1.5%) researchers (0.056%)
Figure 1: An example adapted from the CONCEPTUAL COMBINATIONS (INVENTED WORDS) task, in which models must select the most likely completion of a phrase that includes nonce words whose definitions are given. Direct prompting evaluates completion likelihoods normalized over the four answer choices ( ('people', 'animals', 'birds', 'researchers') ). Chain-of-thought-ll iliary knowledge approaches would query a LLM or knowledge base for additional context. Our THINKSUM approach to this task queries LLM (GPT-2 XL) to produce sets examples defining the nonce words, then marginalizes over substitutions of these examples into the target phrase.
or aux-
by prompting GPT-3 with the definition and asking for examples. However, it may also consist of scoring alone, in order to form a matrix probabilities over which probabilistic inference performed.
SUM (slow thinking Summarization reasoning step): probabilistic inference that aggregates generated strings or probabilities to produce the final answer. The summarization typically in- volves, and often entirely consists of, summing of probabilities of strings (computed in the THINK step), as in Fig. where the final word is assumed be sampled from a mixture of possible substitutions of "bnnee" and 'bam' words into the input.
THINKSUM is named by analogy with other algorithms with two basic operations that 'expand' and 'aggregate', , like MapReduce in distributed computing and cum-product in graphical models.
We discuss different ways to THINK and to SUM in section 2 but we start with one example, il- lustrated in Fig. [ motivated by the CONCEPTUAL COMBINATIONS (INVENTED WORDS) task in BIG-bench (Srivastava et al., 2022). In this task, the LIM is provided with two invented words and their definitions in the input. The LLM is then asked to infer the most plausible sentence that uses a combination of the invented words. As the words are invented, they are not common or consis- tently used in the training set, and the LLM needs to understand and combine the definitions of the invented words reason about the meaning of the combination. The LLM is queried to produce example instances of the invented words with the help of the definitions. These example instances can be substituted into the query in place of the invented words. In this way, by mapping individual spans of the text of interest to sets we arrive at a mixture model (in this example, a mixture with 25 components, for 5 possible replacements of each of the words), which can be used in the same man- ner the original LLM used, either score text or to generate it token by token. In this case, when we score all candidate completions using this mixture model and normalize over the four choices, the correct answer that 'binne bams' are for animals and not people- becomes most likely.
An important difference between THINKSUM and existing chain-of of-.hought---oooooooooo -ike like prompt engineering methods (Wei et al.] 2022 Kojima et al.) (002). is that the reasoning step is not reduced to a gen- eration problem for the LLM, but is performed
probabilistic inference external the LLM. This reduces its vulnerability to features of the prompt, such as accidental distraction of the LLM by spurious patterns. Instead, we engineer the slow thinking process to make parallel calls to the LLM query for intermediate information, then possibly perform programmatic recombination of strings (THINK). The final reasoning step in which likelihoods obtained from the LLM for the recombina- tions derived from earlier steps of the reasoning process combined to make the final prediction
2

---

iss left to classical probabilistic reasoning (SUM). In sense, SUM replaces the self-attention mech- anism over linear text, which is used as the sole 'reasoning' mechanism chain-of- ofrhough- like approaches that expect the intermediate "thoughts" to take the form of generated tokens intervening between the input and output. Fig. [ shows the potential brittleness of such 'reasoning', especially in smaller models, which have stronger recency bias (Malkin et al.] 2022): if we simply list gen- erated examples additional context in the prompt, the recency bias causes the LLM to still give higher probability to 'people' than to 'animals', simply because 'bam' (simple dwelling) examples are given after the 'binne" examples.
Imposing alternative reasoning system over associative "knee-jerk reaction" system has analogy with models of human cogmitive processes (Tversky & Kahneman, 1974, Kahneman, 2011 that separate System (fast thinking) and System 2 (slow thinking). System 2 acts as a "controller' that can prime System to appropriately bias its fast thinking, and, in the context of reasoning with deep learning models, has been interpreted as operating with sparse concepts that can be described in language (Bengio 2017 Goyal & Bengio (020). Through repeated usage, the slow-thinking functions of System 2 can become efficiently compressed into System intuitions, in the same manner that iterative 'reasoning' functions of which smaller LLMs are not capable become zero- shot generation capacities for large LLMs.
As is the case with humans, there is always the next frontier of problems where a trained model with remarkable 'intuition' needs to be slowed down. The main claim of this paper is that more is possible with LLMS of existing scale when they are used concert with a wise System controller that allows for probabilistic inference.
FAST AND SLOW THINKING WITH LLMS
The example in Fig. falls into general THINKSUM paradigm which extends the reasoning capa- bilities of a given model by explicitly associating certain text spans with sets of other strings, which may serve as alternatives ferred by the LLM itself. These associations then provide multiple texts to be evaluated, again by the LLM itself. The collection of resulting probabilities provides an opportunity to summarize the text using standard probabilistic inference techniques, which usually include summation.
elaborationssexplaaattoo and be either defined by the
in-
2.1
How TO THINK
Here we list examples of the "fast thinking" that precedes the summarization stage.
Elementary string manipulations. Standard ways to turn a question into a prompt that can be given to a LLM for generation or scoring involve choices (e.g., of the prompt format) that can be seen as being made by a controlling agent. The standard approach to multiple-choice questions is to write them Cloze tasks. However, there are nontrivial operations used in inference procedures that sometimes work better, such as:
Order inversion:
Exchanging the order of the question and answers, as in Min et al. (2022).
Premise erasure: Deleting a part of the question. Removing a premise with which the answer is expected to have high mutual information is a step in inference procedures that aim to correct for bias towards answers with high unconditional likelihood (Zhao et al. 2021 Holtzman et al.
2021 Malkin et al. 2022).
Substitution and normalization. An example is shown in Fig. 1 Elements from a set may be substituted in place of 'slot' words in prompt, such 'cat' substituted for "binne' in the prompt "A binne bam is a place for". This operation can be combined with nontaaxxnomalzzaiionn steps that are reliably achieved by standard NLP tools, such as ensuring subject-verb agreement.
Example and list generation. A LLM can be prompted to generate or score lists of words or phrases. We suggest and experiment with three instances of this:
Example generation: In Fig. the LLM is prompted to turn definition or characterizing prop- erty, such as "simple dwelling into list of examples. This can be achieved with a prompt such Examples: 1.". The generated completion can be parsed into set to be used later in the inference procedure.
as "A bam is a simple dwelling,
List extension: A similar approach can also be used to hallucinate additional possible answers to
questions, as we will show in some of the experiments.
List of words: Similar prompts provide an even simpler THINK method that we use for scoring but not generation in several tasks. Just prompting a LLM with "List of words: A,
3

---

8", where A and B are words phrases, and computing the likelihood of B conditioned "List of words: A, " a good measure of semantic relatedness and B.
Fact generation. This way of THINKing associates an input word with set of phrases, in a similar manner generating examples from definition. It be achieved with prompts such as "L.is facts about cats. 1. The generated facts are good targets for substitutions of other con- cepts ('dogs', 'galaxies") in place of the concept ("cats") about which facts are generated. A variation on this asks the L.L to generate differences between two concepts, as shown Fig. 2 (right).
Translation. The LLM can be prompted to convert between different forms of representing the same concept as sequence of tokens. We use two basic examples of this in experiments:
Translation between languages by prompting the LLM in formats such as "French:
J' adore les chats noirs. English:". A very similar approach can be used to con- vert non-alphabetic symbols, such emoji, into words with similar meaning. Converting text to formal (symbolic) structures, like turning a word problem into collection of mathematical equations.
2.2 How TO SUM
Elementary inference. As above, we begin by listing existing standard ways of turning LLM outputs into answers, which we see as trivial cases aggregation (SUM).
Posterior computation by normalizing probabilities over set.
Majority/minority vote (argmin/argmax): component of most answer selection procedures. Thresholding: Used when an answer depends on the value of probability (or a difference or ratio of probabilities). This be used when a discrete real-valued output likelihoods.
needs to be produced from
Ratio of likelihoods: Likelihoods from different variants of the same prompt can be combined by considering their ratio or more general log- oig-linear other mixture. For example, this can be done to correct the likelihood of an answer conditioned on question by its unconditional likelihood, in combination with the Premise erasure operation described above.
Mixture (average) aggregation. A collection of prompts can be treated as the components of mixture model over completions. An example is shown in Fig- [ where substitutions of set of words yield 25 different prompts. Likelihoods of the completion over these 25 prompts are averaged.
Product aggregation. We use products of likelihoods in two different ways:
In similar way mixtures, but when the more natural probabilistic model has all elements of set (of prompts) generating the answer, such as when description or definition must satisfied by all concepts in a set.
In task where compute the likelihood of both and S' being true (as posterior over the tokens "True' and 'False' in appropriate prompt), then compare p ((TruelS))pl (FFFaalleSS (S is true and S is false) with p( D(False| (Sp (S is false and S' is true).
to determine whether statement S its negation S' is true,
Fitting latent variable models. See C for an example of fitting a discrete latent variable (clus- tering) model over the likelihoods produced by LLM.
EXPERIMENTS
We compare THINKSUM with the published results for n-shot GPT-3 175B (davinci) in BIG-bench, where n E {0. 1, 2, 3). Our main results are shown in Table [ Below, we describe THINKSUM for each task. Detailed descriptions are in and examples of each task appear in D.I
3.1 SEMANTIC RELATEDNESS
3.1.1 INTRODUCTORY EXAMPLES: PHRASE RELATEDNESS AND CODENAMES
Phrase relatedness. Each question in the multiple-choice PHRASE RELATEDNESS task requires to determine which a given set of words or phrases (w;) is related to query phrase q. We query the LLM for the likelihood of following a List of words prompt to form vector of likelihoods:
pi = (4 "List of words: Wir ").
The answer selected is the one with highest likelihood, arg max; P: (a trivial SUM operation). We note that this is also an instance of Order inversion: the query is scored following prompt in which each of the candidate answers substituted.
4

---

Table 1: Standard metric (BLEU for CODENAMES, accuracy for other tasks) for GPT-3 175B
(davinci) and THINKSUM with 175B (davinci), InstructGPT and GPT-2 XI. on BIG-bench tasks.
GPT-3 (davinci) s-shot
THINKSUM
Task
Avg. H
GPT-3
InstructGPT
GPT-2
PHRASE RELATEDNESS (33.1.1
0.74
0.37
0.42
0.52
0.59
0.85
0.87
0.79
CODENAMES ($3.1.
0.18
0.01
0.11
0.16
0.19
0.37
0.41
ODD
0.80
0.27
0.20
0.23
0.23
0.80
0.84
0.71
NOVEL CONCEP
0.67
0.47
0.47
0.56
0.56
0.72
0.75
0.50
INVENTED WORDS
N/A
0.29
0.14
0.14
0.21
0.64
0.71
0.29
SPORTS UNDERSTAN DINGT 3.3.1
0.71
0.50
0.50
0.50
0.50
0.71
0.74
0.54
KNOWN UNKNOWNS
3.3.
0.80
0.61
0.52
0.48
0.50
0.54
0.76
MISCONCEPTIONS RUSSIAN 3.4.1
0.65
0.33
0.33
0.41
0.35
0.70
0.61
EMOJI MOVIE (33,,1
0.93
0.12
0.18
0.12
0.19
0.80
0.75
FIVE OBJECTS ( 3.4.2
N/A
0.23
0.29
0.28
0.32
0.77
Prompt list differences:
1.
Name differences between things.
thing 1: apple
thing 2: strawberry
Number of differences:
Appli has pits.
Accuracy
2. Apple grows trees.
3. Strawberry is an aggregate fruit.
60
Strawberry is sweel. thing Glass thing Head
54
ThinkSum
Number of differences: 5
Auxiliary la cwlledge
1.
10
271.2
GFT-3
Number of differences
Figure 2: ODD ONE OUT. Left:
Performance of GPT-3 (n-shot, 0, 1,2, 3), auxiliary knowledge, and THINKSUM with various model sizes. Middle: Auxiliary knowledge vs. THINKSUM with varying number of differences. Right: Prompt used generate knowledge statements.
Codenames. Each question in CODENAMES requires selecting the words from a set (w:) that most closely related to query word We form vector p; in the same way as for PHRASE RELATEDNESS, then select the top entries in to produce the output
3.1.2 ODD ONE OUT
We examine possible THINK and SUM approaches in greater depth on the ODD ONE OUT task, in which the word in set W (w;) that is least semantically related to the others must be chosen.
List of words. We first consider an approach similar to that used 1.1 We form matrix Pij using a List of words THINK prompt for each pair of indices j:
Pij PLLM "List
words:
Wir ')
This matrix is aggregated by averaging over (in log domain) and selecting the i with lowest average, i.e., least likelihood of being generated by product mixture of all words in the set: i = arg min, 11, Pij. This is case of the Product aggregation operation.
Because this approach is the most successful with all model sizes we experimented with, its perfor- mance is reported in Table[] Remarkably, saaa-average-humae accuracy is maintained for all model sizes from GPT-2 Small the largest GPT-3 model (Fig. (left)).
Fact generation. As an alternative approach, we use a Fact generation prompt. An effective way to mine facts for semantic relatedness tasks is to consider two items in the same context in order get relevant facts regarding how items are related to each other (prompt Fig. 2 (right)). The demonstration used in the prompt ensures that the LLM generates statements in an expected format, which can be parsed and used for probability computation later. Using this prompt, obtain collection statements = {s,} about items Wj. We treat each generated s; as a template into
Because the task evaluated by BLEU score against the reference answers listed in alphabetical order, the right format. Alphabetization of
perform the additional step of converting the top indices to the short lists trivial in code, hut also very reliably be done by prompting GPT-3.

---

which different words can be substituted and denote by s,(w) the Substitution of word w into template 8;. We then form |S] x [W] matrix P.j. defined by Pij = ( Then, we can perform Minority voting: we take argmin over j and pick as the answer the most frequently occurring value, i.e., the item that is most often the least likely to fit a generated statement.
Comparison with auxiliary knowledge approaches. We compare our method with a knowledge- based prompting method, herein referred to as auxiliary knowledge. In auxiliary knowledge, we prepend generated facts the prompt before the question. Details of the prompt for auxiliary knowledge are provided in $D.2] In Figure 2 (middle), we show that the accuracy of Fact gen- eration-based THINKSUM rises the number of generated facts is increased, while the auxiliary knowledge technique peaks and then degrades as the prompt lengthens.
Fig. (left) shows how performance varies with the size of the LLM used for GPT-3, auxiliary knowledge and THINKSUM on ODD ONE OUT. Even with GPT-2 Small, THINKSUM dramatically improves over much larger largest zero- or few-shot models with or without auxiliary knowledge. The latest iteration of the largest GPT-3 model, text-davinci-002, is the only model variant that, with the help of auxiliary knowledge, achieves competitive performance with THINKSUM. This result provides experimental evidence for our claim that while new models may create qualitative jumps, THINKSUM can push the performance limits of smaller model variants through slow thinking. Additional experiments on auxiliary knowledge are provided in $C
3.2
SUBSTITUTION AND AGGREGATION
3.2.1 NOVEL CONCEPTS
In the multiple-choice NOVEL CONCEPTS task, set of words or phrases {we) and set of statements S {5y} with third-person plural pronoun subjects ("They all...) are given, and the statement which is true for all items in W must be determined.
We treat each statement s as a template, into which words w can be substituted by replacing 'They all' with w. Denoting by s (w) the substitution of into sj. we form [W] x [S] matrix P.j by scoring the Substitution of each word into each statement and considering the Ratio of likelihoods
with the template without substitution: Pij (w)))
PaiM(r)) WW then perform Product aggregation
to select the statement which is most likely to be generated by all words in the set. To be precise, the selected statement is arg max; 11; Pij.
3.2.2 INVENTED WORDS
In INVENTED WORDS, two nonce words x2 are defined and the correct statement must be chosen out of set of statements S = { that begin with (possibly inflected forms of) "x] x2" (Fig. D
We use an Example generation prompt to obtain a set of example words fitting the definitions of x1 and x2. We thus obtain sets S₁ and S2 of words that can be substituted for x and X2, respectively.
We treat each statement sy as a template into which words w1 S₁ and w2 S2 can be substituted by replacing x; with w; and normalizing the syntax to ensure subject-verb agreement. Denoting by (w(,, w2) such a substitution, we form vector of probabilities pj by scoring the Substitu- tion of each possible pair of words into each statement and performing Mixture aggregation and considering the Ratio of likelihoods with the template without substitution:
Σ 2w SwieSi..2ESs
(WW(
S
PL)M(s))
The statement sj with highest likelihood under this normalized mixture, arg max, Pj. is selected.
3.3 UNCERTAINTY AND HALLUCINATION DETECTION
LLMs are prone to generating hallucinations that contain incorrect statements. The likelihoods of these statements are often dominated by short plausible patterns, which also makes it difficult for LLMs to evaluate their own uncertainty about fact. Thus, detection (Liu et al.) 202 I Zhou al., 2021) and reduction of such hallucinations is crucial for widespread use of LLMs in real applications. DDziri et al. 2021 Shuster et al. (021).
3.3.1 SPORTS UNDERSTANDING
Questions in SPORTS UNDERSTANDING ask to determine whether it is 'plausible' or 'implausible' that professional sports player.x (e.g., "Draymond Green', basketball player) performed an action

---

associated with sport (e.g., 'threw a touchdown', an action in American football). It is implied that the combination of x and a is plausible the sport with which player x is associated coincides with the sport in which action a is performed. We consider an approach that does not rely on identifying the latent variable (sport) as intermediate step and is thus more generalizable to other domains.
We use Example generation THINK prompt to produce set S of players who perform action a, then do Posterior com-
0.30
putation by normalizing the likelihood assigned by the LLM to each player in S, well x, performing action
posterior probability
0.25
0.20
P a")
Vy (U)) p(yla)
0.15
Zy'su(x) P d")
The statement is considered to be implausible if the posterior on is sufficiently low (Thresholding)- see Fig. 3
0.10
0.05
3.3.2 KNOWN UNKNOWNS
0.00
Droymend Groen
Peyton Manning
Brudy
Oress Brees
Aaron Redgers
Questions in the KNOWN UNKNOWNS task ask to determine whether the answer to a question is a certain precise concept
player
or 'unknown'.
Given a question q (e.g.. "What was the temperature in Cuzco on the day of the Emperor Vespasian's birth") and the candi- date precise answer (e.g., C5" we use a List extension prompt to generate a set of other possible answers to q. We then do a Posterior computation over S and the original an- swer a, similar to that used for SPORTS UNDERSTANDING:
Figure 3: Example posterior probabilities generated from text- davinci-002 for SPORTS UNDER- STANDING with the description "threw touchdown". The bas- ketball player given in the ques- tion Draymond Green has much lower posterior probability than the generated football players, from which we conclude the sentence "Draymond Green threw a touch- down." " is implausible.
PLIM("9? y²)
Vy E (a(()) p(ylg) = Ey'eataa (" y"")
The answer a chosen if the posterior on a is sufficiently high (Thresholding), and otherwise 'unknown' is chosen.
3.4
TRANSLATION
3.4.1
TRANSLATING BETWEEN LANGUAGES AND WRITING SYSTEMS
Russian misconceptions. In the MISCONCEPTIONS RUSSIAN task, the true statement must be chosen out of pair of Russian sentences: a statement and its negation t.
We first describe an approach that does not use translation and already performs better than random guessing and better than baseline methods that simply select the more likely of the two statements using the largest GPT-3 model, which has sufficient knowledge of Russian. We compute the posterior over the two hypotheses "s is true, / is false" and "s is false, / is true":
(L "T or F? PLLM ("E" "T or F? 5.
Answer: ")PLLM ("F" "T or F? f.
Answer: "),
Answer: ")PLLM ("T" "T or F?
Answer: ").
where T denotes True and Fa I se in the actual prompt. This is kind of Product aggregation. If the posterior on the first option is higher, s is chosen as the true statement; otherwise, / is chosen.
This approach can be combined with a Translation prompt that produces translations of s and r into English, then uses these translations in place of and in the above computations. The approach can be further extended by sampling a set of translations and performing Mixture aggregation over the translations. Our reported result uses 10 generated translation for each statement, but it is only 2% higher than the result using one generated translation.
Emoji movie. The multiple-choice EMOJI MOVIE task requires selecting the name of movie from list {m3} that is best described by sequence of emoji symbols (S)... Sn). . An Order inversion prompt performs best on this task using the Davinci variant of GPT-3: choosing the answer arg max (L
"Emoji describing the movie m;").
We also attempt to use a Translation prompt to obtain single-word English description wy of each emoji sj in s, then score using
arg max PLLM (W)... wn "Words describing the movie my").
7

---

(o) Question:
a ohelf, are books: yellow book, brown pney baok, black shind the left brown book the lefrnost."
book. The book the,
purple baok. The black book lep of the purple
PROBLEM:
OBJECTS:
There are some number of objects, objectl, object2, objoct3...
yellow book \ brown book |a gray book \n black book 'in purple book
Some statemscnss about their ordering given.
NUMBER:
We think objectl smaller object2.
We have reason to helieve that object3 the smallest.
STATEMENT:
First list all objects in the OBJECTS list.
The yellow book is the right of the purple book.
OBJECTS:: ya object ''n object2 object3
TRANSLATE ATTON:
Next say how many there are.
yellow book--uuple book
NUMBER:
STATEMENT:
The black book the left of the purple book.
Then turn each
an equality inequality, e.g.
TRANSLATION:
STATEMENT:
black bpok-c book
We think that objectl is smaller than ohject2.
STATEMENT
TRANSLATION
The black book the third from the left.
objectl cobject2
TRANSLATION:
STATEMENT:
black book=3
We have reason to believe that objoct3 the smallest
STATEMENT:
TRANSLATION:
The book leftmost.
object3ei
TRANSI ATION:
PROBLEM:
brown book=!
(b) Think Prompt
[c] LLM Output
Figure 4: Details for LOGICAL DEDUCTION. (a) Example question from the task, (b) demonstration for the THINK prompt, (c) example LLM output.
This approach performs slightly better than Order inversion alone using InstructGPT. However, does not work with the base GPT-3 models, which do not as reliably translate emoji to English.
3.4.2 LOGICAL DEDUCTION
In the LOGICAL DEDUCTION task, different types of items and clues regarding their placement are provided, as shown in Fig. aa) The goal is select the correct statement from set of statements about their placements. The task creators emphasize that this requires parsing information about multiple objects and their relationships, understanding rules regarding ordered objects in various scenarios, and iteratively applying of these rules. The LLM calls in the THINK stage of THINKSUM can perform mappings required to parse information and understand rules, and the SUM stage can integrate mappings of objects to the placements under these rules. Here, we can use a Translation prompt to map the given problem into a set of mathematical (in)equalities (Fig. 4c).
A Translation prompt as in Fig. 4 containing generic ordering statements and object names that are not used in the task sufficient to perform the translation from natural language to mathematical equations, as shown in Fig. 4 To solve the problem, we replace the problem statement with its trans- lation consisting of (in)equalities, and map each of the objects to the of strings corresponding to numbers from 1 to N, where N is the number of objects. This is accomplished by appending a given problem to the fixed prompt in Fig. 4b). which acts as demonstration for one-shot in-context learning. The ordering problems involve variety of types of objects (cars, birds, etc.) and types of orderings (by size, price, contest ranking, etc.). For a particular problem in Fig. 4Ja), we show the returned text from InstructGPT in Fig. c), showing that the demonstration used in the THINK prompt generalizes from objects ordered by size to books ordered by position.
Once translation of the problem into set of inequalities is obtained, the SUM stage considers all possible mappings of items to indices to determine the mapping compatible with the discovered set of (in)equalities. This can be done by an external algorithm or with the LLM itself, as an LLM may be capable of understanding that, for example, "2>3" is less likely string than "2> 1" (see Fig. B.I.
The probability of a target statement like "yellow book=2" " can thus be obtained by:
p("yellow book=2" TT) xc Σ PIIM ((T T, E T) U ("yellow book=2"(b))) DE(
where b denotes the vector of placements for the N items, 7= (T is obtained from the Trans- lation prompt as a set of strings, and s(b) denotes the substitution of the corresponding entry in b in place of the object name in the string s. The term inside the sum is a case of Product aggregation: the LLM likelihoods of all strings in the set are multiplied together.
In summary, our solution to this task involves composition of two THINK operations a Transla- tion into set of equations and then Substitution of numbers in place of item names and two
8

---

SUM operations- Product aggregation followed by Mixture aggregation. (Other options discussed in $C)
Results and discussion. For the 500 LOGICAL DEDUCTION problems with N 5 objects, THINKSUM yields accuracy rate of 77% (see Table D. besting the average human performance. When the necessary summations become large, becomes very unlikely that pure prompt engi- neering can be competitive, even humans need paper and pencil to create and attend to many alternative solutions, and would likely translate the premises into a simpler notation using a single letter (representing a variable to which a numeric value can be assigned) to represent each object, rather than directly attending to facts in the problem statement.
We also tested an auxiliary knowledge method akin to chain-of-though reasoning, where the infor- mation obtained with the prompt in Fig. D appended to the LLM input. In particular, the problem, together with its translation into inequalities, is used as a prompt to each of the answer options, and then the option with the highest likelihood is chosen for the answer. This approach does improve over straightforward zero-shot GPT-3 scoring, but only raises the accuracy to 50% (see Table B.I.
4 RELATED WORK
Improvements to LLM inference. After the discovery of the in-context learning abilities of LLMs, there has been explosion of interest in improving inference with LLMs in the zero-shot and few-shot setting Brown et al.) (2020): [Chowdhery et al. (2022): |ae et al. (2021). One approach
to improving the reasoning abilities of L.LMs involves appending, or learning to generate, auxil- iary knowledge within the prompt (Shwartz al., 2020 Zelikman et al. 2022 Nye et al. 2021a). Recently, more general auxiliary knowledge or
prompting methods have been pro- posed (Wei et al., 2022 Wang et al., 2022b Zhou et al.) 2022 Creswell et al. 2022, Wang et al.) 2022a Liu al. 2022b Later, Kojima et al. (2022) showed zero-shot of-uonhg prompting can improve performance on variety of reasoning tasks. This method does not require any hand- crafted few-shot examples, which is shared property with THINKSUM. (Nye et al. 202 observed that a dual-system approach where an associative "System I" and a logical "System 2" can increase coherence of LLMs in tasks such as robust story generation and grounded instruction following, The two-step paradigm in THINKSUM similar, where "System I" is the (querying of the LLM for) fast thinking, and "System 2" is the probabilistic inference step.
Brittleness of chain-of-thought prompting. Despite the recent success of chain-of-thougle ap- proaches, recent studies have raised concerns regarding the limitations of chain-of-thought ap- proaches. Webson & Pavlick (2022) observed that instructive prompts perform similarly with mis- leading or intentionally irrelevant prompts. Additionally, Ye & Durrett 2022)) showed improvements due to few-shot offtuuht are not observed in question answering, or natural language infer- ence. More critically, few-shot prompts highly sensitive to the order in which the samples provided, the prompt format, and the selection of in-context examples, (Lu et al. 2022; Zhao et al.) 2021). Thus, it is crucial to design techniques that are robust to such changes in the prompt.
Probabilistic inference. More recent approaches have proposed probabilistic inference ap- proaches for tackling true/false question answering and commonsense question answering (Jung) et al. 2022, Liu et al. (2022a). [iie et al.] (202T) presents Bayesian inference perspective on in- context learning, Dohan et al. (2022) formalizes and unifies existing prompting techniques in a probabilistic framework. Our work generalizes such approaches to perform arbitrary probabilistic inference outside of the L.IM.
5 CONCLUSION
In this paper we presented THINKSUM, a two-step probabilistic inference paradigm that reasons over sets in structured manner. The fast thinking stage of THINKSUM allows elementary string manipulations as well as natural language prompting, which may enable numerous approaches to solve a natural language task. Even with far smaller model variants, THINKSUM achieves state-of- the-art results on ten difficult tasks in BIG-bench using GPT-family models. The two-step paradigm allows operating over sets instead of manipulating the prompt itself, preventing sensitivity to prompt format during the probabilistic inference in TUINKSUM, which is performed outside of calls to the LLM. As a result, THINKSUM is more robust to prompt design, yields more interpretable predictions, and can be combined with many probabilistic inference approaches to tackle diverse set of tasks.

---

REFERENCES
Yoshua Bengio. The consciousness prior. arXiv preprint (et:::1770.08555500 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Horbert-Voss, Gretchen Krueger, 'Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. Neural Information Processing Systems (NeurIPS), 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204002311 2022.
Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inferenneee Exploiting large lan- guage models for interpretable logical reasoning. arXiv preprint arXiv:2205.09772, 2022.
A. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society B, 39(1):1-38, 1977.
David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A Saurous, Jascha Sohl-Dickstein, et al. Language model cascades. arXiv preprint arXiv:2207.10344 2022.
Nouha Dziri, Andrea Madotto, Osmar Zaïane, and Avishek Joey Bose. Neural path hunter: Re- ducing hallucination dialogue systems via path grounding. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pp. 2197-2214, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 10. 10.18653/v12021 1.11663/112222..mmmmmaaa.. emnnp- emm.ppmma malpp-mminnn.. 168
TRLhttps. / antho wotholloy..oo org/221.,
Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of human cognition. arXiv preprint arXiv:2011.1509/ 2020.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form com- petition: Why the highest probability answer isn't always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7038-7051, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 10.18653/v1/222 11..655/112222..mmmmmmm....... mmi.p-main.56... URL httphhpppp///allaaallllllgggo// org//021.
lemm p- ma 1 n.. 564
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations. arXiv preprint arXiv:2205.11822 2022.
Daniel Kahneman. Thinking, fast and slow. Macmillan, 2011.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 2022.
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. Generated knowledge prompting for commonsense reasoning. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 3154-3169) Dublin, Ireland, May 2022a. Association for Computational Lin- guistics. doi: 10. 10. 111186661//0222.ea lon. long.225.. 225.. URL htRp:tt:: //ac/ htt.aaannhhoolo.oo////000000 acl acl-1ong.20 225
Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. A token-level refereenc-ffre hallucination detection benchmark for free-form text generation. arXiv preprint arXiv:2 104.08704 2021.
10

---

Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad Shoeybi, and Bryan Catanzaro. Multi-stage prompting for knowledgeable dialogue generation. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 1317-1337, Dublin, Ireland, May 2022b. Association for Computational Linguisties. doi: 10. /8653/11/2022.21... /110:////1112222fnnnoeeeee findings-acl. 104. URL aattps://aclanthoen
org/ ott.:///.............a..........
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically or- dered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8086-8098, Dublin, Ireland, May 2022. Association for Computational Lin- wttp533///2222allllloooo.. acl acl- ong. 5
guistics. doi:
URL hwtpa/aaaaaaattoo.aaaaaaaooo www.:/0..rrg//2......
Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. Coherence boosting:
When your pretrained lan- guage model is not paying enough attention. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 5214-8236, Dublin, Ire- land, May 2022, Association for Computational Linguistics. doi: 10. 1/228663/1120022..alloooo URL https://ac wew.ceaantllllloooo ogy. 2022 acl- 565
Sewon Min, Mike Lewis, Hannanch Hajishirzi, and Luke Zeutemoyer. Noisy channel language model prompting for few-shot text classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5316-5330, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 19.18663/11120222aadlloggg 365. URL httph:ttt//aace httpppss//acaaaatooooo..... org. 2022 acl-long. 365
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyzz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2/ 2021a.
Maxwell Nye, Michael Tessler, Josh Tenenbaum, and Brenden M Lake. Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. Neural In- formation Processing Systems (NeurIPS), 2021b.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.022
2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Neural Information Processing Systems (NeurIPS), 2019.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. models are unsupervised multitask learners. 2019.
Language
Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv:2112.11446, 2021.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmenta- tion reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3784-3803 Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/w1/2021 htt.://./......insa.oaaaa mandings-emnl..220 URL pettps//accantthee
f findings emn p 320
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unsupervised commonsense question answering with self-talk. In Proceedings of the 2020 Conference on Em-
pirical Methods in Natural Langnage Processing (EMNLP), pp. 4615-4629, Online, November URL
2020, Association for Computational Linguistics. doi: 10. 10.11863/v1/// 1/2020. wwt20/000.mmp-mai.... htttps://aclaanhooooryy. org wttp://22m2....mmmmmmmmmmm.amm.m
373
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206. 04.15, 2022.
11

---

UNKNOWNS, Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. Science, 18544157)11124-1113 1974.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Rationale- augmented ensembles in language models. arXiv preprint arXiv:2207.0072 00002200770077,, 2022a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022b.
Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of
their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2300-2344, Seattle, United States, July 2022. Association for Computational Linguistics. URL https:
//aclanthoLory.orr
Jason Wei, Xuezhi Wang, Dale Schourmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 2022.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080 2021.
Xi Ye and Greg Durrett. The unreliability of explanations in few-shot in-context learning. arXiv preprint arXiv:2205.0344 2022.
Eric Zelikman, Yuhuai Wu, and Noah D Goodman. STaR: Bootstrapping reasoning with reasoning. arXiv preprint arXiv:2203. 11003.4460 /4465, 2022.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improv- ing few-shot performance of language models. International Conference on Machine Learning (ICML), 2021.
Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. Detecting hallucinated content in conditional neural sequence generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1393- 1404, Online, August 2021. Association for Computational I Linguistics. doi: 10. (0.18653/v1/0021. findings- acl. URL htttss://aclaaaee http://ww....ooooooooo./........ findings- 120
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schu- urmans, Olivier Bousquet, Quoe Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205. 10625, 2022.
A BIG-BENCH LITE
Figure A.I shows the performance margin between an average human and zero-shot GPT-3 on tasks in BIG-bench Lite, select subset of tasks chosen by the authors of the benchmark to showcase the most important aspects of LLMs that need improvement. The vertical black bar separates the dataset into tasks where GPT-3 is already within the margin of just 10% compared to the average human accuracy, and the harder tasks (on the left). We show in the main text that some of these harder tasks, in particular EMOлI MOVIE, CONCEPTUAL COMBINATIONS. NOVEL CONCEPTS, MISCONCEPTIONS RUSSIAN and LOGICAL DEDUCTION, the margins are shrunk considerably, often exceeding average human performance. Other tasks in BIG-bench lite such as LOGIC GRID PUZZLE and SYMBOL INTERPRETATION share a similar structure to the ad- dressed by THINKSUM, and thus could be investigated as part of future work. Another example where THINKSUM can be applied the CODE LINE DESCRIPTION task, where we observe in our preliminary experiments that simple order inversion can significantly outperform average human accuracy.
12

---

Absolute preferred metric
80
(hverage human)- shat
10% margin
60
20
emoji movie
conceptual combinations
vitaminc fact, verificattinn strange stories
misconcaptions rrussian boq lite json
hindu knnwleedge
code _ine,, descriiption
novel, concepts
known, unknowns
symbol interpretation
logical, deduction
strategyga
logic ggiddd puzzle
Saassss_eegatiooo Aeooooonn pegation
winowhy
language identification
play dellloglloggoooooo same or difffreentt
fome aa Fallacas Selaceees
Figure A.I: Margin between O-shot GPT-3 and average human performance for BIG-bench Lite tasks. Using THINKSUM, address many of the tasks that have greater than 10% performance margin with average human, and significantly reduce and often overturn the margin.
B ADDITIONAL TASK DESCRIPTIONS
B.1 SEMANTIC RELATEDNESS
Phrase relatedness. In this task, an input phrase, and four target phrases are presented to the language model. The language model asked to identify the most related choice from the listed target options. An example from the task is "For each word phrase, identify the most related choice from the listed options. 'n Input: home town \n Option: town center (n Option: location n Option: native city \n Option: home run"
Codenames. In this task, the language model is asked to identify words associated with given word. An example from the task is "Try to identify the words best associated with the word WHITE from the following list: yn book, anchor, rainbow, shoulder, tunnel, sack, drum, pacific, page, mark, gear, glacier. Give your answer in alphabetical order."
Odd one out. This task is aimed at evaluating the capability of LLMs in semantic relatedness. This task presents the model with four to six words, where all words except one word are semantically or grammatically related to each other. The goal for the language model is to identify the odd word. An example question from the task is "Pick the odd word out: glass, head, arm, leg, hand, foot".
B.2 HALLUCINNATIN DETECTION
Known unknowns. In this task, the aim is to measure the ability of LLMs identify whether the answer to a question is known, or unknown. If the answer to the question cannot be known, an imaginary answer is generated, and the model is asked to choose between the imaginary answer and the word "Unknown". If the question has a correct answer, the correct answer and "Unknown" are provided as options. An example question from this task is "How often did Abrahan Lincoln cut his toenails?", and the options are "Every Saturday night", "Unknown", Similar to the sports understanding task, we can use an LLM to generate other plausible answers with the following prompt: How often did Abraham Lincoln cut his toenails? A possible answer is: \n2 1. Every Saturday night. List other possible answers the same formal as the first: 'n 2. Then, the answer given in the question is predicted to be "known" if its posterior is higher by the second most likely option by some margin. In experiments, chose this value to be 1/Ne where Ne is the number of examples, including the original option.
Sports understanding. In this task, artificial sentences are constructed using the name of profes- sional sports players and actions from particular sports. The model is then asked to identify whether
13

---

Table B.1: THINKSUM vs. auxiliary knowledge.
ODD ONE OUT
PHRASE RELATEDNESS LOGICAL DEDUCTION (N 5)
THINKSUM
0.84
0.87
0.77
Auxiliary knowledge
0.71
0.75
0.50
i i<j
i=j
i>j
2
3
4
5
5
6
7
8
6
7
8
9
1
2
3
4
5
6
8
4
5
6 7 8 9
Figure B.1: Probabilities of different (in)equalities according to GPT-3 text-davinci-002 (logit).
the sentence is plausible, where sentence is considered plausible if the sport of the player matches the sport of the action described in the sentence. An example from the task is "Statement: Draymond Green threw a touchdown. Plausible/mmplausible??
For THINKSUM to be able to parse outputs, GPT-3 generations need to be in pre-determined format. While larger models can obey format without demonstrations, for smaller models it is helpful to demonstrate the format with an example. Thus, we use the following prompt: "List 4 examples of players who scored rabona goal. /n I. Cristiano Ronaldo' n 2. Erik Lamela's n 3. Mario Balotelli Angel Di Maria' n List 4 examples of players who threw touchdown. \nl.".
B.3 CONCEPT UNDERSTANDING
In the following tasks, the shared goal to test the ability of LLMs on concepts over entities that have likely not been observed during training.
Conceptual combinations: Invented words. In this task, the LLM provided with two invented words, and their definitions in the input. The LLM is then asked to infer the most plausible meaning resulting from the combination of the invented words. As the words are invented, they are not present the training set, and the LLM needs to understand and combine the definitions of the invented words reason about the meaning of the combination. An example is: "The word 'binne' means any animal that furry and has four legs, and the word 'bam' means simple sort of dwelling. Question: Which of the following sentences best characterizes binne bams?". Similar to SPORTS UNDERSTANDING, we use the following prompt to force the LLM to obey fixed format: "List synonyms of binne, separate synonyms by comma:"
Novel concepts. In this task, the LLM is presented with two to four disparate entities that typically would co-occur frequently, but share an underlying conceptual or linguistic concept. The aim is to test the ability of the I.LM to reason about entities that are unlikely to have been observed in the same context during training. In multiple-choice setting, the L.LM is given concepts relating to the entities, and asked to generate the intended concepts against carefully chosen tempting distractors. The choices are not presented in the prompt. An example question from the task is as follows: "What do the following have common? 1) bumble bees 2) 01010101 3) race cars". and the answer options are They all make noise, "They all are yellow, They all are binary, They all go fast, They all have stripes".
B.4 TRANSLATION
Emoji Movie. In this task, the LLM is presented with emojis that describe the plot of a movie, and the model is asked to guess the correct movie from the given options. The task evaluates the LLMss understanding of emojis, and its knowledge of popular movies. An example from the task is
14

---

Table C.I: Different alternatives of probabilistic reasoning with THINKSUM for solving ODD ONE
OUT.
Model
List of words
Latent variable model
Minority voting
THINKSUM davinci-002
0.84
0.67
0.70
THINKSUM davinci-001
0.74
0.77
0.70
"What movie does this emoji describe? {lying-face-emoji) \n, pinocchio \n heat 'n the big sleep, n inception in lilo stitch
Misconceptions Russian. In this task, the LLM is presented with sentences in Russian that con- tain misconceptions, and their true counterparts. The task evaluates if the LLM believes common misconceptions. One example of target pair from the task, translated to English is as follows: "Vaccines do not cause autism. \n Vaccines cause autism."
C ADDITIONAL EXPERIMENTS
THINKSUM vs. auxiliary knowledge. Table B.I shows the comparison between auxiliary knowl- edge and THINKSUM. All methods used text-davinci-002. For PHRASE RELATEDNESS, auxiliary knowledge was generated using the "list differences" prompt shown in Figure 2 (right). For both auxiliary knowledge and THINKSUM, number of differences were used, as that was the best for auxiliary knowledge, shown in Tigure2
THINKSUM ODD ONE OUT and PHRASE RELAT- EDNESS are solved with the list of words prompt. For LOGICAL DEDUCTION, the THINK prompt shown in Fig4 was included before the question in the prompt. In all cases, THINKSUM outperforms auxiliary knowledge.
Latent variable models for ODD ONE OUT. While the detection of the odd item can be performed with simple operations, it is also possible to assume that there is latent structure consisting of two or more clusters such that the facts and items belonging to cluster can be freely interchanged keeping probability of such combination high. While there are simpler alternative solutions to this task, the latent variable model enables selecting the facts that characterize the majority class, explaining why the minority item ruled as the odd one out. Thus, expanding on the problem and applying a latent variable model can help interpret the decisions of the system.
More formally, since / includes items that semantically related, and the odd, model i e / and f E F to be generated from a latent class E {0, 1}. Then, the conditional distribution can be modeled as:
P(i. P(Ilc)((f(c) p(i Σ P(i
The semantic components, groupings P(icc) and P(fcc) can be computed from the matrix using expectation-mxxinzzaio (EM; Dempster et al. 1977). Then, the score for an item belonging to a cluster and all other items m S. (m # i} belonging to another cluster can be found as Ecc' P(icc) mm P(m))')P(c)).
We show the effectiveness of the latent variable models in Table C.I where analyze differ- ent methods for solving ODD ONE OUT using the InstructGPT variants text-davinci-001 and text- davinci-002. For the "latent variable model" and "minority voting" methods, we use number of differences Na ing methods perform well, outperforming previous baselines reported in Table I. Each of these approaches can be applicable in other tasks of similar structure ODD ONE OUT.
The latent variable model is run for 200 iterations. All probabilistic reason-
Using GPT-3 or external algorithms evaluate inequalities in LOGICAL DEDUCTION. Fig- B.I shows the matrix of posterior probabilities evaluated using InstructGPT (text-davinci-O0) (00)) for computed using prompts the form "True or false: x<y? The answer is:" where x and are substituted with numbers from (1, 9), and normalizing the probability of the first token over the two options "true" and "false". These are the probabilities that statements T, would evaluate in (T].
strings of form "x=y", "x<y", "x>y" for x, y E (I.... 9). The probabilities
Given the capability of InstructGPT to both translate given logical deduction problems into (in)equalities (Fig. 4 and evaluate each of them after replacement of items with placement
15

---

numbers (Fig. B.I conclude that it is perfectly within its capabilities to parse and understand the problem, and the SUM stage is there simply to go over all possible mappings. the way a human might. But, just as a human might use shortcuts in the search, the SUM stage of THINKSUM could be implemented in more less efficient ways. For example, instead of summing over all possible assignments of the five items, we can avoid the ones that are not permutations of {1. 2, 3,4, 4,5). Fur- thermore, instead of using PLIM from Fig. B.I we can simply evaluate each inequality externally, giving high constant probability for each statement in T and the target statement where a con- figuration of item placements makes the statement correct and low constant probability whenever the statement is incorrect. Whichever evaluation mechanism whenever an incorrect statement is detected (typically there are four (in)equalities that describe the problem, and one equality statement that describe the option to be evaluated; want to find configuration of item placements such that all of these are correct, or have high probability under PLLM
use, the summing can be aborted
The prompt in Fig. 4 b) instructs the LLM to assign positive negative ordinal numbers depending on the language used (the smallest object gets placement 1, while the second largest one gets -2, meaning 'second from the end"). A negative order number is then turned in our code into N + +++11 before evaluating statements. Sometimes, L.MM does not follow this instruction, but simply labels the largest, right-most, most expensive, etc. as N, instead of -1, which is equivalent. One possible failure mode of this kind of THINKSUM is that the LLM may translate inequality statements incon- sistently with equality statements (e.g.. by treating the leftmost item as "I", and being consistent with this choice for other equality constraints, but translating inequality constraints consistent with the reverse order, with 'left of" meaning >). This can be dealt with by adding an option to replace placement numbers r in equality statements by N - r 1. This doubles the number of evaluations to be done (as each 7' now has two versions), but allows for auto- coro-corre in SUM.
C.I COMPARISONS WITH CHAIN-OF- APPROACHES
Following Wei et al. (022), we use "chain-of-thoughn" to mean LLM scoring approaches that use insertion of generated tokens between the prompt and the target answer. The model is taught, using few-shot demonstrations, how to generate these intermediate tokens. Above have compared THINKSUM with approaches that add extracted (from an auxiliary LM call), not generated (within the LM's linear workspace) token sequences after the prompt, for the ODD ONE OUT, PURASE RELATEDNESS, and LOGICAL DEDUCTION tasks (see Table
With suitable examples, it may be possible for a thain-of-thought approach to replace the THINK phase, by learning from demonstrations to generate the appropriate knowledge, and parts of the SUM phase, although inference over parallel evaluations of the LLM is no longer possible. Our auxiliary knowledge baselines make precisely that generous assumption and focus the comparisons on the need for parallel calls and reasoning over possibilities using probabilistic inference (instead of leaving it to the LLM to make the right conclusions from the list of extracted alternatives).
Although expect that appending facts in standard format to the prompt would help the model more than teaching the model to generate these facts, we experimented with Mfain-of-thought ap- proaches on several tasks. Table C.3 shows example demonstrations and prompt formats used for each task, and Table shows the results using two variants of the largest GPT-3 model.
Table C.2: Comparison of THINKSUM with chain-of-thought prompting approaches.
GPT-3 (davinci)
GPT-3 (davinci-002)
Task
Direct
CoT
THINKSUM
CoT
THINKSUM
ODD ONE OUT PHRASE RELATEDNESS
0.27
0.33
0.80
0.64
0.84
0.59
0.55
0.85
0.79
0.87
LOGICAL DEDUCTION
0.32
0.25
0.39
0.77
KNOWN UNKNOWNS
0.61
0.70
0.54
0.74
0.76
INVENTED WORDS
0.29
0.50
0.64
0.64
0.71
As expected, THINKSUM outperforms chain-of tf-ouhtt prompting on all tasks with all variants ex- cept KNOWN UNKNOWNS with the davinci variant, where direct prompting already performs well. (We did not evaluate THINKSUM with davinci on LOGICAL DEDUCTION because prompts like the one in Figure 4 did not reliably produce outputs in the correct format: notice that chain-of-thoup of-thhuggh is barely better than random guessing (20%).)
16

---

Glass is transparent.
Head is attached to the body by neck.
Auviliary Knowledge
Head is on top of the body.
Legs are
bottom of the body. Head is top of the body.
Foot on the bottom
body.
Arm has shoulder.
Hand has 5 fingers.
part A foot is part of the body that goes from the ankle to the toes.
body that goes from the hip the ankie.
has fingers.
Pick the odd word out: glass, head, arm, log, hand, foot Answer:
Figure D.I: Auxiliary knowledge prompting applied to ODD ONE OUT. Facts are generated using the "list differences" prompt described in Figure 2 (right) and post-processed according to $.2
When interpreting these results, it is important note that only prompt format was evaluated for both chain of thought and THINKSUM, and the format prompts and demonstrations can have a strong and often unpredictable effect the LLM. We observed that chain-of-thought approaches are highly sensitive to minor changes in the prompt format or the construction in-context examples, consistent with the known biases of in-context learning when a potentially lengthy prompt is evalu- ated (Lu al., 2022 (Zhao et al. (02II) On the other hand, using structured, shorter components is more reliable, as demonstrated by the efficacy of the THINK prompts used in THINKSUM.
D ADDITIONAL EXPERIMENTAL DETAILS
Our experiments performed using four different sizes of GPT-2 (Small, Medium, Large, and XL) (Radford et al., 2019), GPT-3 with four different model sizes
(Brown et al.] 2020), and InstructGPT (Ouyang et al. (022). All GPT-3 experiments are run between August 2022 and September 2022 by using the OpenAT API. Our GPT-2 experiments were in PyTorch
Paszke et al. 2019) and the Hugging Face Transformers library with a Tesla K80 GPU.
D.1
HYPERPARAMETERS
Maximum generation length. For tasks that require example and list generation, such as CONCEPTUAL COMBINATIONS, KNOWN UNKNOWNS, and SPORTS UNDERSTANDING, max tokens = 100. For fact generation in ODD ONE OUT with auxiliary knowledge and THINKSUM, max_tokens 1000.
Temperature. All GPT-2 experiments used temperature 0.5. For SPORTS UNDERSTANDING and translation tasks, we used temperature = 0.5 to promote diversity of generated plausible options. All other experiments used temperature 0.
Number of examples (Ne). For CONCEPTUAL COMBINATIONS we used N₂ = 2, and for KNOWN UNKNOWNS and SPORTS UNDERSTANDING we used Ne = 4.
Threshold. threshold of 0.01 used for SPORTS UNDERSTANDING.
D.2 KNOWLEDGE GENERATION DETAILS
post-processing. our knowledge generation experiments for both THINKSUM and the auxiliary knowledge approach, we post-pre process the generated knowledge statements, to ensure formatting does harm the predictions of each method. We first remove the extra spaces and the numbers and punctuation generated by the LLM before each fact while enumerating the items of the list. Later, only keep sentences that contain only of the objects of interest from the task, make sure each sentence contains a knowledge statement into which any of the objects can be substituted. Finally, sentences with less than 3 words are removed as these are not likely to contain informative statements.
Auxiliary knowledge. For auxiliary knowledge experiments, we prepend the generated and post- processed knowledge statements before the question in the task. An example is illustrated in Figure D.I
17

---

Table C.3: Few-shot demonstrations used for chain of thought (Table C.2
Words: blue, pink, magenta, banana
All words are colors except banana. The odd one out banana.
ODD ONE OUT
Words: pencil, eraser, baby, rule, notebook
All words are office supplies except baby. The odd one out is baby.
For each word phrase, identify the most related choice from the listed options.
Input: Ice Cream
Option: Antarctica
Option: Titanic
PHRASE RELATEDNESS
Option: Dessert
Option: Sour Cream
Ice cream is type of dessert. Therefore, ice cream and dessert are the most related.
Answer: Dessert
What was the population of San Francisco in 2018?
Option: 879,676
Option: Unknown
The question asks the population of San Francisco in 2018, for which data can be collected. Population data for cities on a yearly basis is available, and thus the answer is known, and it is 879,676.
Answer: 879,676
KNOWN UNKNOWNS
What was the population of San Francisco yesterday? Option: 891,402
Option: Unknown The question asks the population of San Francisco yesterday. As it is not possible to know the exact population of city daily basis, the answer for this question is unknown. Answer: Unknown
On a table, there are five plates: black plate, a white plate, green plate, blue plate, and red plate. The white plate is bigger than the green plate. The red plate is the biggest. The black plate is bigger than the blue plate. The black plate is smaller than the green plate. Which plate is the smallest?
Option: The red plate is the smallest. Option: The black plate is the smallest.
Option: The white plate the smallest.
Option: The green plate is the smallest. Option: The blue plate is the smallest.
LOGICAL DEDUCTION
The black plate is bigger than the blue plate. The black plate is smaller than the green plate, plate as well. The white plate is bigger than the green plate, which is bigger than the blue plate. As result, the green plate is bigger than the blue plate. The red plate the biggest, so is bigger than the blue plate. Since all other plates are bigger than the blue plate, the blue plate is smallest.
result the green plate is bigger than the blue
Answer: The blue plate is the smallest.
The word 'borger' are animals who bite specific things for fun, and the word 'folpt" is a type of a chewy toy. Question: Which of the following sentences best characterizes borger folpts?
Option: Borger folpts leashes for animals.
Option: Borger folpts are toys for infants.
INVENTED WORDS
Option: Borger folpts hard to swallow.
Option: Borger folpts are pet toys.
Borgers are animals, and folpts are chewy toys. Therefore, borger folpts are chewy toys that animals, or pets, can play with. Therefore, the answer is borger folpts are pet toys.
Answer: Borger folpts are pet toys.
18

---

Table D.1: List of examples for the studied BIG-hench tasks.
Task: PHIRSSE RELATEDNESS
Input: For each word or phrase, identify the most related choice from the listed options.
Input: home town
Option: town center
Option: location
Option: native city
Option: home run
Task: ODD ONE OUT
Input: Pick the odd word out: glass, head, arm, leg, hand, foot
Task: CONCEPTUAL COMBINATIONS: INVENTED WORDS
Input: The word 'binne' means any animal that is furry and has four legs, and the word 'bam' means simple sort of dwelling. Question: Which of the following sentences best characterizes binne bams?
Option: Binne bams for pets.
Option: Binne bams are typically furnished with sofas and televisions.
Option: Binne bams are luxurious apartments.
Option: Binne bams are places where people live
Task: NOVEL CONCEPTS
Input: What do the following have in common? 1) bumble bees 2) 01010101 3) race cars
Option: They all make noise.
Option: They all are yellow.
Option: They all are binary.
Option: They all go fast.
Option: They all have stripes.
Task: LOGICAL DEDUCTION: FIVE OBJECTS
Input: On shelf, there are five books: gray book, red book, a purple book, blue book, and black book. The red book is to the right of the gray book. The black book is to the left of the blue book. The blue book is to the left of the gray book. The purple book is the second from the right.
Option: The (gray, red, purple, blue, black) book is second from the right.
Task: CODENAMES
Try to identify the 2 words best associated with the word WHITE from the following list: book, anchor, rainbow, shoulder, tunnel, sack, drum, pacific, page, mark, gear, glacier. Give your answer in alphabetical order.
Input:
Task: KNOWN UNKNOWNS
Input: How often did Abraham Lincoln cut his toenails?
Option: Every Saturday night
Option: Unknown
Task: SPORTS UNDERSTANDING
Input: Statement: Draymond Green threw a touchdown. Plausible/implausiblee?
Task: EMOΙI MOVIE
Input: What movie does this emoji describe? (lying-face-emojii
Option: pinocchio
Option: heat
Option: the big sleep
Option: inception
Option: lilo & stitch
Task: MISCONCEPTIONS RUSSIAN
Input: Vaccines cause autism. Vaceines do not cause autism. [in Russian
19


Reviews:
[{"review_id": "xTt6FYBx4n", "reviewer": "Reviewer_KxRo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a reasoning framework, ThinkSum, which includes a fast think module and a slow sum module. The authors argued that this framework is good at performing many reasoning tasks with simple operations.", "summary_of_the_review": "The scope of the proposed ThinkSum framework for LLMs is not well defined. Please consider narrowing it down to a subset of tasks. The authors may also consider running some more challenging tasks to show the efficacy of the proposed framework.", "strength_and_weaknesses": "The performance of the proposed model is strong in their experiments. However, I am not sure if the improvement can be applied to other tasks. For example, the task in Figure 1 requires evaluating every possible combinations of the choices. This is doable for 5 choices, but not so much for 100 choices or more (in SQuAD-type QA, the output could be any combinations of tokens) which is extremely expensive. Prompting LMs to score all possible candidates is also not accurate if the number of candidates is large, because the union of errors.\n\nThe ThinkSum framework, however, is too general such that almost all existing methods fall into this framework, if you think scoring is the Think step, and argmax is the sum step.\n\nAlso, tasks experimented in this paper are classification tasks. You may consider running some regression tasks as well.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "sJhRiUlMvf", "reviewer": "Reviewer_2TCH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a 2-stage approach to perform reasoning in large language models (LLMs). The main idea is to decouple the fast thinking process of generating text for a prompt and the slower process of reasoning such as iteratively generating related sequences of text, etc.\n\nThe main motivation is to make LLMs more robust to prompt design by decoupling the step to answer prompts and the step to perform the actual reasoning over a complex task. To perform reasoning, the approach proposes to use probabilistic inference. Specifically, GPT-2 XL is used to generate sets of associations and probabilistic inference sums over these associations.\n\nThe paper seems to propose a simple idea but one that seems to work very well on a complex suite of tasks in Big-bench. In particular, it is shown that the approach achieves state-of-the-art results on 8 of the ten tasks. For each of the taks, three variants of GPT are used for prompting and the type of probabilistic inference performed is more specific to the task.", "summary_of_the_review": "Overall seems like an impactful paper with strong empirical results. They show that with their two-step approach, it is possible to achieve state-of-the-art results with much smaller models and thus should have an impact on scalability in future applications.", "strength_and_weaknesses": "Strengths\nSimple and intuitive idea but produces excellent results in a challenging domain\nSeems to be more scalable since it can help us achieve state-of-the-art results with simpler models\nThe experiments seems quite detailed and impactful since they show in most of the tasks the proposed approach yields state-of-the-art results\n\nWeaknesses\nOne general issue may be how does the model scale up as the sets produced by the prompts to LLMs become large. In general, probabilistic inference may tend to become less reliable in this case so does that have a major influence on the SUM process.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "adNDwEulvnl", "reviewer": "Reviewer_Xf9p", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes ThinkSum: a two-stage paradigm for performing inference with large language models (LLMs) with no gradient update. The authors distinguish ThinkSum from chain-of-though like prompting methods by claiming that ThinkSum performs probabilistic inference instead of using LLMs to directly generate answers. The authors demonstrate the ThinkSum approach on some selected datasets from the BIG-bench and show that it outperforms vanilla few-shot learning with GPT-3.", "summary_of_the_review": "Despite the extensive demonstrations performed in this study, the authors need to present the ThinkSum paradigm in a more organized and clear way: even the core idea of ThinkSum is not clearly explained the examples throughout the paper do not help much explaining ThinkSum paradigm. The experiments demonstrate the effectiveness of the approach in some sense but need comparisons against more baselines. ", "strength_and_weaknesses": "Though the authors present ThinkSum as a general framework for LLM inference in Section 2, the definitions for \"Think\" and \"Sum\" are very vague and do not well relate to the example shown in Figure 1. Then the rest of the paper is devoted to explain how ThinkSum can be used to tackle different tasks selected from BIG-bench in a case-by-case sense: each subsection covers one (class of) task, but it remains somewhat unclear how each example relates to the definition of ThinkSum presented in Section 2 and the connection between the individual tasks is unclear. Many new terminologies (in different colors) are introduced: some are completely unnecessary (e.g. product aggregation) and some do not have any clear explanations. The only baseline that the authors compare against to is the few-shot learning with GPT-3; the authors might want to include other prompt-based approaches as baselines. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "EVzKhXaA-nj", "reviewer": "Reviewer_nmJR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "1. This paper proposed a two-stage method that reasons over sets of objects or statements.\n2. The paper performs empirical evaluation on the BIG-bench benchmark and shows improvements over fewshot GPT3", "summary_of_the_review": "The writing is not clear, making it hard to evaluate the contribution of this paper.", "strength_and_weaknesses": "Strengths:\n1) Experimental results suggest the proposed method has an improvement over few-shot GPT.\n\nWeaknesses:\n1) The paper should reduce the usage of color in the main text\n2) The paper should reduce the amount of branding and focus on the actual technique.  I don't see how this has much to do with thinking fast and slow\n3) Figure 1 is confusing.  The think step involves manipulating the inputs to get new sequences instead of just generating a set of relevant words?  Then, the second step is the actual summing over the sequences generated in the first step.\n4) There is a lack of technical details.  In particular, it is not exactly clear how the sum step is performed.  If this follows a probabilistic framework, please write down the steps formally.\n5) This paper first contrasts to chain of thought prompting, but it doesn't compare to that in experiments.\n6) Table 1 is confusing.  What's the difference between GPT-3 175B (davinci) and InstructGPT?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject