SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rule for Concept #1: Something is positive if it is the biggest  $\downarrow$  yellow object in the example
Rule for Concept #2: Something is positive if there is another object with the same color in the example
Rule for Concept #3: Something is positive if it is the same color  $\nrightarrow$  as the smallest triangle in the example
Now please look at the following examples for a new logical rule.
An Example of Concept #4: POSITIVES: none NEGATIVES: (large yellow circle), (small green circle),  $\nless$  (medium green circle), (small yellow triangle)   
Another Example of Concept #4: POSITIVES: (small green circle), (large green circle) NEGATIVES: (large yellow circle), (medium blue circle)   
Another Example of Concept #4: POSITIVES: (small green rectangle) NEGATIVES: (medium yellow circle), (medium blue rectangle),  $\nless$  (large green circle), (medium green circle)   
Another Example of Concept #4: POSITIVES: (medium green rectangle) NEGATIVES: (medium yellow circle), (small yellow  $\nless$  rectangle), (medium yellow rectangle), (medium blue  $\nless$  rectangle)   
Another Example of Concept #4: POSITIVES: (small green rectangle) NEGATIVES: (large yellow rectangle), (small yellow  $\nless$  triangle), (medium green circle), (small blue rectangle)   
Another Example of Concept #4: POSITIVES: (medium green triangle) NEGATIVES: (medium blue triangle), (medium blue rectangle),  $\nless$  large blue triangle), (small yellow triangle)   
Another Example of Concept #4: POSITIVES: none NEGATIVES: (small yellow circle), (large blue circle)   
Another Example of Concept #4: POSITIVES: none NEGATIVES: (large green circle), (small blue rectangle),  $\nless$  small green triangle), (medium blue rectangle)   
Another Example of Concept #4: POSITIVES: (small green rectangle) NEGATIVES: (small yellow circle), (large blue rectangle)
Now we get a new collection of examples for Concept #4: (medium blue triangle) (large yellow triangle) (small blue  $\nrightarrow$  rectangle) (large blue circle) (small yellow circle) Question: Based on the above example, is a (small yellow circle) in  $\nrightarrow$  the concept? Answer (one word, just write yes/no):
A.6 Latent Language Baseline
For fair comparison, we designed our latent language baseline to be as similar to our system as possible. It performs maximum likelihood estimation of a single concept, rather than estimate a full posterior, but uses the exact same prompts and likelihood functions as our model. The most important difference from the original latent language paper [31] is that instead of training our own neural models for language interpretation and language generation, we use pretrained models (Codex/code-davinci-002 and GPT-4).
A.7 Ablation of the proposal distribution
We ablate the proposal distribution by proposing hypotheses unconditioned on  $X_{1:K}$ . We accomplish this by drawing concepts from the following alternative prompt, which is designed to resemble the prompt used by the full model except that it does not include  $X_{1:K}$ :
Python 3   
Here are a few example number concepts: # -- The number is even   
# -- The number is between 30 and 45
The number is a power of 3   
The number is less than 10   
The number is
A.8 Pretrained prior
Our pretrained prior comes from the opensource model CodeGen [44], which was trained on source code. We chose this model because we suspected that pretraining on source code would give better density estimation for text describing precise rules. We formatted the rules as a natural language comment and prefixed it with a small amount of domain-specific text in order to prime the model to put probability mass on rules that correctly talk about numbers or shapes.
For the number game, we would query CodeGen for the probability of  $p(C)$  via
Here is an example number concept: The number is  $C$
For the number game's code baseline, we would query CodeGen for the probability of  $p(C)$  via
Python 3   
# Let's think of a number concept.   
# Write a python function that returns true if num' belongs to  $\nrightarrow$  this number concept.   
def check_if_in_concept(num): return C
For logical concepts we would query CodeGen for the probability of  $p(C)$  via
Here are some simple example shape concepts:  
1. neither a triangle nor a green rectangle  
2. not blue and large.  
3. if it is large, then it must be yellow.  
4. small and blue  
5. either big or green.  
6. C
Because the proposal distribution would generate rules beginning with the prefix "Something is positive if..." we would remove that text before computing  $p(C)$  as above.