SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Direct Prompt Tuning. In this method, the model tries to guess the answer directly without generating a rationale; prompt-tuning to maximize the log-likelihood of the answers in this setup is straightforward, since there is no latent rationale to integrate out. We initialize the soft prompt using 3 examples from the training set and truncate its length to 64. The optimization procedure is carried out over 150 steps with batch size 16 and the same Adam hyperparameters as above, except that the cosine decay period is set to 150 instead of 450. We found these adjustments to the hyperparameters from different training schemes were necessary to reduce overfitting.
Bootstrapped 3-shot CoT Prompt for Formal Fallacies BBH Task
Question: "It is not always easy to grasp who is consuming which products. The following argument pertains to this question: First of all, being a frequent consumer of KMS shampoo is necessary for being a regular consumer of Yardley London soap. Next, whoever is a regular user of Joico shampoo is not a frequent consumer of KMS shampoo. All this entails that no regular consumer of Yardley London soap is a regular user of Joico shampoo."
Is the argument, given the explicitly stated premises, deductively valid or invalid?
- valid
- invalid
Answer: Let's think step by step.
Regular use of a product always entails being a frequent consumer of that product. Although a Joico shampoo user might still buy bottles of Yardley London soap sometimes, he/she can never be a regular product of both soaps. So, the answer is valid.
Question: "Here comes a perfectly valid argument: To start with, whoever is an expert of BSC Young Boys is not an ex-fan of Real Betis Balompié. Now, whoever is a friend of FC Dynamo Kyiv is not an expert of BSC Young Boys. Hence, no friend of FC Dynamo Kyiv is an ex-fan of Real Betis Balompié."
You can friend FC Dynamo Kyiv but not be an expert of BSC Young Boys. So, the answer is invalid.
Question: "It is not always easy to see who is related to whom - and in which ways. The following argument pertains to this question: Some classmate of Terri is a workmate of Dolores. Whoever is not a workmate of Dolores is an ancestor of Cheryl. So, necessarily, some ancestor of Cheryl is not a classmate of Terri."
Reasoning: Some classmate of Terri is not a workmate of Dolores. Whoever is not a workmate of Dolores is an ancestor of Cheryl. So, necessarily, some ancestor of Cheryl is not a classmate of Terri. Remember that Terri's classmate and Dolores's workmate can be in any case: One of the three can be any person. So, the answer is invalid.
Figure 6: The bootstrapped 3-shot Q-R-A soft prompt which is used in the formal_fallacies BBH experiments.
Bootstrapped 5-shot CoT prompt for GSM8K
Figure 7: The bootstrapped 5-shot Q-R-A soft prompt which is used in GSM8K experiments.