RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Figure 6: A ChatGPT-generated output of a failed negative monotonicity reasoning sample. The output was generated using ChatGPT Feb 13 Version; A ChatGPT-generated output of a failed negative monotonicity reasoning sample. The output was generated using ChatGPT Feb 13 Version
Figure 7: Zero-shot performance of OPT on MWR using different prompts; Zero-shot performance of OPT on MWR using different prompts
Figure 8: Zero-shot performance of OPT on SAR using different prompts; Zero-shot performance of OPT on SAR using different prompts
Figure 9: Zero-shot performance of OPT on MKR using different prompts; Zero-shot performance of OPT on MKR using different prompts
Figure 10: Zero-shot performance of OPT on NLI tasks using different prompts; Zero-shot performance of OPT on NLI tasks using different prompts
Summary of the negation-related benchmark datasets used in this paper.
Prompts used for each task
Mean perplexity (PPL) calculated using the GPT-J-6B model. Only the strings enclosed in square brackets are considered during calculation in order to provide a fair comparison with similar token length. For Corpus, PPL is calculated using the provided gold completion.
Example output of GPT-J-6B on MWR. bolded words are related to target words, but are neither synonyms nor antonyms. underlined are wrong antonyms but are not in the given set of wrong completions.
Zero-shot results on the different benchmarks. “NA” denotes that Negation prompts are not applicable to MKR-NQ and MWR. The best results are bolded for each task (row).
Example outputs of models. Wrong answers are highlighted