Repository: hpcaitech/ColossalAI

Issue:
[BUG]: ColossalEval/AGIEvalDataset loader causing IndexError when few shot is disabled
### 🐛 Describe the bug

## Describe the bug

AGIEvalDataset loader in ColossalEval incorrectly set `few_shot_data` to `[]` when `few_shot` is disabled, causing IndexError at https://github.com/hpcaitech/ColossalAI/blob/main/applications/ColossalEval/colossal_eval/utils/conversation.py#L126.

## To Reproduce

Use the following inference config.json, to run https://github.com/hpcaitech/ColossalAI/blob/main/applications/ColossalEval/examples/dataset_evaluation/inference.sh

```json
{
  "model": [
    {
      "name": "<model name>",
      "model_class": "HuggingFaceCausalLM",
      "parameters": {
        "path": "<model>",
        "model_max_length": 2048,
        "tokenizer_path": "<tokenizer>",
        "tokenizer_kwargs": {
          "trust_remote_code": true
        },
        "peft_path": null,
        "model_kwargs": {
          "torch_dtype": "torch.float16",
          "trust_remote_code": true
        },
        "prompt_template": "plain",
        "batch_size": 4
      }
    }
  ],
  "dataset": [
    {
      "name": "agieval",
      "dataset_class": "AGIEvalDataset",
      "debug": false,
      "few_shot": false,
      "path": "eval-data/AGIEval/data/v1",
      "save_path": "inference_data/agieval.json"
    }
  ]
}
```

The following exception will occur.

```bash
agieval-aqua-rat Inference steps:   0%|               | 0/64 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/ColossalAI/applications/ColossalEval/examples/dataset_evaluation/inference.py", line 260, in <module>                             
    main(args)                                                               
  File "/ColossalAI/applications/ColossalEval/examples/dataset_evaluation/inference.py", line 223, in main                                 
    answers_per_rank = model_.inference(                                     
                       ^^^^^^^^^^^^^^^^^                                                                                                                   
  File "/usr/local/lib/python3.11/dist-packages/colossal_eval/models/huggingface.py", line 373, in inference
    batch_prompt, batch_target = get_batch_prompt(                                                                                                         
                                 ^^^^^^^^^^^^^^^^^                                                                                                         
  File "/usr/local/lib/python3.11/dist-packages/colossal_eval/utils/conversation.py", line 195, in get_batch_prompt   
    few_shot_prefix = get_few_shot_prefix(conv, few_shot_data, tokenizer, language, max_tokens)           
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/colossal_eval/utils/conversation.py", line 141, in get_few_shot_prefix
    few_shot_prefix = few_shot_data[0] + "\n\n"                                                                                                            
                      ~~~~~~~~~~~~~^^^                           
IndexError: list index out of range 
```

## Expected behavior

With `few_shot` set to false, `few_shot_data` should be set to None, so `get_batch_prompt`(https://github.com/hpcaitech/ColossalAI/blob/main/applications/ColossalEval/colossal_eval/utils/conversation.py#L182) will skip few shot prefix generation.

### Environment

Python 3.11.0rc1


Discussion:

Resolving patch:
diff --git a/applications/ColossalEval/colossal_eval/dataset/agieval.py b/applications/ColossalEval/colossal_eval/dataset/agieval.py
--- a/applications/ColossalEval/colossal_eval/dataset/agieval.py
+++ b/applications/ColossalEval/colossal_eval/dataset/agieval.py
@@ -201,7 +201,7 @@ def load(
         for file in files:
             dataset_name = os.path.basename(file)[0 : -len(".jsonl")]
 
-            few_shot_data = []
+            few_shot_data = None
             if few_shot:
                 # process demo once if it is few-shot-CoT
                 few_shot_data = combine_prompt(prompt_path, dataset_name, load_explanation=False, chat_mode=False)


Test patch: