OpenBookQA has missing and inconsistent field names
State: closed | Author: vblagoje (CONTRIBUTOR)
Labels: name: dataset bug

## Describe the bug
OpenBookQA implementation is inconsistent with the original dataset.

We need to:

1. The dataset field [question][stem] is flattened into question_stem. Unflatten it to match the original format.
2. Add missing additional fields:
    - 'fact1': row['fact1'],
    - 'humanScore': row['humanScore'],
    - 'clarity': row['clarity'],
    - 'turkIdAnonymized': row['turkIdAnonymized']
3. Ensure the structure and every data item in the original OpenBookQA matches our OpenBookQA version.

## Expected results
The structure and every data item in the original OpenBookQA matches our OpenBookQA version.

## Actual results
TBD

## Environment info
- `datasets` version: 2.1.0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.13
- PyArrow version: 7.0.0
- Pandas version: 1.4.2

Comments: Thanks for reporting, @vblagoje.

Indeed, I noticed some of these issues while reviewing this PR:
- #4259 

This is in my TODO list. 

Ok, awesome @albertvillanova How about #4275 ?

On the other hand, I am not sure if we should always preserve the original nested structure. I think we should also consider other factors as convenience or consistency.

For example, other datasets also flatten "question.stem" into "question":
- ai2_arc:
  ```python
  question = data["question"]["stem"]
  choices = data["question"]["choices"]
  text_choices = [choice["text"] for choice in choices]
  label_choices = [choice["label"] for choice in choices]
  yield id_, {
      "id": id_,
      "answerKey": answerkey,
      "question": question,
      "choices": {"text": text_choices, "label": label_choices},
  }
  ```
- commonsense_qa:
  ```python
  question = data["question"]
  stem = question["stem"]
  yield id_, {
      "answerKey": answerkey,
      "question": stem,
      "choices": {"label": labels, "text": texts},
  }
  ```
- cos_e:
  ```python
  "question": cqa["question"]["stem"],
  ```
- qasc
- quartz
- wiqa

Exceptions:
- exams

I think we should agree on a CONVENIENT format for QA and use always CONSISTENTLY the same.

@albertvillanova I agree that we should be consistent. In the last month, I have come across tons of code that deals with OpenBookQA and CommonSenseQA and all of that code relies on the original data format structure. We can't expect users to adopt HF Datasets if we arbitrarily change the structure of the format just because we think something makes more sense. I am in that position now (downloading original data rather than using HF Datasets) and undoubtedly it hinders HF Datasets' widespread use and adoption. Missing fields like in the case of #4275 is definitely bad and not even up for a discussion IMHO! cc @lhoestq 

I'm opening a PR that adds the missing fields.

Let's agree on the feature structure: @lhoestq @mariosasko @polinaeterna 

IMO we should always try to preserve the original structure unless there is a good reason not to (and I don't see one in this case).

I agree with @mariosasko . The transition to the original format could be done in one PR for the next minor release, clearly documenting all dataset changes just as @albertvillanova outlined them above and perhaps even providing a per dataset util method to convert the new valid format to the old for backward compatibility. Users who relied on the old format will update their code with either the util method for a quick fix or slightly more elaborate for the new. 

I don't have a strong opinion on this, besides the fact that whatever decision we agree on, should be applied to all datasets.

There is always the tension between:
- preserving each dataset original structure (which has the advantage of not forcing users to learn other structure for the same dataset),
-  and on the other hand performing some kind of standardization/harmonization depending on the task (this has the advantage that once learnt, the same structure applies to all datasets; this has been done for e.g.  POS tagging: all datasets have been adapted to a certain "standard" structure).
   - Another advantage: datasets can easily be interchanged (or joined) to be used by the same model

Recently, in the BigScience BioMedical hackathon, they adopted a different approach:
- they implement a "source" config, respecting the original structure as much as possible
- they implement additional config for each task, with a "standard" nested structure per task, which is most useful for users.

@albertvillanova, thanks for the detailed answer and the new perspectives. I understand the friction for the best design approach much better now.  Ultimately, it is essential to include all the missing fields and the correct data first. Whatever approach is determined to be optimal is important but not as crucial once all the data is there, and users can create lambda functions to create whatever structure serves them best. 

Datasets are not tracked in this repository anymore. I think we must move this thread to the [discussions tab of the dataset](https://huggingface.co/datasets/openbookqa/discussions)

Indeed @osbm thanks. I'm closing this issue if it's fine for you all then