SlowGuess/NAACL_2022_OCR huggingface dataset:
- Wrong ordering: the order in the sequence of steps is not correct (e.g., wearing shoes is described before wearing socks).
- Flexible ordering: some steps can be done in a flexible order (e.g., you can wear left sock or right sock first). A good script captures such flexibility.
- Missing critical steps: a bad script might have missed critical steps (e.g., the script can say: "wait for a plane" followed by "get off the plane" - here an obvious step "get on the plane" is missing). There is no strict definition for a critical step, so the annotators were instructed to use their commonsense judgment.
- Wrong step: a bad script might have irrelevant and wrong steps (e.g., the script describing “go to a party” might describe an irrelevant step such as read a book, open a book, etc.).
For every data point, the annotators were asked to answer the following:
- Explicit feedback type-1: the error type (missing step, wrong step, wrong order, wrong partial order)
- Explicit feedback type-2: localize the error by providing the erroneous node or edge id
- Implicit feedback type-1: give feedback in a few words, explaining the error
- Implicit feedback type-2: An explanation of the error that would potentially make sense to a five-year-old. Such an explanation of the feedback helped gather the general principle that is violated, and is targeted in the feedback.
Fig. 6 shows a sample of our Mechanical Turk task. Annotators were required to list only one critical error that they believe was most important. Each data point is annotated by three annotators, adding some diversity in the errors. The annotators were paid $15 an hour. The annotators were English
speaking crowdworkers on Mechanical Turk from USA. The average time for completion of one script was 2 minutes.
fb type	count	example
explicit fb type-1	1,553	Remove node ‘put the shirt on’
explicit fb type-2	1,553	The following step is not right: put the shirt on
implicit fb type-1	1,553	It tells you to iron your shirt while it's still on your body.
implicit fb type-2	1,553	If you hold a hot iron against the clothes you're currently wearing, you'll get terrible burns.
total	6,212	https://anonymous.4open.science/r/interscript/data.json
Table 6: Dataset statistics. In this paper, we use the hardest feedback (implicit-feedback-type-2). This example is from the input script: input script for the following table (goal: press the wrinkles out) = 1. put the shirt on, 2. find place to press, 3. grab iron from drawer, 4. place iron on shirt, 5. wait for iron to heat up, 6. use iron to smooth out wrinkles, 7. press the wrinkles out
We measured the agreement on labels (which graph is worse), and on explicit feedback type-1 and type-2. It was difficult to measure agreement on implicit feedback because it is not easy to perform binary comparison on the generated text without accounting for linguistic variations. On the labels, the Fleiss Kappa agreement was 0.90 (almost perfect agreement) and on explicit feedback the agreement was 0.75 Fleiss Kappa (substantial agreement). This also shows that there is some diversity in what the users perceive as a serious mistake in  $y_{bad}$ .
Eventually, we compiled these annotations into a dataset of 1,553 4-tuples where each entry consists of (explicit feedback type-1, explicit feedback type-2, implicit feedback type-1, implicit feedback type-2).
8.3 Model Output Examples
Table 7 provides some sample model outputs.
9 Performance of FBNET by error type
Table 8 breaks down the performance of FBNET by error type.
x	fb	y*	y
1. ...2. set alarm for early morning3. get out of bed4. prepare for yoga5. go to the bathroom6. do yoga7. do yoga in the morning	People don't leave their alarms ring- ing all day.	insert node ‘wake up and turn off alarm’ before ‘get out of bed’	1. ...2. set alarm for early morning3. wake up and turn off alarm4. get out of bed5. ...6. ...7. ...
1. ...3. put on shoes ...5. open the door6. drive to the train station7. get into the car8. reach the train station	You must get into a vehicle, before driving to any place.	reorder edge be-tween ‘< drive to the train station, get into the car >’	1. ...3. ...5. open the door6. get into the car7. drive to the train station8. ...
1. ...3. pick up the butterfly4. put the butterfly in container5. look for a butterfly ...6. Take the butterfly home ...	You don't need to look for a butterfly if it's already in a container.	remove node ‘look for a butterfly’	1. ...3. pick up the butterfly4. put the butterfly in container5. Take the butterfly home6. ...
Table 7: Task: Applying the graph edit to the bad script.
Edit type	EM%
Overall	38.6
Add partial order exactmatch	10.5
Add partial order type	44.7
Missing step exactmatch	2.8
Missing step type	65.5
Remove partial order exactmatch	0.0
Remove partial order type	0.0
Wrong ordering exactmatch	45.1
Wrong ordering type	72.8
Wrong step exactmatch	63.0
Wrong step type	78.6
Table 8: Performance of FBNET by error type