Venue: NAACL 2024

# Order-Based Pre-training Strategies for Procedural Text Understanding

Abhilash Nandy  Yash Kulkarni   Pawan Goyal Niloy Ganguly

nandyabhilash@kgpian.iitkgp.ac.in

Indian Institute of Technology Kharagpur

India

###### Abstract

In this paper, we propose sequence-based pre-training methods to enhance procedural understanding in natural language processing. Procedural text, containing sequential instructions to accomplish a task, is difficult to understand due to the changing attributes of entities in the context. We focus on recipes, which are commonly represented as ordered instructions, and use this order as a supervision signal. Our work is one of the first to compare several 'order-as-supervision' transformer pre-training methods, including Permutation Classification, Embedding Regression, and Skip-Clip, and shows that these methods give improved results compared to the baselines and SoTA LLMs on two downstream Entity-Tracking datasets: NPNCooking dataset in recipe domain and ProPara dataset in open domain. Our proposed methods address the non-trivial Entity Tracking Task that requires prediction of entity states across procedure steps, which requires understanding the order of steps. These methods show an improvement over the best baseline by \(1.6\%\) and \(7\text{-}9\%\) on NPN-Cooking and ProPara Datasets respectively across metrics.1

Footnote 1: Code is available at [https://github.com/](https://github.com/) abhil1nandy2/Order_As_Supervision

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 821–828 June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

Procedural text comprises a series of sequential instructions aimed at guiding individuals through a task by presenting information in a step-by-step manner. A procedure describes a step-wise interaction between multiple participating entities and their attribute changes. For instance, "Photosynthesis" as a procedure consists of interaction between entities such as water, light, CO2, sugar, etc. Recently, there has been an increase in the number of studies in NLP that use procedural texts. Procedural text is common in natural language in recipes Marin et al. (2018a); Bien et al. (2020a); Chandu et al. (2019); Majumder et al. (2019); Bosselut et al. (2017), how-to guides Nandy et al. (2021), and scientific processes Mishra et al. (2018). In this study, we focus on recipes as they are commonly represented as ordered instructions. We utilize this order as a supervision signal to develop customized pre-training techniques to solve non-trivial tasks that require anticipating the implicit effects of actions on entities.

Understanding procedural text is difficult due to the changing attributes of entities in the context. Previous works such as Lee et al. (2020) used Sentence-level Language Modeling (SLM) to learn contextualized sentence-level representation by training a hierarchical transformer to reconstruct the original order of a shuffled sequence Tang et al. (2020) proposed Interactive Entity Network (IEN) to model different types of entity interactions using a recurrent network with memory for state tracking, and Zhang et al. (2021) combined external knowledge with a BERT model to improve entity tracking. However, such works do not compare pre-training techniques which consider sequential order of the steps of the procedure.

In this paper, we try to solve the non-trivial Entity Tracking Task that requires prediction of entity states across procedure steps. Solving such a task requires understanding the _sequential nature/order of the steps_. Explicitly learning the order within data has been shown to enhance performance of tasks such as Video Representation Learning, solving Jigsaw Puzzles to learn image representations Noroozi and Favaro (2016), etc. Similarly, ALBERT Lan et al. (2020) shows that sentence-order prediction between two sentences is a useful pre-training objective to improve performance on various downstream NLP tasks. Inspired by such works, our work is one of the first to introduce and compare several novel 'order-as-supervision' pre-training methods such as Permutation Classification, Skip-Clip, and Embedding Regression to enhance procedural understanding.



These pre-training methods give a significant improvement of \(1.6\%\) and \(7-9\%\) compared to baselines across metrics on two downstream Entity-Tracking datasets, namely, NPN-Cooking dataset Bosselut et al. (2017) in the recipe domain and ProPara dataset Mishra et al. (2018) in the open domain. Our methods also outperform SoTA LLMs in terms of Average Accuracy on ProPara.

## 2 Pre-training Methods

We propose three new pre-training methods that help in learning sequential context for procedural texts: Permutation Classification, Embedding Regression, and Skip-Clip. For all the methods, a set of recipes with the same number of steps are sampled. Such a recipe of \(N\) steps can be represented by \(x=(x_{1},x_{2},...,x_{N})\).

### Permutation Classification

In this method, the original recipe is shuffled by permuting its steps by some index permutation \(\psi_{i}=(\psi_{i1},\psi_{i2},...,\psi_{iN})\). The set of all possible permutations \(\psi^{*}\) contains \(N!\) elements. If, for example, \(N\) = 9 the total number of possible permutations equals 9! = 362,880. For practical reasons, as a pre-processing step, we reduce the set of all possible permutations by sampling a set \(\psi\) of maximally diverse permutations from \(\psi^{*}\). Following Noroozi and Favaro (2016), we iteratively include the permutation with the maximum Hamming distance to the already chosen ones. For every recipe, we select a random permutation from this set and assign its index as a label. To solve the permutation classification task, we input the permuted sequence into a transformer and use the \(
\) (classification token) embedding to perform sequence multi-class classification. The number of output classes is equal to the size of the permutation set. Figure 1 shows the Permutation Classification Architecture.

### Embedding Regression

Following Korba et al. (2018), we modify the permutation classification method defined above. Thus, instead of predicting the index, we convert the permutation into an embedding vector and perform a regression task on this embedding. We experiment with 2 different embedding constructions, considering \(\psi\) as a permutation of length \(N\) - Hamming and Lehmer Embedding. Hamming Embedding (\(h\)) is a vector of size \(N^{2}\) formed by concatenating one-hot vectors for each value of the permutation - \(h_{N,i+j}=I\{\psi(i)=j\}\), where \(I\) is the Identity Function. Lehmer Embedding (\(l\)) is a vector of size \(N\), where the value at \(i^{th}\) index is the number of indices less than \(i\) with a greater permutation value - \(l_{i}=\#\{j:j<i,\psi(j)>\psi(i)\}_{1\leq i\leq N}\). E.g. for the permutation (4,3,1,2) the Hamming Embedding is (0,0,0,1,0,0,1,0,0,0,0,1,0,0) and Lehmer Embedding is (0,1,2,2). We use Mean Squared Error (MSE) as the loss function. It can be theoretically shown that minimizing this loss on the selected embeddings is equivalent to optimizing a ranking metric like Kendall's Tau Kendall (1938) or Hamming Hamming Hamming (1950) distance. Figure 1 shows the Embedding Regression Architecture.

### Skip-Clip

In this method inspired by El-Nouby et al. (2019), given the first few steps as the context, other steps closer to the context have more similar representations to that of the context than the ones that are farther. Here, we sample the first \(K\) steps of a recipe with \(N\) steps, as the context \(c=(x_{1},x_{2},...,x_{K})\) and randomly sample \(M\) target steps \((x_{t_{1}},x_{t_{2}},...,x_{t_{M}})\), where \(t_{i}\) is the index in the original recipe for the \(i^{th}\) target step, with \(M,K<N\) and \(t_{1}>K\). Using transformer model \(f\), we get latent representations of the context \(h=f(c)\) and each step in the target, \(z_{i}=f(x_{t_{i}})\). We also define a scoring function, \(\Gamma(h,z_{i})\), e.g. cosine similarity, representing the relationship between the context and the target steps. The objective is hinge rank loss formulated as: \(L=\sum_{i=1}^{M-1}\sum_{j=i+1}^{M}max(0,-\Gamma(h,z_{i})+\Gamma(h,z_{j})+\delta)\), where the constant \(\delta\) is the margin. Figure 2 shows the Skip-Clip architecture.

## 3 Downstream tasks

Entity Tracking consists of two sub-tasks - entity state and location tracking. Both tasks output the result for every entity at each step of the process. Entity state tracking task is a 4-way classification task that predicts if the entity is created, moved, unchanged, or destroyed at that step of the process. Entity Location tracking is formulated as a span-based question-answering problem that outputs the location of the entity at a particular step taking the entire text of the process as the input. We perform this task on NPN-Cooking and ProPara datasets2. NPN-Cooking dataset Bosseltu et al. (2017) consists of 65,816 training, 175 development, and 700 evaluation recipes. ProPara Dataset Mishra et al. ( 

2018) consists of 488 human-authored procedures (split 80/10/10 into train/dev/test) with 81k annotations regarding changing states (existence and location) of entities in those paragraphs. The model is evaluated in 3 ways corresponding to a given entity \(e\). **Category 1**: which of the three transitions - created, destroyed, or moved undergone by \(e\) over the lifetime of a procedure; **Category 2**: steps at which \(e\) is created, destroyed, and/or moved; and **Category 3**: the location (span of text) which indicates \(e\)'s creation, destruction or movement. Following Faghihi and Kordjamshidi (2021), Entity Tracking is formulated as a question-answering problem. Fine-tuning Hyperparameters are the same as in the default open-source implementation3.

Footnote 3: [https://github.com/HLR/TSLM](https://github.com/HLR/TSLM)

## 4 Experiments and Results

### Pre-training Setup

**Parameter Initialization:** For fast convergence, we initialize the transformer in each pre-training method with RoBERTa-BASE Liu et al. (2019)4. **Dataset:** We use a dataset of 2.5 million+ recipes in total collected from various different sources on the internet such as Recipe1M+ dataset Marin et al. (2018b), RecipeNLG dataset Bien et al. (2020b), datasets collected by Majumder et al. (2019) and Chandu et al. (2019). For each recipe in the dataset, a sentence with the ingredients is also added as a step before the original recipe. The dataset is filtered to include recipes with more than 4 steps. The statistics of the dataset is shown in Table 5_in Section D.1 of Appendix_. Permutation Classification and Embedding Regression require all recipes to have the same number of steps per recipe. Hence, we use a subset of recipes that have a certain, fixed number of steps. **Hyperparameters:** We pre-trained for 1 epoch using AdamW optimizer with batch size of 32, learning rate of 5e-5, weight decay of 0.01, and 500 warmup steps.

[TABLE:S4.T1][ENDTABL

Figure 1: Permutation Classification and Embedding Regression for a 4-step recipe. Recipe steps are reordered via a randomly chosen permutation from a predefined permutation set and then fed to the transformer model. The Permutation Classification Task is to predict the index of the chosen permutation which in this case is 23, and Embedding Regression Task is to predict the corresponding Lehmer/Hamming Embedding.

\begin{table}
\begin{tabular}{|l|r|c|} \hline
* Model &
- & 51.3 \\ KG-MRC &
- & 51.6 \\ DYNAPO &
- & 62.9 \\ RoBERTa-BASE & 65.07 & 64.28 \\ \hline \(Permutation\ Classfn.\) & **65.48** & 64.75 \\ \hline \(Emplanning\) & 65.03 & 64.92 \\ \(Embt_{charn}\) & 63.96 & 64.29 \\ \hline \(Skip\text{-}Clip\) & 63.87 & **65.33** \\ \hline \end{tabular}
\end{table}
Table 1: Results on NPN-Cooking Dataset. Numbers in bold and underlined are the highest and the second-highest scores, respectively.

Figure 2: Skip-Clip model with a 6-step context and 3 target steps. The task is to rank the target steps based on scores obtained from a scoring function and their order in the recipe using hinge rank loss.

 



### Fine-tuning

We fine-tune and evaluate models pre-trained using techniques mentioned in Section 2 on ProPara and NPN-Cooking Datasets. Note that Embedding Regression has two variants based on the type of embedding used - Hamming Embedding (\(Emb_{Hamming}\)) and Lehmer Embedding (\(Emb_{Lehmer}\)). We use hyperparameter grid search on development sets corresponding to each pretraining variant to get the best set of hyperparameters, as mentioned _in Section D.2 of Appendix_.

### Baselines

**NPN-Cooking Dataset:** We use Neural Process Network Model (**NPN-Model**) Bosselut et al. (2017), **KG-MRC** Das et al. (2018), **DYNAPRO**Amini et al. (2020), and RoBERTa-BASE Liu et al. (2019) as baselines.

**ProPara Dataset:** We use a **rule-based** method called **ProComp**Clark et al. (2018), a **feature-based** method Mishra et al. (2018) using Logistic Regression and CRF, **ProLocal**Mishra et al. (2018), **ProGlobal**Mishra et al. (2018), **EntNet**Henaff et al. (2016), **QRN**Seo et al. (2016), and RoBERTa-BASE Liu et al. (2019) as baselines.

### Analysis of Results

Table 1 shows results of proposed methods and baselines on NPN-Cooking Dataset. We see that all proposed methods perform better than baselines w.r.t Test Accuracy. Permutation Classification gives the best dev set result, but falls behind on the test set, as classification on \(100\) classes leads to overfitting. Skip-Clip gives best test accuracy, with an improvement of \(1.6\%\) compared to RoBERTa-BASE, suggesting that predicting next step from a given context helps in Entity Tracking in Recipe Domain. \(Emb_{Hamming}\) gives the second-highest test accuracy, showing that predicting permutation as an embedding is useful for this task.

Table 2 shows results of proposed methods and baselines on ProPara. RoBERTa-BASE is the best baseline. Most proposed methods beat baselines. Skip-Clip does not perform as well, suggesting that this pre-training method of predicting a future step in recipes does not transfer to open domain. Permutation Classification and Embedding Regression perform much better. \(Emb_{Lehmer}\) performs better on 5 out of 6 metrics compared to \(Emb_{Hamming}\). Permutation Classification has the best Status Accuracy and Average Category Score and gives an improvement of \(7.4\%\) and \(9\%\) respectively compared to RoBERTa-BASE, showing that predicting a permutation helps in a task in another domain.

* (1) Open-source LLMs such as Falcon-7B-Instruct (instruction-fine-tuned Falcon-7B) Penedo et al. (2023), Llama 2-7B-Chat (instruction-fine-tuned Llama 2-7B) Touvron et al. (2023) (2) OpenAI's GPT-3.5 OpenAI (2021). The LLMs are used in a 1-shot and 3-shot In-Context Learning Setting Dong et al. (2022). Table 8 shows that
- (1) even though Falcon-7B-Instruct and Llama 2-7B-ChaT have almost 14x the number of parameters compared to the proposed permutation-based methods, they perform considerably worse in comparison (2) the proposed methods outperform GPT-3.5 in 1-shot setting across all metrics, and GPT-3.5 in 3-shot setting across 3 out of 4 metrics, even though the number of parameters and pre-training data is just a small fraction of that of GPT-3.5.

Additionally, we compare predictions of Permutation Classification and well-performing baseline RoBERTa-BASE on a procedure _in Table 9 in Section D.4 of Appendix_. We infer that Permutation Classification is able to better predict the step when an entity ceases to exist, compared to the baseline.

## 5 Combination of different pre-training strategies

In this section, we explore sequential combinations of pre-training strategies. As Permutation Classification performs consistently well, we experiment with one of either Skip-Clip, \(Emb_{Hamming}\), or \(Emb_{Lehmer}\), followed by Permutation Classification.

Tables 3 and 4 reveal combinations of pre

training strategies being inferior to using individual strategies, possibly because these strategies use different supervision cues. For instance, while Permutation Classification treats each permutation as an independent target class, Skip-Clip pushes representations of nearby steps closer, and vice versa. Hence, Skip-Clip + Permutation Classification performs poorly. \(Emb_{Lehmer}\), unlike Permutation Classification, uses distance between each step before and after permuting as target encoding, hence, different permutations are not independent, making methods slightly inconsistent. \(Emb_{Hamming}\), like Permutation Classification, has different targets for each permutation, but has a larger target vector than PC. Hence, \(Emb_{Hamming}\) + Permutation Classification is reasonably good, but is inferior to each.

## 6 Conclusion

Our work is one of the first to propose order-based in-domain pre-training for procedural data to enhance Entity Tracking performance. We introduce 3 pre-training tasks - Permutation Classification, Embedding Regression, Skip-Clip to learn sequential nature of procedures. Skip-Clip performs best on the in-domain NPN-Cooking Task, while Permutation Classification and Embedding Regression perform best on the open-domain ProPara Task. We believe such methods could be extended to procedures in E-Manuals, manufacturing guides, etc.

Limitations
1. Our work focuses on recipes as a type of procedural text. It would require further study to see if the results can be generalized to other types of procedural text such as science processes or how-to guides.
2. The Entity Tracking Task is only one aspect of understanding procedural text. Other aspects such as identifying entities and their attributes, and understanding causal relationships between entities may also be important for some applications.
3. Our work evaluates the proposed methods on two specific datasets, which may not be representative of all possible scenarios. The performance of the methods on other datasets or real-world applications may vary.

### Ethics Statement

The proposed methodology can be used for any type of procedural text, including user-generated procedures. However, before applying the model to such procedures, it is important to consider exposure bias patterns. Additionally, the interpretability of the model's output is limited, so users should exercise caution when using it.

## References

* A. Amini, A. Bosselut, B. Dalvi Mishra, Y. Choi, and H. Hajishirzi (2020)Procedural reading comprehension with attribute-aware context flow. External Links: Link Cited by: SS4.
* M. Bien, M. Gilski, M. Maciejewska, W. Taisner, D. Wisniewski, and Agnieszka Lawrynowicz (2020)RecipeNLG: a cooking recipes dataset for semi-structured text generation. In Proceedings of the 13th International Conference on Natural Language Generation, Dublin, Ireland, pp. 22-28. External Links: ISBN 978-3-540-3861-1, Document Cited by: SS1, SS2.
* M. Bien, M. Gilski, M. Maciejewska, W. Taisner, D. Wisniewski, and A. Lazwrynowicz (2020)RecipeNLG: a cooking recipes dataset for semi-structured text generation. In Proceedings of the 13th International Conference on Natural Language Generation, Dublin, Ireland, pp. 22-28. External Links: ISBN 978-3-540-3861-1, Document Cited by: SS1, SS2.
* Antoine Bosselt, O. Levy, A. Holtzman, C. Ennis, P. Fox, and Y. Choi (2017)Simulating action dynamics with neural process networks.

\begin{table}
\begin{tabular}{|l|r|r|r|r|} \hline  & **Catl** & **Cat2** & **Catd3** \\  & **Ace.** & **Acc. Acc. Acc.

 

Khyathi Raghavi Chandu, Eric Nyberg, and Alan W. Black. 2019. Storyboarding of recipes: Grounded contextual generation. In _ACL_.
* Clark et al. (2018) Peter Clark, Bhavana Dalvi, and Niket Tandon. 2018. What happened? leveraging verbnet to predict the effects of actions in procedural text.
* Das et al. (2018) Rajarshi Das, Tsendsuren Munkhdalai, Xingdi Yuan, Adam Trischler, and Andrew McCallum. 2018. Building dynamic knowledge graphs from text using machine reading comprehension.
* Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. _arXiv preprint arXiv:2301.00234_.
* El-Nouby et al. (2019) Alaaeldin El-Nouby, Shuangfei Zhai, Graham W Taylor, and Joshua M Susskind. 2019. Skip-clip: Self-supervised spatiotemporal representation learning by future clip order ranking. _arXiv preprint arXiv:1910.12770_.
* Faghihi and Kordjamshidi (2021) Hossein Rajaby Faghihi and Parisa Kordjamshidi. 2021. Time-stamped language model: Teaching language models to understand the flow of events.
* Hamming (1950) R. W. Hamming. 1950. Error detecting and error correcting codes. _The Bell System Technical Journal_, 29(2):147-160.
* Henaff et al. (2016) Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2016. Tracking the world state with recurrent entity networks.
* Kendall (1938) M. G. Kendall. 1938. A new measure of rank correlation. _Biometrika_, 30(1/2):81-93.
* Korba et al. (2018) Anna Korba, Alexandre Garcia, and Florence d'Alche Buc. 2018. A structured prediction approach for label ranking. In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_, NIPS'18, page 9008-9018, Red Hook, NY, USA. Curran Associates Inc.
* Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In _International Conference on Learning Representations_.
* Lee et al. (2020) Haejun Lee, Drew A. Hudson, Kangwook Lee, and Christopher D. Manning. 2020. Slm: Learning a discourse language representation with sentence un shuffling.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
* Majumder et al. (2019) Bodhisattwas Prasad Majumder, Shuyang Li, Jianmo Ni, and Julian McAuley. 2019. Generating personalized recipes from historical user preferences.
* Marin et al. (2018a) Javier Marin, Aritro Biswas, Ferdna Ofli, Nicholas Hynes, Amaia Salvador, Yusuf Avtar, Ingmar Weber, and Antonio Torralba. 2018a. RecipeIm+: A dataset for learning cross-modal embeddings for cooking recipes and food images.
* Marin et al. (2018b) Bhaveau Dalvi Mishra, Liftu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018b. Tracking state changes in procedural text: A challenge dataset and models for process paragraph comprehension.
* Nandy et al. (2021) Abhilash Nandy, Soumya Sharma, Shubham Maddhashiya, Kapil Sachdeva, Pawan Goyal, and Niloy Ganguly. 2021. Question answering over electronic devices: A new benchmark dataset and a multi-task learning based QA framework. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 4600-4609, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Noroozi and Favaro (2016) Mehdi Noroozi and Paolo Favaro. 2016. Unsupervised learning of visual representations by solving jigsaw puzzles. In _Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI_, pages 69-84. Springer.
* OpenAI (2021) OpenAI. 2021. Gpt-3.5 turbo documentation.
* Penedo et al. (2023) Guilherme Penedo, Quentin Malaric, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesar Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcton Illr: outperforming curated corpora with web data, and web data only. _arXiv preprint arXiv:2306.01116_.
* Seo et al. (2016) Minjeon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Query-reduction networks for question answering.
* Tang et al. (2020) Jizhi Tang, Yansong Feng, and Dongyan Zhao. 2020. Understanding procedural text using interactive entity networks. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7281-7290, Online. Association for Computational Linguistics.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Zhang et al. (2021) Zhihan Zhang, Xubo Geng, Tao Qin, Yunfang Wu, and Davin Jiang. 2021. Knowledge-aware procedural text understanding with multi-stage training. In _Proceedings of the Web Conference 2021_. ACM 



## Appendix A Introduction

### Pre-training Methods

C Downstream Tasks

D Experiments and Results

### Pre-training Setup

**Compute Details:** Number of trainable parameters for each pre-training method is 488,126,475. We use Tesla V100 GPUs for our experiments. Permutation Classification and Embedding Regression Methods take about 24 GPU-Hours, while Skip-Clip takes about 8 GPU-Hours (GPU-Hours is the number of GPUs used multiplied by the training time in hours).

**Pre-training Data:** The statistics of Pre-training Data is elaborated in Table 5.

### Fine-tuning

The set of hyperparameters used for performing Grid Search are mentioned in Tables 6 and 7.

The best set of hyperparameters obtained are as follows - (1) **Permutation Classification:** No. of recipe steps = 6, Size of permutation set = 100 (2) **Embedding Regression:** No. of recipe steps = 6, Size of permutation set = 50 (3) **Skip-Clip:** No. of steps used as input context = 4, No. of target steps = 4.

### Baselines

#### Analysis of Results

**Comparison with LLMs:** Table 8 compares performance of our proposed methods with that of LLMs in 1 and 3-shot In-Context Learning setting.

Table 9 shows annotated ground truth, predictions of Permutation Classification, and well-performing baseline RoBERTa-BASE for an entity in the procedure.

\begin{table}
\begin{tabular}{|c|l|} \hline
**Hyperparameter** & **Set of values** \\ \hline Number of recipe steps & \multicolumn{1}{r|}{\{4, 6, 9\} \\ \hline Size of the permutation set & \multicolumn{1}{r|}{\{2, 10, 50, 100\}} \\ \hline \end{tabular}
\end{table}
Table 6: Set of hyperparameters used for grid search for Permutation Classification and Embedding Regression

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt}|} \hline  & **Cat1 Acc.** & **Cat2 Acc.** & **Cat3 Acc.** & **Avg. Cat Acc.** \\ \hline Falcon-7B-Instruct (1-shot) & 50.42 & 5.42 & 0.38 & 18.74 \\ Falcon-7B-Instruct (3-shot) & 48.44 & 3.15 & 1.94 & 17.84 \\ Llama 2-7B-Chat (1-shot) & 47.88 & 9.74 & 6.44 & 21.35 \\ Lima 2-7B-Chat (3-shot) & 51.27 & 13.98 & 11.97 & 25.74 \\ GPT-3.5 (1-shot) & 53.25 & 24.66 & 11.37 & 29.76 \\ GPT-3.5 (3-shot) & 62.43 & 34.66 & 15.81 & 37.63 \\ \hline \(Permutation\ Classfn.\) & **73.72****q3.16** & 32.72 & **49.87** \\ \hline \(Emb_{harmoning}\) & 68.5 & 30.43 & **36.16** & 45.03 \\ \(Emb_{ebmmer}\) & 72.33 & 38.82 & 35.05 & 49.06 \\ \hline \(Skip\text{-}Clip\) & 66.94 & 34.46 & 32.73 & 44.71 \\ \hline \end{tabular}
\end{table}
Table 8: Results on the ProPara Dataset - LLMs vs. proposed permutation-based methods

\begin{table}
\begin{tabular}{|p{113.8pt}|p{113.8pt} p{113.8pt}|} \hline
**Dataset** & **No. of Recipes** & **No. of words (only step)** & **No. of words (only ingredients)** \\ \hline
**RecipeM+** & 1,027,20 & 137,364,594 & 54,523,129 \\ \hline
**ReRepresentLG** & 1,643,978 & 147,281,377 & 1,763,585 \\ \hline
**Melmore et al. (2019)** & 179,217 & 23,774,704 & 3,834,978 \\ \hline
**Chaudo et al. (2019)** & 33,720 & 26,243,714 & \\ \hline
**Total** & 2,885,755 & 334,664,389 & 132,014,485 \\ \hline \end{tabular}
\end{table}
Table 5: Statistics of Pre-training Data



\begin{table}
\begin{tabular}{l|c|c|c} \hline  & Ground Truth & Permutation Classification & RoBERTa-BASE \\ \hline Procedure & flower & flower & flower \\ \hline (Before the process starts) & - & - & - \\
1. A seed is planted. & - & - & - \\
2. It becomes a seedling. & - & - & - \\
3. It grows into a tree. & - & - & - \\
4. The tree grows flowers. & tree & tree & tree \\
5. The flowers become fruit. & - & - & tree \\
6. The fruits contain seeds & - & - & tree \\ for new trees. & - & - & tree \\ \hline \end{tabular}
\end{table}
Table 9: Analysis of the ground truth and the predictions of Permutation Classification vs. a well-performing baseline on a sample from the ProPara Dataset.