$\textbf{\emph{CLMSM}}$: A Multi-Task Learning Framework for Pre-training on Procedural Text
Authors: Abhilash Nandy
Manav Nitin Kapadnis
Pawan Goyal
Niloy Ganguly
Venue: EMNLP 2023 Findings (2023)
Keywords: pre-training
procedural reasoning
contrastive learning
masked language modeling
multi-task learning
nlp
TL;DR: We propose CLMSM, a pre-training framework that uses multi-task learning with contrastive learning and mask-step modeling objectives for downstream procedural NLP tasks, and outperforms baselines on in-domain as well as open-domain tasks.

Abstract: In this paper, we propose ***CLMSM***, a domain-specific, continual pre-training framework, that learns from a large set of procedural recipes. ***CLMSM*** uses a Multi-Task Learning Framework to optimize two objectives - a) Contrastive Learning using hard triplets to learn fine-grained differences across entities in the procedures, and b) a novel Mask-Step Modelling objective to learn step-wise context of a procedure. We test the performance of ***CLMSM*** on the downstream tasks of tracking entities and aligning actions between two procedures on three datasets, one of which is an open-domain dataset not conforming with the pre-training dataset. We show that ***CLMSM*** not only outperforms baselines on recipes (in-domain) but is also able to generalize to open-domain procedural NLP tasks.

CLMSM: A Hulti-Task Learning Framework for Pre-training on Procedural Text
Abhilash Nandy* Manav Nitin Kapadnis"
Pawan Goyal
Niloy Ganguly
nandyabhi hawyybbbiasshkkgpian.... ii tkgp. ac. in Indian Institute of Technology Kharagpur India
iammanavk@gmail com
Abstract
the effects of pre-training to solve such procedural NLP tasks thoroughly.
In this paper, we propose CLMsM, a domain- specific, continual pre-training framework, that learns from large set of procedural recipes. CLMSM uses a Multi-Task Learning Frame- work to optimize two objectives a) Con- trastive Learning using hard triplets to learn fine-griined differences across entities in the procedures, and b) a novel Mask-Step Mod- elling objective to learn step-wise context of a procedure. We test the performance of CLMSM on the downstream tasks of tracking entities and aligning actions between two pro- cedures on three datasets, one of which is an open-domain dataset conforming with the pre-traiee pre-training dataset. We show that CLMSM not only outperforms baselines on recipes (in- domain) but is also able generalize to open- domain procedural NLP tasks.
Current self-supervised pre-training techniques such as Masked Language Modeling (MLM) (Liu et al., 2019; Devlin et al., 2018), Next Sentence Pre- diction (NSP) (Devlin et al., 201 8), text denoising (Raffel et al., 2020; Lewis et al., 2019), etc. have been shown perform well on downstream NLP tasks such as Question Answering (QA), Named Entity Recognition (NER), Natural Language Infer- ence (NLI), etc. While such methods could capture local contexts proficiently (Lai et al., 2020; Sun et al., 2021), these methods fail in capturing pro- cedural, deep-wise contexts required for tracking entities and aligning actions in procedural texts (Mishra et al., 2018; Dalvi et al., 2019) as the learn- ing objectives do not consider the context of the entire procedure, and the interaction between enti- ties and actions.
1 Introduction
To address these challenges, in this puper, we propose CLMSM which is a combination of Con- trastive Learning (CL) using domain-speciffe meta- data, and a self-supervised Masked Step Modeling (MSM) to perform domain-specific continual pre- training for procedural text. We use CL with hard triplets (Cohan et al., 2020; Ostendorff et al., 2022) which enables learning fine- sociated with participating entities and actions, by pulling representations of highly similar procedures closer and pushing representations of slightly dis- similar procedures further away from each other. Prior works on instructional video understanding such as Narasimhan et al. (2023) have also shown to benefit from the of MSM in learning step- wise representations capture global context of the entire video. Inspired from such works, we incor- porate MSM for procedural text. MSM considers a step as a single entity and learns step-wise context, as it predicts tokens of randomly masked step(s), given the other steps of the procedure. We pre-train
Procedural text consists of a of instructions that provide information step-by-step format to ac- complish a task. Procedural text can be found in dif- ferent domains such as recipes (Marin et al., 2018; Bień et al., 2020; Chandu et al., 2019; Majumder et al., 2019; Bosselut et al., 2017), E-Manuals (Nandy et al., 2021), scientific processes (Mishra al., 2018), etc. Procedural reasoning requires un- derstanding the interaction between participating entities and actions associated with such entities in procedures. Recent works have made attempts at tracking entities (Mishra et al., 2018; Dalvi et al., 2019; Faghihi and Kordjamshidi, 2021) and align- ing actions (Donatelli et al., 2021) across the dif- ferent steps of procedures to evaluate procedural reasoning. The performance of these tasks can pos- sibly be enhanced by specialized pre-training on large procedural corpora, such that the pre-training objectives are aligned with the downstream tasks at hand. However, none of the prior art has studied
fine-grainen grained contexts as-
Equual contribution.
'Contrastive Learning cum Masked Step Modeling

---

our model in the recipe domain, as recipes pro- vide step-wise instructions on how to make a dish, making them an ideal candidate for pre-training.
2.1 Contrastive Learning using procedure similarity (CL)
As shown in Figure 1, we use a Triplet Network (Cohan et al., 2020; Ostendorfi et al., 2022), where three procedures serve as inputs for three bidirec- tional encoders (such as BERT (Devlin al., 201 8), RoBERTa (Liu et al., 2019)), the first (Anchor) and the second (Positive) procedures being simi- lar, and the first (Anchor) and the third (Negative) procedures being dissimilar (based on metadata). Note that we use hard triplets to learn fine-grained similarities and differences in entities and actions across procedures. The encoders have hard param- eter sharing (Caruana, 1993). The three encoded representations used to formulate a triplet mar- gin loss function, denoted by Lt. Mathematically,
We experiment with various downstream proce- dural reasoning tasks that require understanding of the changes in states of entities across steps of the procedure, triggered due to actions carried out in each step. The first task of tracking the move- ment of constituent entities through procedure is carried out on the NPN-Cooking Dataset (Bosse- lut al., 2017) of the recipe domain, and ProPara Dataset (Mishra et al., 2018) of the open-domain (to evaluate the generalizability of domain sppccifc pre-training), while the second task is to align ac- tions across different recipes of the ARA dataset (Donatelli et al., 2021). We show that CLMSM performs better than several baselines when eval- uated on domain-specific as well as open-domain downstream tasks. Specifically, CLMSM provides performance improvements of 1.28%, 2.56%, and 4.28% on the NPN-Cooking, ProPara, and ARA action alignment datasets, respectively. Our exten- sive ablation study suggests that a) the proposed multi-task learning framework outperforms using any one of the two objectives using these ob- jectives sequentially, b) masking an entire step is much more effective than masking random span, and c) the use of hard triplets during CL is much more effective than casy triplets.
f = (our(d(P(,PP)) P3 + 1,0 (1)
where P1, P2, P3 refer to the representations of the three procedures, and d(...) represents the L2 norm distance.
2.2 Mask-Step Modeling (MSM)
Here, we formulate a self-supervised multiple- masked token prediction task. Given procedure (a procedure comprises multiple steps, a step com- prises one or more sentences), tokens of randomly selected M (2 1) step(s) are masked and the task is to predict all such masked tokens.
The remainder of the paper is organized as fol- lows: We begin by introducing the proposed pre- training framework in Section 2, followed by the experimental setup in Section 3. and a description of the datasets, fine-tuning, and evaluation frame- works used for the two downstream tasks, along with experiments, results and analysis in Sections and 5. Finally, we conclude with a summary of our findings in Section 6. The code and models used in this paper are available on Github
In CL.MMM, each input procedure (with tokens of random step(s) masked) is passed through Bidi- rectional Transformer. The MSM Loss function LMSM is the sum of the MLM Loss (Devlin et al.. 2018; Liu et al., 2019) over all the masked tokens, for all the N input procedures per training sample (here, 3, as in Figure 1). Mathematically,
N
CMSM Σ E (i.za
(2)
2 Pre-training
i=1 i=jjEs
where S; is the set of masked token indices in the 7th input procedure, and LM (i, j) is the MLM Loss for predicting the juh masked token, given the 4th input procedure.
CLMSM follows a Multi-Task Learning (MTL) framework, where CL and MSM are performed simultaneously on 3 input procedures per training sample (as shown in Figure 1). In this section, we first describe the CL and MSM methods, followed by detailed description of the CLMSM frame- work, ending with the details of pre-training in the domain of recipes.
2.3 CLMSM Framework
CLMSM framework is shown in Figure/ 1. Here, MSM and CL are both carried out simultancously in an MTL setting for K epochs, and the Trans- former obtained after pre-training is used for fine- tuning. Intuitively, CLMSM learns the step-wise
htt //gi http://ghhuuuucccmaaaaaaaaaaaeee http://...mmmmaaaaaaaaais///SSS EMNLP_2823

---

Triplct
Nanguitvve
Representation
Massi
Mask
Mank
Mark
Mank
MSM Loss
hard sharing
hond
Thile
TALAN
Bidirectional Transformer
Bidirectional Transformer
Bidirectional Transformer
Face
Taska
(AAS)
Masked Step
Masked Step
Masked Step
Anchor
Positive
Negative
Procedure
Procedure
Procedure
Figure I: CL.MSM pre-training framework: Step-Masking and Contrastive Learning objectives are used. MSM Loss obtained by masking step(s) in each recipe is linearly combined with Triplet Loss, and total loss backpropagated.
and fine-grained procedural contexts simultane- ously via MSM and CL, respectively.
considering different recipe of the same dish as positive, and recipe of "similar" dish nega- tive. For a given recipe, set of recipes of "similar" dishes can be filtered in the following way (1) We take the given recipe name*, and find the top 20 most similar recipe names in the corpus based on cosine similarity between pre-trained Sentence Transformer (Reimers and Gurevych, 2019) embed- dings. (2) This is followed by filtering the recipe names that have common noun phrase root (for example, if banana smoothie is given, strawberry smoothie similar) We provide an example of anchor, positive, and negative recipes in Figure 2. We observe that (1) most the ingredients are com- mon, making the anchor and positive highly similar, and the anchor and negative marginally dissimilar, and (2) non-common ingredients with the nega- tive sample are thus discriminative (e.g. Chicken and Mixed vegetables); CL loss helps CLMSM quickly learn those fine-grained features.
The loss (referred to "Total Loss' in Figure backpropagated during pre-training the sum of the triplet margin loss and scaled step masking loss. Mathematically,
= C. + LMSM
(3)
where a is positive scaling constant for LMSM-
2.4 Pre-training in the Recipe Domain
The pre-training data contain over 2.8 million recipes from 4 different sources, described in detail in Section B.4 of Appendix. We further filter the dataset to remove recipes that contain less than steps, which results in 537, 065 recipes. Since a recipe contains a list of ingredients in addition to the steps (procedure) for making the dish, we modify the recipe by adding the list ingredients as step before the original steps the recipe. We randomly choose from 20 manually curated tem- plates to convert the list of ingredients to step (E.g. "Collect these items: (ingredients separated by commas)". "Buy these ingredients from the su- permarket: (ingredients separated by commas] )". etc.). An example of such modification of a recipe is discussed in Section B.4 of Appendix.
Pre-training Setup: Prior applying CLMSM, the bi-directional transformer pre-trained on the corpus of recipes via MLM, as in Liu et al. (2019). In CLMSM, we use batch size of 32 along with AdamW W optimizer (Loshchilov and Hutter, 2017) with an initial learning rate of 5 10-5, which linearly decays to O. CLMSM is pre-trained using MSM and CL simultaneously on 2 Tesla V100 GPUs for K 1 epoch We scale down the MSM loss to bring both the MSM and CL losses in the
To apply MSM, the number of steps masked per recipe is taken randomly as M = or 2 in our experiments. We propose the following hcuristic sampling procedure of hard triplets for CL. Given recipe as an anchor, triplet is sampled by
"the term "recipe name" refers the name of the dish s 'used chunk. root. text inttps::/ssaat http:///sacccccscssaaeeeeee io/usage/ MIT License
'a large number of templates is used to provide variety to this added step across recipes
"Details on number of parameters and compute are dis- cussed in Section B.4 of Appendix

---

Ingrodlnnt
Ingredarit
Ingredlents
4 Experimental Details: Tracking Entities
Roomaat Riza
Basmatl Rice
Hell kg chichent
Half ccichan
Wied
taalespoinn pinger ganlo
Luikogerame yogat
ttabkasp ploger garlz
tableepcome plan yopurt
For the task of entity tracking, we evaluate on (1) NPN-Cooking Dataset in the recipe domain to see how helpful domain-specifer training is for in-domain tasks, and (2) Propara Dataset in the open-domain see how generalizable a domain- specific pre-trained model is.
tablespoon
park
tablospcon
10 to cab paran
menall
mosell
12 Iaappcn
13 besapoon ground
14 Lampunn
55 to besspoon cii
1/2 tu luxapeon chill poadder
surment
Pepper
Wist Loance
Recipe
Positive Recipe
Negative Recipe
Utickonn diryar
Chicken ba
Vegetatre dryum
4.1 Problem Definition: Tracking Entity
Figure 2: Ingredients of the anchor, positive, and nega- tive recipes of a triplet. Ingredients in black common in all recipes, those in green and red are non-commn common ingredients in positive and negative recipes respectively
On execution of step in procedure, an entity can have three predefined locations: non-existence, unknown-location, or known-location, where the "known location" expressed by a span of text describing where the entity is. At the beginning of the procedure, each entity is assigned a location. Besides the location, each entity is also assigned a status. At the step where the entity first appears, it gets status of created. If the location changes, the status changes moved and once the entity ceases to exist, then the status assigned is destroyed. An entity that has not been created till now or has been destroyed has no status.
same range, by setting the coefficient a in Equation 3 (for getting the total loss) to 0.05.
3 Experimental Setup
We evaluate CLMMMM on the tasks of entity track- ing and recipe alignment'. We assess the per- formance of CLMSM vis-a-vis baselines and vari- ous variations of CLMSM, and list inferences that can be drawn. Note that, from here onwards, the subscript in any model notation represents the en- coder used (RB- ROBERTA-BAEE. RL RoBERTa- LARGE, BERT- BERT-BASE-UNCASE
We elaborate on it using an example. The proce- dural text of photosynthesis is described in Table I which contains list of steps. Each row in the table represents step in the procedure, with the first column containing the sentences that describe each step, and the remaining columns containing information about entities involved in the process and their locations after each step. The location of the entities "Before the process starts" is their initial location and is not affected by the process. Considering the entity "water", it is in the soil be- fore the procedure begins, and moves to the root after execution of the first step. stays in the leaf after the second and third steps. The fourth step converts the water, light, and CO2 into a mixture, and thus, water ceases to exist thereafter. Thus, "water" is destroyed and "mixture" is created in the fourth step.
3.1 CLMSM Variations
For all the tasks we consider the following varia- tions of CLMSM for comparison as baselines (I) CLMSM (CL) uses only the CL pre-training ob-
jective (2) (MSMM uses only the MSM pre-training objective (3) CLMSM(CAS.) uses a cascaded framework (CAS.) instead of MTL, i.e, it performs CL followed by MSM, instead of op- timizing both the objectives simultancously. (4) CLM)M(RS) uses a variation of MSM by mask- ing random spans instead of step(s). This varia- tion is used to establish the utility of step-masking as a supervision signal for procedural text. For a fair comparison, we mask the same percentage of tokens when masking random spans and when masking step(s). (5) CLAMSMEEASY uses easy triplets instead of hard triplets to see if fine-grained differences between anchor and negative proce- dures are indeed helpful. For sampling casy triplets, the positive recipe name is similar to the anchor recipe name (definition of "similar" is mentioned in Section 2.4), and the negative recipe name is not similar to the anchor recipe name.
Formally, given procedure P consisting of list of n steps P {pi, (p.,,..,P)) P.)). the entity tracking task identifies the location and the status S of entity e at step ps V(i). Mathematically, (S, L.) = F(e,
4.2 Fine-tuning
The transformer model provides a step-dependent entity representation which used to output the sta- tus and location of the entity. The status is derived using a linear classification module. To predict the location of an entity the approach of Span-Base Based
Dutascts are in the English Language

---

Lannes
nent of the model is not employed in this evaluation
Proceuue
Mixuue
[Before process stari)
(K)
Sun (K)
9.4
(as in Faghihi and Kordjamshidi (2021)).
Bouncrbuch soner from will
Rant [1]
San [K]
0115
S1N
80N)
The lowsro
Lka
Sun [K]
8/U)
Light
and CO2
(K)
3.00
000)
4.4.2 Dataset-specifie Baseline
eeirr the
Water. light and CU2 combine
(())
0.0)
0/N)
We Neural Process Network Model (NPN- Model) (Bosselut et al., 2017) as baseline, de- scribed in detail in Section D.4.1 of Appendix.
into
foons sugas
KNN)
0.00
0.30
(K)
Table An annotated sample from the ProPara dataset. "Q" means the location of the entity is undefined. The location of the entity in each step is represented within brackets ("K' known-location, 'U' 'N' non-existence)
4.4.3 Results
We compare CLMSM (pre-trained CLMSM model with RoBERTa-BAR (RB) encoder) with the Baselines and CLMSM Variations. The results are shown in Table 2.
Question Answering is followed, whereby loca- tion span output. Sum of Cross-entropy losses pertaining to the location and status prediction is used to fine-tune the model.
Comparison with Baselines: We can infer that (1) CL.MMMBB gives significantly better test accuracy (p < 0.05) than all baselines, performing 1.28% better than the best baseline (RECTPFSHB)) This suggests that the pre- training objectives are helpful. (2) RECIPESRB is the best baseline due to domain-specific pre- training (3) DYNAPRO uses an end-to-end neural architecture and a pre-trained language model to get entity representations and performs better than NPN-Model and KG-MRC, but performs worse than RECIPESRB due to lack of in-domain knowledge before fine-tuning. (4) MPN-Model (which is data-specific baseline) and KG-MRC give poor results due to the lack of an end-to-end neural architecture and prior knowledge through pre- tre-trained models.
The fine-tuning is carried out in two stages as recommended by Faghihi and Kordjamshidi (2021) first on the SQuAD 2.0 Dataset (described in Sec- tion D.2 of Appendix), and then on one of the two target datasets. The
used for fine- tuning are described in Section D.2 of Appendix.
4.3
Common Baselines
The following baselines have been shown to work satisfactorily well for both in-domain and open-domain downstream tasks in prior works, and are hence common for both ProPara and NPN-Cooking Datasets- - KG-MRC (Das et al.. 2018), DYNAPRO (Amini et al., 2020), and RECIPESRD (model obtained after pre-training RoBERTa-BASE on the corpus of recipes using MIM, fine-tuned as described in Section 4.2.). The baselines are described in detail in Section D.3 of Appendix.
Comparison with CLMSM Variations: We can infer that (1) CLMSM nB performs better than CLMSMRE(CAS)), CLMsMnR(RS), and CLMSM (EAYY). suggesting that perform- ing MTL on CL (using hard triplets) and MSM objectives is indeed helpful for downstream in- domain Enity Tracking (2) CLMSMRB performs only slightly inferior compared to when using the objectives individually in CLMSMRB(CL) and (s with minimal drop of 0.78% and 1.12% respectively. As pre-training and fine-tuning are performed in the same domain, the simple framework of using single loss is suffi- cient. We shall see later that in case of ProPara, where pre-training and fine-tuning are carried out in different domains, combination of the two losses in CLMSM is helpful in enhancing performance.
4.4 Entity Tracking on NPN-Cooking Dataset The NPN-Cooking dataset (Bosselut et al., 2017) consists of 65, 816 training. 175 development, and 700 evaluation recipes". The dataset consists of Human Annotations corresponding to changes in the status and location of the entities throughout the procedure.
4.4.1 Evaluation Metrics
The model is evaluated on the accuracy the pre- dicted locations at steps where the location of the ingredients change. The status prediction compo-
4.5 Entity Tracking on ProPara Dataset
The ProPara Dataset (Mishra et al., 2018) consists of 488 human-authored procedures (split 80/10/10 into train/dev/test) with 81 annotations regarding
'NPN stands Neural Process Networks
*This dataset a subset of the larger "Now You're re Cook- ing" dataset (Kiddon et al., 2016).

---

TEST
Comparison with Baselines: We can infer that (1) CLMSMRL performs the best w.r. Cat. 1, Cat. 2, and average Scores, even though CLMSM is pre-trained on domain-specifie corpora (which is different from the domain of the ProPara Dataset).
ACCURACY
NPN-Model
51.3
KG-MRC
51.6
DYNAPRO
62.9
RECIPESBB
63.8
CLMSM CL)
65.13
CLMSM (MSM)
65.35
CLMSM RL. gives significant improvement (p < 0.05) of 2.56% over the best baseline (DYNAPRO) in terms of average category score. (2) CLMSMRE outperforms 5 baselines on all 3 metrics, and gives better Average Score than 8 baselines. (3) NCET and DYNAPRO achieve high evaluation metrics, as NCET uses extra action/verb information, while DYNAPRO uses an open-domain pre-trained lan- guage model in its framework. This is the rea- son behind these models performing better than RECIPESRB. which is pre-trained on recipes. DYNAPRO especially gives high Cat. 3 Score due an additional Bi-LSTM that smoothens pre- dicted locations across steps, while adding addi- tional fine-tuning overhead. (4) KG-MRC performs well, as it builds a dynamic Knowledge Graph of entities and states, which then becomes a structured imput to the MRC, making it casier to figure out the entity's 's state. (5) EntNet and QRN perform poorly, as they both follow very simple recurrent update mechanisms, that are not complex enough to under- stand procedures effectively. (6) ProGlobal shows considerable improvement in the Cat. Score, as attends across the entire procedure, thus having better global consistency.
CLMSMRR(CAAL (CAS)
63,99
CLMSMaR RS)
63.46
CLMSMRR((AAS)
63.68
CLMSMRB
64.62
Table Results when fine-tuned on the NPN- IPN-Cooking Dataset- Baselines and CLMSM Variations CLMSM
the changing states (existence and location) of enti- ties in those paragraphs, with the goal of predicting such changes. We focus on evaluating the entity state tracking task in this work.
4.5.1 Evaluation Metrics
The model is evaluated whre corresponding a given entity They are: Category 1: which of the three transitions- created, destroyed, or moved undergone by an entity over the lifetime of a procedure; Category 2: the steps at which e is created, destroyed, and/or moved; and Category 3: the location (span of text) which indicates e's creation, destruction or movement. We also report the average of the 3 evaluation metric scores.
4.5.2 Dataset-specific Baselines
We use the following dataset-specific baselines a rule-based method called ProComp (Clark et al., 2018). a feature-based method (Mishra et al., 2018) using Logistic Regression and a CRF model, ProLocal (Mishra et al., 2018), ProGlobal (Mishra et al., 2018), EntNet (Henaff et al., 2016), QRN (Seo et al., 2016), NCET (Gupta and Dur- rett, 2019), and RECIPESRL (which is similar RECIPESRB. except that, RoBERTa-LARGE is used instead of RoBERTa-BASER The base- lines are described in detail in Section D.5.1 of Appendix.
(7) ProLocal performs slightly better, but the improvement in the Cat. 3 Score (corresponding to the location of an entity) is negligible. This can be due to ProLocal making local predictions, thus having poor global consis- tency. (8) Rule-hased and Feature-based baselines perform poorly, as they cannot generalize well the unseen vocabulary in the test set.
Comparison with CLMSM Variations: (1) CLMSMRØ gives the best Average Score, giving an improvement of 1. 1.73% compared to the best vari- ant CLMsM CLMsMRe(CASS AS.). suggesting that MTL is es- sential in enhancing downstream task performance. (2) CLMS outperforms all the variants on aver- age score, and 2 out of 5 variations on all 3 metrics. It falters on Cat.3 score. The impact of MSM seems to be high on determining Cat.3 score whereby CLMSM with only MSM or MSM employed af- CL performs better. (3) CLMSM (RS) gives the best Cat. Score among the C1.MSM Varia- tions and CLMSMRB- This might be attributed to random span masking being helpful in extracting
4.5.3 Results
We compare CLMSMRB and CLMSMRL (CLMSM with an encoder having ROBERTa-LARGE archi- 1ecture) with Baselines and CLMSM Variations. The results are shown in Table 3. Additionally, compare CLMSM with much larger LLMs (Large Language Models).
We use RoBERTa-LARRE based models here similar to
Faghihi and Kordjamshidi (2021)

---

TEST
TEST
TEST
AVERAGE SCORE
Case Study. Table shows the ground truth an- notations and the predictions of CLMSMRL and RECIPESRI on a procedure from the ProPara Dataset that describes the process of the growth of a tree. We can infer that (1) RE tends to predict location span in the same step as the entity (while the actual location might be present in some other step), which suggests that RECIPESEL is not able to focus on long-range contexts across steps (2) CLMSM is able to bet- ter understand long-range contexts. E.g. in the case of the entities "seedling" and "root system", CLMSM correctly predicts the location correspond- ing to the ground truth "ground" for most steps (unlike RECIPESRL), even though "ground" is mentioned only once in the first step of the procedure. This is because CLMSM learns local as well as global procedural contexts during pre- training. Hence, CLMSM performs better than RECIPES for tracking entities whose loca- tions are implicit in nature, i.e., the location at the current step requires retention of knowledge from (3) when it comes GPT-4 in 3-shot setting, CLMSM Rt still beats GPT-4 (3-shot) in Cat. 1, Cat.
CAL1 1
CAT.2
C SCORE
SCORE
SCORE
Rule-based
57.14
20.33
2.4
26.62
3, and Average Scores, and a close second in Cat Score (4) CLMSM is a highly capable model with a well-tailored pre-training scheme, even though its number of parameters and pre-training data is just small fraction of that of GPT-3.5 and GPT-4.
Seature-aased
58.64
20.82
9.66
29.71
62.7
10.4
ProGlobal
63
36.4
35,9
EntNet
51.6
18.8
7.8
26.07
QRN
52.4
15.5
10.9
26.27
KG-MRC
62.9
40.0
38.2
47.03
NCET
73.7
47.1
41.0
53,93
DYNAPRO
72.4
4.5.4
Performance Analysis: CLMSM and RECIPESRL
RECIPESEB
71.33
34.11
30.86
45.43
RECTPESSL
74,58
47,53
38.37
53.49
CT MSM (CT.
67.8
35.62
33.61
45.68
CLMSMRaI (MSM) (MSM)
67.09
36.83
34.03
45.98
We present a case study to elaborate the function- ing of CLMSM vis-a-vis the most efficient baseline RECIPES Also, we present detailed quan- titative results elaborating the difference between
CLMSMan (CAS.)
67,94
36.85
35,94
46.91
CLMSMRa (SS)
65.4
38.25
MSMMa (FASY)
68.22
35.7
32.1
45.34
CLMSMR0
69.92
39.35
33.89
47.72
CLMSMRE
77.26
54,86
38.34
56.82
the two closely competing models.
Table 3: Results when fine-tuned on the ProPara Dataset Baselines and CLMSM Variations vs. CLMSM
location spans.
Additionally, we analyze why CLMSM pre- trained on recipes performs well on open-domain tasks based on attention weights Section D.5.2 of Appendix.
Comparison with LLMs: We compare with the following LLM baselines (in Table -)- (1) Open-source LLMs such Falcon-7B (pre-trained LLM) and Cercon-7B-Instrrre
(instruction- fine-
tuned Falcon-7B) (Penedo et al., 2023) (2) Ope- nAI's GPT-3.5 (gpt) and GPT-4 (OpenAI, 2023; Eloundou al., 2023) models. The LLMs are used in a I-shot and s-ot In-Context Learning Setting (Dong al., 2022).
TEST
TEST
TEST
AVERAGE SCORE
CAT.2
CAT.3 SCORE
SCORE
SCORE
Faleon-78 (1-chot)
50.42
7.11
4.70
20.77
Falcon (3-shot)
1ooon-78-lostreet (1-shot)
50.42
5.42
0.38
18.74
Faloon-78-Instrun (3-shor)
48.44
3.15
1.94
17.84
step far in the past.
5P
53.25
24.66
11.37
29.76
Quantitative Comparison. We perform quanti- tative analysis of the type of predictions made by
GPT-3.5 (3-shot)
62.43
34.66
15.81
37.63
GPT4
GPT-4(3-shon
23.87
$7.7
26.78
52.78
CLM
69.92
39.35
33.89
47.72
CLMSM and the RECIPESRI baseline on the ProPara Test set. We found that (1) CLMSM is able to correctly predict the statuses of "non-existence" and "known-location" of an entity 81.37% and 70.1% 1% of the times respectively, which is better than RECT which does so 77.59% and 67. 67.68% of the times. (2) When the location of the entity is known, CLMSM predicts the span of the location with 49.04% partial match and 37.62% exact match, while RECTPFSSK falls behind again, with 48.39% partial match, and 36.17% ex- act match. Thus, CLMMM performs better than
CLMSMRL
Table 4: Results on the ProPara Dataset CLMSM
LLMs vs.
Table 4 shows that (1) even though Falcon-7B and Falcon-7B-Instreen have almost 6x and 14x the number of parameters compared to CLMSM and CLMSM respectively, perform consider- ably worse in comparison (2) both CL.MSM and CLMSMRL still outperform GPT-3.5 in both set- tings, and GPT-4 in I-shot setting on all 4 metrics.

---

Ground Truth
CLMMM
RECEPESp
material
matetial
material for now growth
Procodure
seedling
not
tree
for
scedling
mot system
seedling
rool system
system
growth
gowwh
(Refore the process starts)
ground
1. A seadling is grown the ground.
ground
ground
ground
2. The seedling grows bigger, and fomms
ground
ground
ground
ground
poot system.
3. The roods start to gain more nourishmnnn
ground
ground
ground
ground
ground
ground
ground
as tree grows.
4. The tree sturts to attract animals and planss
will prant
nourishment.
ground
ground
ground
ground
ground
ground
ground
5. The tree mattrrss
ground
ground
ground
ground
ground
ground
ground
6. number years the
grow old and die, hecoming mnterial growth.
ground
ground
ground
ground
trae
ground
1rcc
Table 5: Analysis of the ground truth and the predictions of CLMSM vs, a well-performing baseline on a sample from the ProPara Dataset. CLMSM is able to predict the ground truth location "ground" in most steps, even though "ground" is mentioned only once in the first step
RECIPESRL in cases where the entity's location is known, well as, when the entity does not exist.
action alignments between recipes from a subset of (Lin et al., 2020) dataset. The dataset includes 1 110 recipes for 10 different dishes, with approximately 11 lexically distinct writeups for each dish. These recipes have been annotated with 1592 action pairs through crowdsourcing. 10 fold cross-validation is performed, with recipes of 1 dish acting as the test set each time.
5 Experimental Details: Aligning Actions
As mentioned in Donatelli et al. (2021), action alignment implies each action in one recipe being independently assigned an action in another sim- ilar recipe or left unassigned. Given two similar recipes R3 (source recipe) and R2 (target recipe), we need to decide for each step in Rg the align- ing step in the Rg, by mapping the word token corresponding to an action in R to one or more word tokens corresponding to the same action in R2. However, some actions in the source recipe may not be aligned with actions in the target recipe.
5.2 Results
We compare CLMSMBERE with the baseline RECIPESPERT (BERT-BASE-UNCASEE) pre- trained on the pre-training corpus of recipes using MLM) and CLMSM Variations. Note that we use models with BERT-BASE-UNCARE similar to Donatelli et al. (2021), which proposed the task. The evaluation (test accuracy) is based on the fraction of exact matches between the ground truth and the predicted action tokens the test set. The results are shown in Table 6.
architecture
An example of the task is explained as follows Let R1 and R2 be recipes to make waffles. We observe the alignment of the step "Ladle the batter into the waffle iron and cook [...]" in Rı with the step "Spoon the batter into preheated waffle iron in batches and cook [..]" in R2 (Words in bold are the actions). The relationship between the specific actions of "Ladle" in R1 and "Spoon" in Ra as well as between both instances "cook" may not be easy to capture when using coarse- grained sentence aligmment, which makes the task challenging.
Comparison with Baseline: CLMSMBERT per- forms significantly better (p < 0.05), showing an improvement of about 4.28% over the baseline RECIPESBERT- This considerable improve- ment is obtained intuitively due to CL on hard triplets (capturing similarity between entities of the recipes helps in aligning actions across recipes) and MSM (predicting a masked step given other steps implicitly helps learn similarity of actions across different steps).
The details about fine-tuning on the task are de- scribed in Section E of Appendix.
5.1 Aligned Recipe Actions (ARA) Dataset
Comparison with CLMMMM Variations: We
The ARA dataset (Donatelli et al., 2021) consists of same recipes written by different authors, thus the same recipe might have different number of steps. Thus the recipes in this dataset are lexically dif- ferent, however semantically similar. The dataset is an extension of crowdsourced annotations for
can infer that (1) CLMSMBERT outperforms all the variations, as using an MTL framework on the proposed pre-training objectives enhances downstream action alignment performance, and (2) CLMSMBERT ( M SM) performs the best among all the variations, as MSM helps capture step-wise

---

procedural context, helping in action alignment.
Ethics Statement
The proposed methodology is, in general, appli- cable to any domain containing procedural text. Specifically, it can potentially be applied to user- generated procedures available on the web and is likely learn patterns associated with exposure bias. This needs be taken into consideration before applying this model to user-generated pro- cedures crawled from the web. Further, like many other pre- tre-trained language models, interpretability associated with the output is rather limited, hence users should use the output carefully.
TEST ACCURACY
RECIPES BURY
64.22
CLMSMBRRP(CL)
55,96
CLMSMBERT (MSMM)
61.47
CLMSMBERT CAAS.)
55,96
CI CIMM RRS)
54.31
L.MMMMRRR(EAAY
57.06
CIMMMMEER''
66.97
Table 6: Results when fine-tuned on the ARA Dataset- Baselines vs. CLMSM
6 Summary and Conclusion
References
Gpt-3.5 turbo documentation.
In this paper, we propose CLMSM, a novel pre- training framework specialized for procedural data, that builds upon the dual power of Contrastive Learning and Masked Step Modeling. The CL framework (with hard triplets) helps understand the identities (entities and actions) which drive a process, and MSM helps to learn atomic unit of process. CLMMSM achieves better results than the state-of-the-art models in various procedural tasks involving tracking entities in a procedure and align- ing actions between procedures. More importantly, even though CLMSM is pre-trained on recipes, shows improvement on an open-domain procedural task on ProPara Dataset, thus showing its general- izability. We perform extensive ablation study with various possible variants and show why the act design of CL and MSM (within the pre-training framework) is befitting for the lasks at hand. We believe that our proposed solution can be extended to utilize pre-training data from multiple domains containing procedural text.
Aida Amini, Antoine Bosselut, Bhavana Dalvi Mishra, Yejin Choi, and Hannaneh Hajishirzi. 2020. Proce- dural reading comprehension with attribute-aware context flow.
Michal Bierí, Michal Gilski, Martyna Maciejewska, Wo- jeiech Taisner, Dawid Wisniewski, and Agnieszka Lawrynowicz. 2020. RecipeNLG: cooking recipes dataset for semi-structured text generation. In Pro- ceedings of the 13th International Conference Natural Language Generation, pages 22 28, Dublin, Ireland. Association for Computational Linguistics.
Antoine Bosselut, Omer Levy. Ari Holtzman, Corin Ennis, Dieter Fox, and Yejin Choi. 2017. Simulating action dynamics with neural process networks.
Richard Caruana. 1993. Multitask learning: A knowledge-basee based source of inductive bias. In Pro- ceedings of the Tenth International Conference on Machine Learning, pages 41-48. Morgan Kaufmann.
Khyathi Raghavi Chandu, Eric Nyberg, and Alan W. Black. 2019. Storyboarding of recipes: Grounded contextual generation. In ACL.
Jingjing Chen and Chong-wah Ngo. 2016. Deep-based ingredient recognition for cooking recipe retrieval. In Proceedings of the 24th ACM International Con- ference on Multimedia, MM '16, page 32-41, New York, NY, USA. Association for Computing Machin- ery.
Limitations
Our work performs pre-training on procedural text in the domain of recipes. Metadata other than recipe names (e.g. cuisine) could be used for sam- pling similar and dissimilar recipes. This work can be extended to other domains containing procedu- ral text, however, that may bring in new challenges which we would need to solve. For example, for the text from the scientific domain, E-Manuals, etc. that contain a mixture of factual and procedural text, an additional pre- preprocessing stage of extract- ing procedural text needs to be devised.
Peter Clark, Bhavana Dalvi, and Niket Tandon. 2018. What happened? leveraging verbnet to predict the effects of actions in procedural text.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. 2020. SPECTER: Document-level Representation Learning using Citation-informed Transformers. In ACL.
Bhavana Dalvi, Niket Tandon, Antoine Bosselut, Wen- tau Yih, and Peter Clark. 2019. Everything happens

---

for reason: Discovering the purpose of actions in procedural text. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4496-4505.
Angela Lin, Sudha Rao, Asli Celikyilmaz, Elnaz Nouri, Chris Brockett, Debadeepta Dey, and Bill Dolan. 2020. A recipe for creating multimodal aligned datasets for sequential tasks. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4871-4884, Online. Asso- ciation for Computational Linguisties.
Rajarshi Das, Tsendsuren Munkhdalai, Xingdi Yuan, Adam Trischler, and Andrew McCallum. 2018. Building dynamic knowledge graphs from text us- ing machine reading comprehension.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018, Bert: Pre- Pre-trrinnng of deep bidirectional transformers for language understand-
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization.
ing.
Lucia Donatelli, Theresa Schmidt, Debanjali Biswas, Arne Köhn, Fangzhou Zhai, and Alexander Koller. 2021. Aligning actions across recipe graphs. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 6930- 6942, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni. and Julian McAuley. 2019. Generating personalized recipes from historical user preferences.
Javier Marin, Aritro Biswas, Ferda Ofli, Nicholas Hynes. Amaia Salvador, Yusuf Aytar, Ingmar Weher, and Antonio Torralba. 2018. Recipelm+: A dataset for learning modsl embeddings for cooking recipes and food images.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2330.0000 230.00344...
Bhavana Dalvi Mishra, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018. Tracking state changes in procedural text: A challenge dataset and models for process paragraph comprehension.
Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023. Gpts are gpts: An early look at the labor market impact potential of large language models. arXiv preprint ar3iv:2303. 2303..0130...
Abhilash Nandy, Soumya Sharma, Shubham Mad- dhashiya, Kapil Sachdeva, Pawan Goyal, and NIloy Ganguly. 2021. Question answering over electronic devices: A new benchmark dataset and a multi-task learning based QA framework. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4600-4609, Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Hossein Rajaby Faghihi and Parisa Kordjamshidi, 1021. Time-stampa stamped language model: Teaching language models to understand the flow of events.
Aditya Gupta and Greg Durrett. 2019. Tracking discrete and continuous entity state for process understanding. In Proceedings of the Third Workshop on Structured Prediction for NLP, pages 7-12, Minneapolis, Min- nesota. Association for Computational Linguistics.
Medhini Narasimhan, Licheng Yu, Sean Bell, Ning Zhang, and Trevor Darrell. 2023. Learning and veri- fication of task structure in instructional videos.
Mikacl Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2016. Tracking the world state with recurrent entity networks.
OpenAl. 2023. Gpt-4 technical report.
Malte Ostendorff, Nils Rethmeier, Isabelle Augenstein, Bela Gipp. and Georg Rehm. 2022. Neighborhood contrastive learning for scientific document repre- sentations with citation embeddings. arXiv preprint a.2.:.220..06671.
Chloé Kiddon, Luke Zettlemoyer, and Yejin Choi, 2016. Globally coherent text generation with neural check- list models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Process- Austin, Texas. Association for Computational Linguistics.
ing, pages
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zach VeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation pytorch.
Yi-An Lai, Garima Lalwani, and Yi Zhang. 2020. Con- text analysis for pre-trained masked language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3789-3804.
Guilherme Penedo, Quentin Malartic. Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei. and Julien Launay. 2023. The refinedweh dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint 230iv:2306.01116.
Mike Lewis. Yinhan Liu. Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising sequence-to-sqqeeee pre-training for natural language generation, translation, and comprehension.

---

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21()))48555551
2. RecipeNLG dataset (Bieñ et al., 2020): This dataset was further built on top of Recipe IM+ dataset. Deduplication of the dataset was car- ried out in order to remove similar samples using the cosine similarity score, which was calculated pairwise upon the TF-IDF repre- sentation of the recipe ingredients and instruc- tions. In the end, the dataset added an addi- tional 1.6 million recipes our corpus.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't t know: Unanswerable ques- tions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Pupers), pages 784-789.
(Majumder et al., 2019): They collect recipes along with their user reviews for a duration of the past 18 years and further filter them using a certain set of rules to create a dataset of about 180,000 recipes and 700,000 reviews which could be used recipe metadata in future experimentation.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pages 3982-3992.
Minjoon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Query-reduction networks for ques- tion answering.
4. (Chandu et al., 2019) : This dataset was scraped from how-to blog websites such as instructables.com and snapguide. com- prising step-wise instructions of various how- to activities like games, crafts, etc. This a relatively small dataset comprising of around 33K recipes.
Simeng Sun, Kalpesh Krishna, Andrew Mattarella-
Micke, and Mohit lyyer. 2021. Do long-range lan- guage models actually use long-range context? In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 807- 822.
Appendix
The Appendix is organized in the same sectional format as the main paper. The additional mate- rial of a section is put the corresponding section of the Appendix so that it becomes easier for the reader to find the relevant information. Some sec- tions and subsections may not have supplementary material so only their name is mentioned.
Datuset
Number Recipes
Number words
(only steps)
(oaly ingredicnts)
Re
1,029,720
137. 361,91
51.523.219
RecipeNEG
1.643,009
9777281.977
83658,858
(Majumder et al., 2019)
179,217
23,774,704
3,834,978
(Chandu 2019)
38,720
26,243,714
Total
2,885,758
334,664,989
151001,0555
Table 7: Statistics of Recipe Corpus
A Introduction
Table 7 shows the statistics of the pre-training corpus containing over 2.8 million recipes. Modification a recipe by adding ingredients as an extra step: An example of such a modification of a recipe is shown in Figure 3. Pre-training Setup
B
Pre-training
B.1 Contrastive Learning using procedure similarity (CL)
B.2
Mask-Step Modeling (MSM)
B.3
CLMSM Framework
CLMSM ENCODER
NUMBER OF TRAINABLE
NUMBER OF GPU-HOURS
B.4 Pre-training in the Recipe Domain
ARCHITECTURE
PARAMETERS
Source of the pre-training data
BERT-BASE-UNCASE
397, 0988
14
ROBERTa-BASE
126.445
16
1. MM dataset (Marin et al., 2018): This dataset consists of 1 million textual cook- ing recipes extracted from popularly available cooking websites such as cookingpad.com. recipes.com, and foodnetwork.com others. They include recipes of all the cuisines unlike (Chen and Ngo, 2016) which only con- tain recipes of Chinese cuisine.
RoBERI--LARGE
1,517,99,,007
Table 8: Number of trainable parameters in CLMSM for different encoder architectures and the corresponding pre- tre-training compute in terms of GPU-HOURS
amongst
Table 8 shows the number of trainable param- eters in CLMSM for 3 different encoder architec- tures and the corresponding pre-training compute

---

PROCESSED RECIPE
Ingredients: water, salt, bailin potatoes, fresh spinach leaves, unsalted butter, coarse salt, fresh ground black pepper, and INGREDIENTS
boiling potatoes, fresh spinach leaves, unsalted butter, coarse salt, fresh ground black pepper, and nutmeg.
Collect
nutmeg.
cut the potatoes half lengthwise.... INSTRUCTIONS
bring water and salt to a boil.
brrig water and šalt to a boil.
1/2 thick slices.
3. cut the potatoes in haif lengthwise and then into 1/ 2" thick slices. boll over medium-high heat for twenty minutes.
3. boil over medium-high heat for twenty minutes.
4. add the spinach and boil another 10 minutes being careful not overcook in order to maintain it's bright green color.
add the spinach and boll another 10 minutes being careful not overcook in order to maintain it's bright green color.
5. drain, reserving 1-1 / cups cooking liquid.
drain reserving
cooking liquid.
6. a food processor, process the potatoes and spinach until very smooth, adding the butter 2 at time.
food processor process the potatoes and spinach until very smooth, adding the butter 1/ 1 time.
7. return the puree to the pot and add the reserved liquid until the desired consistency.
return puree to pot desired consistency.
the
liquid until
reheat slowly and season with salt and pepper.
9. reheat slowly and season with salt and pepper.
9. ladle into shallow soup plates and garnish with nutmeg or shaved cheese.
ladle into shallow soup plates gamish nutmeg shaved cheese.
Figure 3: Modifying recipe by adding ingredients step the top of the original recipe
in terms of GPU-HOURS (number of GPUs used
D.3 Common Baselines
multiplied by the training time in hours).
KG-MRC uses dynamic knowledge graph of en- tities and predicts locations by utilizing reading comprehension models and identifying spans of text. DYNAPRO is an end-to-end neural model that predicts entity attributes and their state transi- tions. It uses pre-trained language models to obtain the representation of an entity at each step, iden- tify entity attributes for current and previous steps, and develop attribute-awan representation of the procedural context. It then uses entity-aware and attribute- aware representations to predict tran- sitions at each step.
C Experimental Setup
C.1 CLMSM Variations
Experimental Details: Tracking Entities
D.1 Problem Definition:
Tracking Entity
D.2 Fine-tuning
Intermediate Fine Tuning on SQuAD Dataset:
The SQuAD 2.0 dataset (Rajpurkar et ul., 2018) is a span-based open-domain reading comprehension dataset that consists of 130, 319 training, 11, 873 dev, and 8, 862 test QA pairs. To improve perfor- mance on downstream tasks, the encoder is fine- tuned on the SQuAD 2.0 training set before being fine-tuned on the task-specific dataset. The hyper- parameters used in this process are the same as those mentioned in Rajpurkar et al. (2018).
D.4 Entity Tracking on NPN-Cooking Dataset
D.4.1 Dataset-specific Baselines
NPN-Model converts a sentence to a vector using GRU-based sentence encoder. The vector rep- resentation is then used by an action selector and entity selector to choose the actions and entities in the step. A simulation module applies the chosen actions the chosen entities by indexing the ac- tion and entity state embeddings. Finally, the state predictors make predictions about the new state of the entities if state change has occurred.
Hyperparameters for NPN-Cooking Dataset: The hyperparameters used for fine-tuning are the same as that used in the default implementation of (Faghihi and Kordjamshidi, 2021)- SGD optimizer (Paszke et al., 2017) with 1 10-6 as the learning rate and a scheduler with coefficient of 0.5 to update the parameters every 10 steps.
D.5 Entity Tracking on ProPara Dataset
D.5.1 Dataset-specific Baselines
ProComp uses rules to map the effects of each sen- tence on the world state, while the reature-based method uses a classifier to predict the status change type and a CRF model predict the locations. Pro- Local makes local predictions about state changes and locations based on each sentence and then glob- ally propagates these changes using a persistence rule. ProGlobal uses bilinear attention over the
Hyperparameters for ProPara Dataset: The hy- perparameters used for fine-tuning same that used in the default implementation of (Faghihi and Kordjamshidi, 2021), SGD optimizer (Paszke et al., 2017) with 3 and scheduler with a coefficient of 0.5 to update the parameters every 50 steps.
as the learning rate

---

entire procedure and distance values make pre- dictions about state changes and locations, taking into account the previous steps and current repre- sentation of the entity. EntNet consists of a fixed number of memory cells, each with its own gated re- current network processor that can update the cell's value based input. Memory cells are intended to represent entities and have gating mechanism that only modifies cells related to the entities. QRN is single recurrent unit that simplifies the recurrent up- date process while maintaining the ability to model sequential data. It treats steps in a procedure as a sequence of state-changing triggers and transforms the original query into a more informed query as it processes each trigger. NCET uses entity men- tions and verb information to predict entity states and track intit--location pairs using LSTM. It updates entity representations based each sen- tence and connects sentences with the LSTM. To predictions, NCET
Where
<9>


water
our
water
Where
</SP
bover
water
absorb
water
from
the
pot
</3>>
</S>>
The
and
water
until
flows
smooth
get
leaf
the
</E>
(a)
(b)
Figure 4: Attention weights between question and procedure steps for Layer 19, Attention Head Self-attention is represented lines connecting tokens, and the line weights reflect attention scores
ensure the consistency uses a neural CRF layer over the changing entity representations.
From Figure 4, we can infer the following sim- ilarities between the open-domain and in-domain scenarios When considering an inner transformer layer in between (here Layer 19), the question shows high attention towards a location or an en- tity closer to the beginning of the procedure. For instance, in the open-domain case, the question attends highly the location "Roots", while in the in-domain case, it attends highly to the entity "oranges".
D.5.2 Reason behind domaain-generalizabiiiy
Hence, this example shows that there a similar- ity in the attention maps between the open-domain and in-domain cases, which leads to improved per- formance in both cases. This similarity might stem from the similarity in the structures of procedures across domains, which could be potential direc- tion of future work.
We observe the attention weights between the ques- tion and the steps of the open-domain as well as in-domain procedures for the transformer obtained after fine-tuning CLMSM on the open-domain ProPara Dataset.
E Experimental Details: Aligning Actions
Fine-tuning: This is treated as an alignment prob- lem where each action in the source recipe is inde- pendently aligned to an action in the target recipe or left unaligned. The task uses a two-block archi- tecture with an encoder that generates an encoding vector for each action in the recipe and a scorer that predicts the alignment target for each action using one-versus-all classification. The encoder is an extended model that incorporates structural information about the recipe graph. The scorer uses multi-layer perceptron to compute scores for the alignment of each source action to every target action, including a "None" target represent-
Figure 4 shows the attention weights. Figure 4a shows the attention weights in layer 19 of the transformer corresponding to tracking the location of the entity "water" in the first two steps of pho- tosynthesis, taken from the open-domain ProPara Dataset. Figure 4b shows the attention weights for the same layer and attention head corresponding to tracking the location of the entity "water" in the first two steps the Orange Juice recipe, which is an in-domain procedure.

---

ing no alignment. The model is trained using cross-entropy loss function with updates only on incorrect predictions to avoid overfitting.
Hyperparameters for ARA Dataset: The hyper- parameters used are the ones mentioned in the de- fault implementation of Donatelli et al. (2021).
E.1
Aligned Recipe Actions (ARA) Dataset Results
E.2
F
Summary and Conclusion


Reviews:
[{"review_id": "B3TsMM2rsU", "reviewer": "Reviewer_z7wU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. The authors could add one or more examples to demonstrate the procedural data the model is trained on. It is not easy for people unfamiliar with the recipe data to get a good intuition of this method.**\n\nTable 1, Table 4 show examples of open-domain processes, and Figure 3 in Appendix shows an example recipe. However, we acknowledge the reviewer’s comment and would add some example recipes in the main paper for the benefit of the reader.\n\n**2. Also, many sections seem a little bulky, like lines 435-468**\n\nFrom lines 435-468, we have compared the results between our method and baselines, with a reasoning behind the relative difference in model performance. In order to fit such content within the confines of available space, perhaps the content has become dense. However, as per the reviewer’s suggestion, we would carefully review the entire paper and ensure that the content becomes more lucid.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper presents a multi-task framework for pretraining/continual training on procedural data and outperforms the baseline, which is the state-of-the-art method when this work is done.\n\n- The authors conducted extensive and comprehensive experiments, oblation study the effectiveness of the proposed framework.", "reasons_to_reject": "- The writing of this paper requires improvement. The authors could add one or more examples to demonstrate the procedural data the model is trained on. It is not easy for people unfamiliar with the recipe data to get a good intuition of this method. Also, many sections seem a little bulky, like lines 435-468.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a multi-task learning framework for continual training on procedural data, which involves both Contrastive Learning using procedure similarity and Mask-Step Modeling. The authors evaluate this framework on recipe data and achieve good performance and generalization performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yLI1Lz13xV", "reviewer": "Reviewer_9spE", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Why modeling procedural texts is useful to begin with?**\n\nIn this work, we explore fundamental tasks within procedural reasoning - tracking (1) entities and (2) actions acted upon those entities, across the steps of a procedure. Automated tracking of entities in a complex procedure helps an end user understand the life cycle of different entities, along with the extent to which an entity participates/is acted upon in the process. We get a glimpse of this when tracking entity locations in the Propara Dataset, and this notion can be extended to even more complex biological and chemical processes. Secondly, aligning actions across procedures (written by different people) helps an end-user understand similarities and the exact subtle differences across the procedures that ultimately have the same goal. For instance, two Biryani recipes could have many common but subtly different cooking actions (\"saute vegetables\" vs. \"fry vegetables\"), giving two very similar dishes, having subtle differences in tastes. \n\nFrom the perspective of NLP, Procedural Reasoning is challenging, as entity states and actions across steps of a procedure might not be explicitly mentioned in a step/could require information present in a past/future step (see \"Case Study\" in Section 4.5.4). This requires NLP models to have world knowledge as well as commonsense reasoning capability.\n\nAlso, reasoning on Procedural text is essential for a myriad of other applications such as - (1) making a tutorial for understanding complex life processes and chemical reactions  (2) Helping robots perform a task step-by-step without any human intervention. (3) Question Answering on Instruction Manuals having instructions for operating different devices to help in troubleshooting and customer support.\n\n **2. How relevant and novel is the paper?**\n \n**Relevance of the paper** - As answered in the previous question, modeling procedural texts is extremely useful in today’s AI research landscape. Our paper discusses a pre-training pipeline customized to procedural text, making our work relevant to the conference, as well as to the NLP research community as a whole.\n\n**Novelty of the paper** - Our work is the first attempt at proposing a domain-specific, pre-training framework for procedural reasoning. We propose a never-before-tried combination of triplet loss (used in previous works) and masked-step-modeling (MSM is inspired by the notion of masked span modeling, as in SpanBERT, extended to accommodate the step-wise nature of procedures) in our proposed pre-training pipeline. Additionally, we fine-tune and evaluate such a pre-trained model on different datasets on procedural reasoning (for which such pre-training on a large corpus of procedural data has not been explored previously), and observe that our proposed approach beats many competitive baselines. Hence, from the viewpoint of (1) architecture, (2) loss functions, and (3) use of such a pipeline for procedural reasoning, we would argue that our proposed CLMSM is indeed a novel addition to the procedural reasoning paradigm in the research community.\n\n**3. Is the choice of baselines relevant to today’s day and age?**\n\nYes, the baselines we choose to compare CLMSM with, are indeed recent and competitive. For instance, 9 out of 11 baselines in the case of ProPara are no older than 5 years, and all the baselines have been shown to be relevant to procedural reasoning in several prior art. Also, the pool of baselines we use is diverse, with the following broad categories - (1) architecture specific to the procedural task at hand (E.g. ProLocal, ProGlobal) (2) model infused with external knowledge (e.g. KG-MRC) (3) baselines using pre-trained models (e.g. RECIPES_RB, RECIPES_RL). Hence, we feel that our choice of baselines is also comprehensive.\n\n**4. Did you compare CLMSM to off-the-shelf LLM in zero-shot or few-shot setting**?\n\nCLMSM is compared to several open-domain and domain-specific LLMs such as RoBERTa-BASE, RoBERTa-LARGE, BERT-BASE, RECIPES_RB, RECIPES_RL, RECIPES_BERT (Note: In the Wikipedia Article on \"Large language model\", under the \"List\" Section, BERT has been mentioned. Since RoBERTa has a similar architecture, it is also therefore an LLM. Subsequently, continually pre-training such a model would also result in an LLM). Since our work is the first work on domain-specific pre-training in the procedural reasoning paradigm, we keep the fine-tuning setup the same as the one followed by the corresponding dataset papers, to ensure equitable and fair comparison between CLMSM and other baselines.\n\nHowever, for completeness, we are adding some results of larger, instruction-tuned LLMs such as Falcon-7B (pre-trained LLM) and Falcon-7B-Instruct (instruction-fine-tuned Falcon-7B) [Penedo et al 2023] on the ProPara Dataset. Also, we use the few-shot (in-context learning) setting, as LLMs have shown remarkable results in such settings in a variety of NLP tasks.\n\n|  | Cat 1 Accuracy | Cat 2 Accuracy | Cat 3 Accuracy | \n| - | - | - | - |\n| Falcon-7B (1-shot) | 50.42 | 7.11 | 4.79 | \n| Falcon-7B (3-shot) | 50.85 | 7.73  |  5.3  |\n| Falcon-7B-Instruct (1-shot) | 50.42 | 5.42 | 0.38 | \n| Falcon-7B-Instruct (3-shot) | 48.44 | 3.15  | 1.94   |\n| CLMSM_RB | 69.92 | 39.35 | 33.89 |\n| CLMSM_RL | **77.26** | **54.86** | **38.34** |  \n\nWe observe that such an LLM, which has almost 6x and 14x the number of parameters compared to CLMSM_RB and CLMSM_RL, performs considerably worse in comparison. Thus, in-context learning using very large LLMs, which work in several NLP tasks, fails to match up with the performance of smaller fine-tuned models, such as our solution. This also goes on to show that the problem of procedural reasoning is indeed non-trivial. We consider the suggestion and would explore the performance of large LLMs on procedural reasoning, and potential ways to improve performance.\n\n**5. Paper is very dense. The liberal use of opaque acronyms, the general absence of helpful examples, the lack of clear definitions for the input units and outputs of the model, and interpretive commentary that fails to motivate why this is an important problem or why the proposed solution is a good one, means that it is a generally difficult paper to parse.**\n\nWe would like to address each concern of the reviewer (regarding the writing of the paper) as follows -\n\n- **Use of opaque acronyms** - The meaning/full-form of all acronyms (e.g. CL, MSM, MTL, etc.) used in the paper are written when they are defined for the first time, and most of the acronym expansions can be seen in the Introduction. For later usages of the acronym, we do not mention the full form any more to conserve the available paper space. In the final version, we will provide a table with expansion of all the acronyms for ready reference. \n\n- **General absence of helpful examples** - Table 1, Table 4 show examples of open-domain processes, and Figure 3 in the Appendix shows an example recipe. However, as per the reviewer’s suggestion we would add some example recipes in the main paper for the benefit of the reader.\n\n- **lack of clear definitions for the input units and outputs of the model** -\n\n| Training Phase | Task/Objective | Input Units | Output Units | Reference in  paper |  \n| - | - | - | - | - |\n| Pre-training | Contrastive Learning | sampled anchor, positive, and negative (step-masked) procedures | procedure representations, which are later utilized in calculating the triplet margin loss | Section 2.1, Fig. 1 |\n| Pre-training | Mask-Step Modeling | Procedure with masked step tokens | predicted mask tokens, which are learned via the MSM Loss | Section 2.2, Fig. 1 |\n| Fine-tuning | Entity Tracking | entity, step of the process | State of the entity | last paragraph of Section 4.1 |\n| Fine-tuning | Action Alignment | similar recipes | Aligned Actions across steps of the recipes | first paragraph of Section 5, and the same is explained through the use of an example in the following paragraph |\n\n\na. **During pre-training -** Sampled anchor, positive, and negative (step-masked) procedures are inputs, and the procedure representations are the outputs, which are later utilized in calculating the triplet margin loss in Equation 1 (this is clearly mentioned in Section 2.1 and in Fig. 1). Also, procedure with masked step tokens is input, and the outputs are the predicted mask tokens, which are learned via the MSM Loss (see Section 2.2 and Fig. 1). \n\nb. **During Fine-tuning -** for tracking entities, the inputs (entity and step of the process) and outputs (state of the entity) are formally defined in the last paragraph of Section 4.1; for aligning actions, the inputs (similar recipes) and outputs (Aligned Actions across steps of the recipes) are formally defined in the first paragraph of Section 5, and the same is explained through the use of an example in the following paragraph.\n\n- **interpretive commentary that fails to motivate why this is an important problem or why the proposed solution is a good one** - The answer to the first question shows why this is an important problem. The answer to the second question shows why the proposed CLMSM is novel. Also, Sections 4.4.3, 4.5.3, and 5.2 show that CLMSM outperforms relevant and competitive baselines. Hence, the proposed solution is satisfactory in our opinion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, the paper has a rigorous set of experiments and does a thorough job of testing the proposed model on several benchmarks and in several different settings.", "reasons_to_reject": "Unfortunately, a few issues prevent me from recommending this paper.\n\nMy biggest gripe is with the novelty and relevance of the paper. The method appears to be a combination of previously proposed approaches, including triplet networks and masked step modeling for procedural video understanding. This isn't, in and of itself, a disqualifying factor, but combined with a generally weak motivation it isn't clear why the community would benefit from adopting this method (or why modeling procedural texts is useful to begin with). Finally, combine this with the fact that the authors do not compare their approach to an off-the-shelf LLM, whether in zero-shot or few-shot setting, means that there is no way of knowing whether these findings (even if they appear to improve on some baselines), continues to be still relevant in today's day and age.\n\nMore generally, while the paper isn't poorly written, it is very dense. The liberal use of opaque acronyms, the general absence of helpful examples, the lack of clear definitions for the input units and outputs of the model, and interpretive commentary that fails to motivate why this is an important problem or why the proposed solution is a good one, means that it is a generally difficult paper to parse. Overall, I am left with an impression of an interesting model with some nice results but not much more.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper tackles the problem of modeling processes described in natural language, such as recipes or e-manuals etc. Specifically, it uses a pretraining objective that consists of two components: contrastive learning and masked step modeling. In contrastive learning, a hard input selection isused to teach the model to distinguish between identical vs. similar processes. Meanwhile, masked step modeling (much like masked language modeling), is used to train the model in capturing step-wise context of a process. The authors conduct a series of experiments on a couple of task specific problems, demonstrating that their models outperform several baselines.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oGT9auAwYu", "reviewer": "Reviewer_uD45", "summary": "", "questions": "A) Can you please provide more details on how you arrived at the values of statistical significance stated in the paper?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. The proposed approach requires four training stages: general-domain mask language modeling (MLM) pretraining, in-domain MLM pretraining, finetuning on the SQuAD 2.0 dataset (Rajpurkar et al., 2018), and finetuning on the downstream task. This leaves many levels of freedom regarding pretraining data and parameters, which in turn may result in brittleness for some domains or for a different choice of hyperparameters.**\n\nThe stage of general domain MLM is prior to our pre-training - we directly take off-the-shelf models such as RoBERTa-BASE.\nFollowing this, in-domain pre-training works on a massive amount of diverse pretraining data. Hence, it is less task-specific and less sensitive to hyperparameters (For reference, check Table 3 [Liu et al. 2019], the original paper where RoBERTa was introduced). The set of hyperparameters used is exactly the same as that of pre-training RoBERTa, as mentioned in its original paper. We would also like to mention that there is consistency in result (i.e.robustness) amidst plenty of variation in the architecture, loss function, token masking scheme, choice of triplets while pre-training (as shown in our ablation studies).\nFor both the fine-tuning tasks, we use the same set of hyperparameters as their corresponding original papers.\nThis shows the results obtained did not have to undergo any hyperparameter tuning and thus inherit the robustness (or brittleness) of the original modules which all other baselines also use. \n \n**2. Can you please provide more details on how you arrived at the values of statistical significance stated in the paper?**\n\nTo see if our results are statistically significant, we use the paired T-test [Student. (1908). “The probable error of a mean”. Biometrika, 6(1), 1-25.] to compare the results of our proposed solution with baselines, and observe that the p-score is less than 0.05, which intuitively means that our solution is performing significantly better than the corresponding baselines.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a pretraining scheme for two downstream tasks on procedural text which is able to outperform state-of-the-art baselines.\n- The authors conduct an extensive ablation study for each experiment, providing insights regarding contribution of individual parts of the proposed approach.\n- Besides measuring the test accuracy, the authors discuss the performance of individual baselines, providing more intuition regarding the outcomes.\n- The authors provide an anonymized version of the code used for training the models.", "reasons_to_reject": "- The proposed approach requires four training stages: general-domain mask language modeling (MLM) pretraining, in-domain MLM pretraining, finetuning on the SQuAD 2.0 dataset (Rajpurkar et al., 2018), and finetuning on the downstream task. This leaves many levels of freedom regarding pretraining data and parameters, which in turn may result in brittleness for some domains or for a different choice of hyperparameters.\n- No details are provided regarding which statistical test was used to compute the p-values.", "questions_to_authors": "A) Can you please provide more details on how you arrived at the values of statistical significance stated in the paper?", "paper_topic_and_main_contributions": "The paper proposes a pretraining framework for the tasks of (1) entity tracking and (2) action aligment in a procedural text. The paper focuses on the recipe domain (NPN-Cooking Dataset; Bosselut et al., 2017) with additional experiments on an open-domain dataset (ProPara Dataset; Mishra et al., 2018). Each example in the datasets is a procedure comprising of multiple steps. The model is required to (1) track the location and status of entities mentioned in the text (e.g., ingredients) or (2) align semantically equivalent steps in two similar procedures.\n\nThe pretraining framework proposed by the authors builds on a Triplet Network (Cohan et al., 2020; Ostendorff et al., 2022) with a BERT (Devlin et al., 2018) or RoBERTa (Liu et al., 2019) encoder. The loss for the network is a sum of two objectives: contrastive learning with positive and negative examples (in which the model is required to minimize the distance between representations of positive examples and vice versa), and mask-step modelling (in which the model is required to infill masked tokens of randomly masked step(s) in the procedure).\n\nThe authors conduct extensive experiments for both datasets and tasks, including multiple baselines and ablations of the proposed pretraining scheme. The results show the proposed approach is able to (significantly) outperform state-of-the-art baselines and generalize to open-domain tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Although I understand the appeal of Appendix containing all the corresponding sections for easier orientation, I'd suggest using a more standard version which would contain only the necessary sections."}}]

Author rebuttal:

Meta-review: The paper proposes a multi-task learning framework for continual training on procedural data, combining the methods of contrastive learning and mask-step modeling. 

The paper contains a thorough set of experiments and the final system achieves state-of-the-art results. 
There is also sufficient analysis and ablation experiments.

As a downside, the framework seems to be a combination of two existing methods, which limits its novel contribution. 
Relevance compared to zero-shot LLMs is left somewhat unclear.
Several reviewers point out that the paper is difficult to read and understand, containing dense text, many acronyms, and very few examples.
Decision: Accept-Findings