Evaluation
To first holistically evaluate the modelling of salience, we report pairwise Pearson’s correlation coefficients between each set of labels above and the annotations of human A1. In Table 7 , we report a “macro correlation”, namely the mean of correlation of salience scores in each procedure. 6 6 6 To avoid NaN due to constant input array, a 0 is appended to each array as smoothing. First, the correlation between the two annotators is high but imperfect, implying subjectivity in the annotation of entity salience. In comparison, the LM predictions come close with especially impressive predictions for global salience.

To understand when and how entity salience can be subjective among humans, in Figure 3 we show salience correlation for the first 20 procedures. Some with low inter-human correlation such as “Defeat a Thug” expose a document-frequency problem: one human labels the entity you (actor) with a salience of 5, believing that without the actor the procedure cannot be undertaken at all, while the other labels 1, believing that the actor occurs in every procedure and thus the mention of it makes no difference. Other procedures such as “Make Taco Filling” expose a cluster-individual problem: all individual ingredients are grouped under one cluster, for which annotators are divergent on the salience. Both issues give rise to subjectivity and can be mitigated by better defining entity salience in a potential future annotation process.

Role Content User A person walks into a dark room in a freezing windy night . In the very beginning before anything is done, how likely is it that she can see what’s in the room? Answer on a scale of 1-5, with a 1-sentence explanation. Assistant 1 - it is highly unlikely for the person to see anything in a dark room without any source of light. User Next, she lights up a torch and hangs it near a window. At this point, how likely (same question)… Assistant 5 - hanging the torch near a window maximizes the distribution of light in the room, greatly enhancing the visibility and increasing the likelihood of seeing what’s inside. 1 User Next, she opens the window . At this point, how likely (same question)… 1 Assistant 5 - opening the window allows natural light to enter the room, combining with the torchlight to provide ample illumination, greatly increasing the likelihood of seeing what’s in the room. 2 User Next, she opens the window. The most salient entities for this step are the window, the wind, and the flames. How does each of these entities change? 2 Assistant When she opens the window, the following changes occur: Window: The window is now open , allowing for air and potentially some outside light to enter the room. Wind: The wind becomes more noticeable and enters the room through the open window , causing a draft and potentially affecting the flames of the torch. Flames: The flames of the torch may flicker or change direction due to the entry of wind through the open window, altering their shape and intensity . 2 User With such information, at this point, how likely is it that she can see what’s in the room? 2 Assistant 3 - opening the window allows some outside light to enter, the presence of wind and potential disruption to the flames may still impede visibility to some extent.