dataesr/scientific-paragraphs-categorization huggingface dataset:
Contributions: 1. we propose a methodology to adapt TLMS to the dynamic estimation of thematic fit, using a dataset that contains several types of argument combinations differing for their typicality; 2. we present a comprehensive evaluation of various TLMS on this task, performed by comparing them to a strong distributional baseline; 3. we conduct further analysis aimed at identifying the potential limitations of TLMS as models of GEK. Our results are relevant for researchers interested in assessing the linguistic abilities of TLMS, as well as those working on applications involving TLMS, such as text generation.
Notice that the two sentences differ only for one argument, and that the "atypical" one might, however, be a common filler with respect to the verb target role (e. g. , report is a typical patient for check, it is just less plausible in combination with mechanic as an agent). Several models have tried to tackle the "dynamic" version of the thematic fit task, either based on classical distributional spaces  (Chersoni et al. , 2016 (Chersoni et al. , , 2019) )  or on more sophisticated neural network architectures  (Tilk et al. , 2016; Hong et al. , 2018). On the evaluation side, those works made use of the experimental materials of the study by  Lenci (2011) , which are, however, limited to agentverb-patient triples. The recently-introduced DT-Fit dataset  (Vassallo et al. , 2018)  is, in comparison, larger in size and provides more variety of fillers and roles (including instruments, locations and time).
Dataset The DTFit  (Vassallo et al. , 2018)  dataset has been specifically designed for the evaluation of dynamic thematic fit. 1 The dataset contains pairs of tuples that differ only for one element, which can be either a typical or atypical filler of a given role in the event described by the tuple (cf. Table  1 ). The dataset includes tuples of different lengths, and the typicality of a given argument depends on its interaction with all the other elements. For each tuple, the authors collected typicality judgments by asking English native speakers how common was the event described. Scores range from 1 (very atypical) to 7 (very typical). The dataset mainly targets knowledge about professions, but also other typical everyday situations (e. g. , what a dog typically eats, what a grandmother typically does).
The authors created several datasets, which differ with respect to the semantic role of the candidate filler. For our experiments, we selected the datasets created by the authors for the following relations: {Instrument, Time, Location} DTFit. Additionally, from the original dataset containing agent-verb-patient triples, we derived two datasets, that we named Agent DTFit and Patient DTFit. In Agent DTFit , the tuples forming a pair differ with respect to the typicality of the agent. In Patient DTFit , they differ for the typicality of the patient. We thus get a total of five datasets, each of which covers a different semantic relation. The latter two datasets have the same properties of the others, but they put stronger emphasis on the dynamic nature of thematic fit, as the atypical filler is still a typical complement of the verb alone.
Conversely, the atypical candidate fillers in the other datasets are appropriate fillers of the role, but, in most cases, they do not relate to the other elements of the tuple. Therefore, Agent DTFit and Patient DTFit are more challenging for computational models, as the typicality of a filler can only be determined through the composition of the verb with another argument. Accordingly, models have to update their predictions by accurately taking into account the whole context. For each tuple in DTFit, the task for our models is to predict the upcoming argument on the basis of the previous ones. Models were evaluated in terms of Spearman correlation between the human ratings and the models' scores. Moreover, we performed a second evaluation for Agent DTFit and Patient DTFit , consisting of measuring the accuracy of each system in assigning a higher thematic fit score 1 All the datasets used for the experiments described in this paper can be found at the link: https://github.
, 2009) , a dump of Wikipedia 2018 and the British National Corpus  (Leech, 1992). The final graph contains words with a minimum frequency of 300 and events with a minimum frequency of 30. We used as lexical embeddings the publicly-available FastText vectors extracted from Wikipedia. 2  For our experiments, we built a semantic representation for each tuple in the dataset, like in  Chersoni et al. (2019). We used the information in LC and AC to assign a typicality score to each candidate filler of a role in the dataset. The scoring function for a given role filler is the following: cos( f , LC(sent)) + cos( f , AC(sent)) 2 (1) where f is the embedding of the candidate filler; LC(sent) is a vector obtained from the sum of the embeddings of the verb and of the argument other than f ; AC stands for the updated expectation prototype for the role filled by f.
Transformer-based Language Models We experimented with four TLMS to test how different architectures, training objectives, and sizes of the training corpus affect performance. 3 BERT  (Devlin et al. , 2019)  consists of a series of stacked Transformer encoders. It was trained using both a masked language modeling objective (i. e. , predicting a masked word from its left-and rightcontext), and a next sentence prediction objective (i. e. , whether a sentence follows another sentence or not), on a combination of the BooksCorpus and English Wikipedia (13GB in total). The model uses WordPiece vocabulary. To test if the model size can affect BERT performance, we used both the base (Number of layers=12, Hidden size=768) and the large (L=24, H=1024) versions. RoBERTa  (Liu et al. , 2019) , which we used in the large version, is based on the same architecture as BERT, but it was trained on a much larger corpus (160GB) and without the next sentence prediction objective.
In our experiments, we used the large version (L=24, H=1024). In contrast with the bidirectional nature of BERT and RoBERTa, GPT2  (Radford et al. , 2019 ) is a uni-directional LM, which means that the training objective is to predict the next word, given all of the previous ones. It was trained on WebText, for a total of 8 million documents of data (40 GB). We employed the medium version of GPT2 (L=24, H=1024). We chose GPT2-medium since its dimensions are comparable to those of BERT and RoBERTa large. Moreover, both RoBERTa and GPT2 make use of a Byte-Pair Encoding tokenizer. For our investigation, we designed the experiment as follows. First, we derived simple sentences from the tuples by adding definite articles to the words, [CLS] at the beginning of the input and a period to signal the end of the sentence (e.
g. ,  [CLS]  The tailor sewed the dress. ). Then, we masked the candidate filler (dress in the example) and we computed the probability distribution of the entire model's vocabulary for that position. The model typicality score is the probability assigned to the candidate filler, when the candidate filler is included in the model's vocabulary. In case a word to be scored is not included in the vocabularies of all the models that we used, we decided to disregard its tuple and the respective typical/atypical counterpart. For this reason, the final results only take in consideration a subset of the original datasets, which varies from model to model. Additionally, we computed a baseline for each Transformer model, where the model is prevented from attending to the other tokens in the sequence when making predictions.
Results and Analysis In this section, we provide the results of the experiments on the DTFit datasets. Since the models cover different portions of the original tuples, we performed the evaluation over the common pairs. Table  2  reports the correlation scores for all the five datasets. 4 Values in brackets refer to the Spearman correlation obtained by the baseline. As the baseline scores are very similar across models, we reported the results only for BERT-base. At a glance, we observe that both SDM and TLMS obtain quite strong correlations, going from 0. 46 to a maximum of 0. 74 across datasets and models. Specifically, we notice that TLMS tend to reach higher performances compared to the distributional approach. However, a marginally significant improvement of the correlations over SDM is obtained only for Location DTFit (p < 0.
Do Transformers Really Encode GEK? The above results prima facie suggest that TLMS are able to model the dynamic interaction between the sentence elements to compute the typicality value of a candidate filler. However, analyzing the errors of the TLMS can be revealing of how they make their predictions. Table  4  presents some of the Patient DTFit pairs where BERT-base prefers the atypical filler. In all these cases, BERT simply seems to rely on frequent verb objects, without composing and integrating the verb expectations with information from other elements of the context (the agent in this case), which is a key aspect of human GEK and is reflected in the typicality judgments. However, we cannot make any claims about the event knowledge of TLMS from these examples alone, as only in some cases (such as The cat drank the coffee) the atypical tuples evoke events unlikely to take place in the real world (i.
e. , it may happen frequently that a chemist pours the juice, even if this is not a typical action for a chemist). To better understand if this can lead TLMS to make really implausible predictions, we carried out an additional experiment where we tested the models on a diagnostic dataset controlled for the frequency of the association between the verb and the filler. In this experiment, we also tried to address the question of whether TLMS rely more heavily on the local context when making predictions. Furthermore, TLMS' natural preference for what is more frequent could help them in the typicality task, as a typical event is often a frequent one. Their good performance could be due to the fact that they memorize frequent sequences during training. Therefore we tested TLMS on a different dataset, in which atypical but physically plausible events (e.
g. , The cloth erased the cream) are distinguished from atypical and implausible ones (e. g. , The cloth erased the house). Frequency effects on performance should be alleviated in this setting, as both types of events in the dataset are atypical and, hence, rare. This task requires fine-grained knowledge of the properties of arguments, which is still an important component of GEK. Additionally, different frequency variations in the training data could influence TLMS performance. Since the models' knowledge of the world is mediated by language, it is likely that an argument filler may or may not be predicted depending on the frequency of the word chosen to refer to it. We investigated this issue by testing the models on another diagnostic dataset obtained by replacing typical fillers with low-frequency synonyms.
The last question we explored is whether TLMS can be influenced by the way statements of event typicality are syntactically expressed. So, we evaluated TLMS by feeding them with sentences encoding typical events with a transformed and more complex syntactic form than the one used in the DTFit experiments. I. TLMS seem to prefer frequent collocations, but only when they are plausible. Errors reported in Table  4  suggest the tendency of TLMS to predict frequent complements of the verbs, irrespective of whether they are coherent with the rest of the tuple. We questioned to what extent salient local word co-occurrences make the models "blind" to the rest of the context and thus compromise the plausibility of their predictions. To investigate this behavior, we generated a new diagnostic dataset.
The dataset is a small (31 pairs) subset of Patient DTFit , where the atypical filler in each pair was replaced with another noun that has a very strong association with the verb in the tuple. We computed the association between the verb and its direct objects using Local Mutual Information (LMI)  (Evert, 2008). Since LMI is computed by multiplying the Pointwise Mutual Information and the frequency of the two words in a grammatical relation, it assigns higher values to combinations that are both common and informative. We chose the new atypical fillers among the words with the highest LMIs. We chose words that give rise to odd events when integrated with the rest of the context. To evaluate BERT performance, we computed the accuracy scores on the diagnostics dataset in the same way as in the main experiment (binary classification task).
Results show that the models generally assign low probabilities to atypical fillers. They choose the atypical event in some cases (9 in BERT-base, 6 in large), but mainly when the contrast between the atypical event and our expectations is less evident (The smuggler sold the property is preferred to weapon, The soldier throw the ball is preferred to bomb). As already observed in the main experiment, BERT seems to be able to look beyond salient local associations and build representations of global events flexibly. However, this issue should be further explored for the other roles as well. For instance, given the sentence The engineer completed the project in the , the models must consider more contextual elements to make the correct prediction. On the other hand, even if SDM design aims at capturing this aspect of GEK, the manipulations we made in this dataset cause a drop in the model performance (14 pairs out of 31 are classified wrongly).
II. TLMS know more about what is typical than what is possible. The use of typicality datasets such as DTFit for the estimation of the models' GEK has some limitations. TLMS' ability to reproduce combinations encountered frequently during training could be the reason for high performances in the typicality task, since what is most typical often occurs most frequently. However, GEK is not just memory of exemplars, but it requires fine-grained knowledge of the properties of objects and it involves reasoning processes such as abstraction and comparison between objects and prototypical concepts. To evaluate TLMS on a setting where frequency variations in the training corpus have a minor impact, we used the dataset realized by  Wang et al. (2018) (henceforth, Wang2018). This dataset represents a benchmark for the task of semantic physical plausibility  (Bagherinezhad et al.
, 2016) , that is, distinguishing an atypical but physically plausible event such as The student climbed the ship from an atypical and physically implausible one such as The student climbed the water. The dataset contains agent-verb-patient (SVO) triples divided into plausible and implausible. From the original dataset, which contains 1, 540 plausible and 1, 540 implausible triples, we derived two subsets containing pairs of triples differing either for the agent or for the patient role filler (obtaining 222 and 394 pairs respectively). Table  3  reports the resulting accuracy values. In general, the models' scores are lower than in the typicality task (min. 0. 64, max. 0. 77), and in some cases they are not much higher than random performance. Moreover, in many cases the models could be facilitated by the existence of an association between the plausible filler and the verb of the event, as in The ant built the wall and in The chair absorbed the water.
Nevertheless, the results demonstrate that the notion of plausibility is harder to model compared to typicality, and invite caution when making claims about TLMS world and event knowledge. In fact, the results suggest that even if it were true that TLMS develop some generalization skills from training, they still miss many predictions about possible events, which instead humans easily make on the basis of their commonsense knowledge. This dataset is also difficult for SDM, which obtains scores lower than those of the TLMS (0. 65 for Agent and 0. 66 for Patient). Even if SDM should be better at reproducing generalization through the construction of prototypical fillers, the model's distributional representations seem to fail to capture the specific properties that are relevant for the dataset items, namely physical properties of objects  (liquid-solid, large-small, etc.
). The lack of such properties constitutes a limitation of distributional models of word meaning based on text data only, which is why, in previous studies, world knowledge was explicitly injected into the models for the physical plausibility task  (Wang et al. , 2018). III. TLMS do not extend fit judgments to low frequency synonyms. To test whether TLMS consider an entity more or less likely to take part in an event depending on the word used to refer to that entity, we evaluated them on a new diagnostic dataset of 39 pairs, generated from a subset of Patient DTFit. In this setting, the typical filler in each pair was replaced with a low-frequency word that is semantically related to the original one. To choose an appropriate substitute, we first extracted a set of synonyms according to two lexical resources  (WordNet, Lexico.
com). Then, we picked a word that 1) is less frequent than the original filler and 2) has a frequency lower than 300, 000. For the same reasons described in the first additional experiment, we extracted statistics from a 2018 dump of English Wikipedia and evaluated only BERT on the new dataset. Examples of substitutions are the following: The botanist examined the plant → flora, The waiter cleared the restaurant → tavern, The veterinarian examined the dog → puppy. It is interesting to observe that these variations pose serious difficulties to the models, as their accuracy on the diagnostics dataset is close or lower to the random level (BERT-base: 0. 37, BERT-large: 0. 53). For example, BERT considers The terrorist released the captive as less probable than The terrorist released the /book, and the same occurs for The mother prepared the provisions/gun, and The carver built the bust/house.
These results cast doubts that current TLMS can constitute plausible models of event knowledge: they tend to reproduce the patterns that are frequently observed in the data, and their good performance is disrupted once these are replaced with semantically equivalent, but less frequent ones. This means that they lack the abstract semantic knowledge of human subjects, whose predictions are more flexible thanks to inference mechanisms such as generalization to concepts sharing semantic features. At least in principle, models aiming at building prototypes of ideal role fillers (such as the distributional models of thematic fit) are more cognitively realistic, since they are less dependent on specific words. However, they may still show sub-optimal performance in this diagnostic dataset as they are based on the quality of the distributional representations, which is lower for words that have low frequency in corpora.
This is confirmed by the performance of SDM on the dataset (the accuracy is 0. 51). IV. TLMS can be influenced by the surface structure of sentences Finally, we analyzed to what extent TLMS' ability to predict the fit of a word in a given role arises from the observation of recurrent word order patterns during pre-training (e. g. , the fact that an actor's award-winning event is canonically expressed with active sentences, in which award follows the words actor and won), rather than being based on a deep understanding of the semantic relations between the words. To explore this issue, we modified DTFit tuples to create two different versions of the dataset, each with examples of a syntactic construction different from the English canonical word order. Specifically, we experimented with cleft (It was the award that the actor won, It was on the ring that the boxer delivered the punch) and wh-interrogative sentences (Which award did the actor win?, On which ring did the boxer deliver the punch?).
Conclusions In this paper, we tested Transformer-based Language Models on tasks related to Generalized Event Knowledge. In the main experiment, we evaluated their ability to model event typicality, that is, discern typical from atypical events, on a dataset designed for this task, DTFit. Results show that TLMS scores positively correlate with human judgments. However, they do not significantly outperform the distributional prototype-based model (SDM) that we selected for comparison. This confirms the ability of SDM to dynamically update the semantic representation of a sentence, which was recently shown for the challenging task of logical metonymy interpretation  (Rambelli et al. , 2020). However, we decided to go beyond the simple evaluation against human judgments. We carried out several additional small-scale experiments with the specific aim to understand which factors could affect the predictions of TLMS.
e. δx(0) = 0) and normalized by p 0 • δp(0) = 0 (since p 0 is restricted to S 1 ), then t c is a conjugate time if and only if t c is a solution of the following equation: t → det(δx(t), ẋ(t)) = 0. ( 17 ) See  [7, 18]  for more details about algorithms to compute conjugate times in a more general setting and  [13]  for details about the numerical implementation of these algorithms into the HamPath software.
Resolution of the shooting equation We introduce the shooting mapping S(T, p 0 ) := exp x0 (T, p 0 ) -x f , (19) where x f is the target and exp is the exponential mapping defined by  (13). The shooting mapping is defined for (T, p 0 ) ∈ (T, p 0 ) ∈ R + × S 1 T < t p0 where t p0 ∈ R * + ∪ {+∞} is the maximal time such that exp x0 (•, p 0 ) is well defined over [0 , t p0 ). Let (T, p 0 ) be a solution of S = 0 such that the associated extremal is normal. The shooting mapping is differentiable at (T, p 0 ) and if T is not a conjugate time, then its Jacobian is of full rank at (T, p 0 ), which is a necessary condition to compute numerically the BC-extremals by means of Newton-like algorithms. We present in the following some examples of hyperbolic geodesics fixing the initial condition to x 0 := (2, 0) and solving the shooting equations S = 0 thanks to the HamPath code  [13] , for different final conditions and for different strengths of the drift.
HamPath code. A Newton-like algorithm is used to solve the shooting equation S(T, p 0 ) = 0. Providing H and S to HamPath, the code generates automatically the Jacobian of the shooting function. To make the implementation of S easier, HamPath supplies the exponential mapping. Automatic Differentiation is used to produce #-H and is combined with Runge-Kutta integrators to assemble the exponential mapping and the variational equations (  16 ) used to compute conjugate times. See  [13, 18]  for more details about the code. Example 1. For this first example we want to steer the particle from x 0 to x f := (-2, 0) with µ := 2 x 0 (strong drift). In this case, we obtain a final time T ≈ 1. 641 and the shooting equation S = 0, is solved with a very good accuracy of order 1e -12 (it is the same for the others examples but it won't be mentioned anymore).
Numerically, we compute the splitting line with the differential path following method (or homotopy method) of the HamPath software. Under some regularity assumptions, the set F -1 split (0) is a disjoint union of differential curves, each curve being called a path of zeros. To compute a path of zeros, we look for a first point on the curve by fixing the homotopic parameter λ to a certain value λ and calling a Newton method to solve F split (•, λ) = 0. Then, we use a Prediction-Correction (PC) method to compute the differential curve. The HamPath code implements a PC method with a high-order Runge-Kutta scheme with variable step-size for the prediction, hence the discretization grid of the homotopic parameter is computed by the numerical integrator. Besides, the Jacobian of F split is computed by automatic differentiation combined with variational equations of the exponential mapping.