dataesr/scientific-paragraphs-categorization huggingface dataset:
Results Empirical evidence. We built our empirical R&D network using the SDC Platinum database  21  that reports approximately 672,000 publicly announced alliances in all countries, from 1984 to 2009, with a granularity of 1 day, between several kinds of economic actors (including manufacturing firms, investors, banks and universities). We then select all the alliances characterized by the ''R&D'' flag; after applying this filter, a total of 14,829 alliances, connecting 14,561 firms, are listed in the dataset. An R&D alliance is defined as an event of partnership between two or more firms, that can span from formal joint ventures to more informal research agreements, specifically aimed at research and development purposes. Every firm listed in the dataset is associated with a SIC (Standard Industrial Classification) code -a US-government system that allows us to univocally assign each firm to its corresponding industrial sector.
We employ this sectoral classification to test the invariance and robustness of some empirical features of the R&D network across sectors. A salient feature of the R&D alliances in the SDC dataset is the variable number of partners they involve. Most of the collaborations (93%) are stipulated between two partners, but some alliances -the so-called consortia -involve three or more partners. The distribution of the number of firms per alliance event, as shown in Fig. 1 , spans one order of magnitude and is right-skewed. This feature holds independently of the industrial sector to which the alliance partners belong (see the Supplementary Information, SI). In our network representation, we draw an undirected link connecting two nodes every time an alliance between the two corresponding firms is announced in the dataset.
When an alliance involves more than two firms, we assume that all the corresponding nodes are connected in pairs, forming a fully connected clique. This choice derives from the fact that consortia, although representing only a minority of the alliances, require great coordination and resource availability from the partners. More precisely, following this procedure we obtain a total of 21,572 links from the 14,829 alliance events listed in the dataset. However, in the definition of our model, we do not make any difference between a consortium and a ''standard'' two-partner alliance, which is only a special case of it (and can be thought of as a fully connected clique of size 2). Another distinctive measure we introduce and analyze in this study is the firms' activity distribution  22. Developed in the field of temporal networks  23  , the activity has been studied on various datasets, such as coauthorship, online microblogging, or actor/movie networks.
We report our finding in Fig. 2. The firm activity distributions are right skewed and dispersed over several orders of magnitude, as in many other social and technological systems  [24] [25] [26]. Contrary to most of the R&D network indicators, that display strong variability and dependence on time  6  , the activity is a stable attribute that can be assigned to firms and effectively estimate their prom-  inence. Indeed, empirical firm activities are robust with respect to the time t at which they are measured: shifting the time window -of any length Dt -along the 26 years reported in the dataset does not affect the results (see SI). In addition, we find that the activity distribution is robust to the sectoral classification of the firms (see SI). The empirical R&D network, as well as the networks we generate by means computer simulations, are then characterized with respect to the three following measures:.
570, with an extremely small variance of 0. 001. Such a result constitutes a significative indicator of modularity in the R&D network. The model. The empirical observations of the R&D network indicate clear heterogeneities in activity and connectivity patterns, smallworld features, a moderate level of transitivity, and a highly modular structure, not simply associated to the firms' industrial sectors. Starting from this evidence, here we present a novel modeling framework for R&D networks. We consider a network composed of N nodes; each of them is endowed with two fundamental attributes, an activity and a label. Such attributes define the nodes' interaction rules, which are organized in five distinct phases, as described below: Node activation. We assign to each of the i 5 1, …, N nodes an activity a i , analogous to the empirical activities extracted from the SDC dataset.
Indeed, we sample without replacement all the values a i from the empirical activity distribution. The activities we assign to the N nodes are computed by considering the entire observation period 1984-2009 (therefore a i :a Dt~26years i,t~2009 ). Given the strong robustness of empirical activities to the time window, we decide to use the 26-year window, because it contains complete information about the dataset and gives activities a i that are always strictly greater than 0 (all firms listed in the SDC dataset, by definition, must be involved in at least 1 alliance). The activity defines the propensity of each node to be involved in an R&D alliance event. We use this quantity to model the activation probability of each firm. In particular, at every time step, a node i initiates an alliance with probability p i 5 ga i Dt, and the number of active nodes N A is N A ~g a h iNDt, ð2Þ where AEaae is the average node activity and g is a rescaling factor that allows to adjust the activation rates, and consequently the number of active nodes per time step.
We find that the model is strongly robust to the choice of g, showing no measurable changes for g ranging from 10 25 to 1; however, we fix g 5 0. 01 to obtain N A roughly equal to 1. 5, the number of alliance events per day actually reported in the dataset. Without loss of generality, we fix Dt 5 1. Selection of the alliance size. When a node gets activated, it selects the number of partners m with whom the alliance is formed. We assume that the value of m is totally independent of any characteristic of the active node: we sample it, without replacement, from the empirical distribution of number of partners per alliance. In other words, we shuffle the sequence of number of partners per alliance (directly measured from the dataset) and then extract a value every time an activation event occurs; m can be thought of as the number of partners involved in every alliance event, diminished by 1, because the active node is not counted twice.
Model implementation. We perform extensive computer simulations by applying the above-described model and varying the values of its independent parameters. We fix the model parameters that we can directly measure from the data, namely the number of agents N 5 14, 561, the distribution of the node activities a i , and the distribution of number of partners m per alliance event. We stop every computer simulation when the total number of formed alliances equals the number of alliance events reported in the SDC dataset, E 5 14, 829. We vary the values of p L s , p L d and p NL nl in discrete steps spaced by 0. 05, in the interval (0, 1). The parameters p L s and p L d are bounded by the condition p L n ~1{p L s {p L d §0, meaning that their sum has to be smaller or equal to 1. This condition translates into 3,420 points to explore in the 3-dimensional parameter space, for each of which we run 200 simulations (for a total of 684,000 runs).
We argue that this highlights how effectively the label propagation mechanism can model the expansion of the firm circles of influence within the R&D network. Finally, we reproduced the distribution of path lengths between every pair of nodes at the moment of link formation, clearly indicating the goodness of our model at capturing the microscopic rules driving the alliance formation between real firms. Although the model captures many features of the empirical R&D network, it can be further improved to account for other real world observations. One of the limitations is assuming fixed node labels; this condition could be relaxed by introducing a label decay, representing the exit of a firm from its circle of influence. Such an extension might be useful especially when validating a dataset with a longer time extension.
SQUID magnetometry The temperature dependent magnetization was measured with an MPMS XL-7 Quantum Design SQUID magnetometer on the polycrystalline sample in the range of T = 2 to 300 K at B = 0. 1 T, and the isothermal magnetization was measured at T = 2 and 4. 6 K up to magnetic field B = 7 T. Experimental data were corrected for the diamagnetism of the constituent atoms by using the Pascal constants and for the diamagnetism of the sample holder. Pulsed magnetic field measurements Pulsed high-field magnetization measurements were performed using a 1. 5 mm long susceptometer coil with a 1. 5 mm diameter bore made of 50 gauge, high-purity copper wire of approximately 1500 turns. 41 Approximately the first 1000 inner turns are in the opposite direction to the final outer 500 turns such that the coil is compensated, and when a magnetic field is passed through the coil with sample absent, the induced emf and current in the inner turns is cancelled by that induced in the outer turns.
The sample is placed inside the coil in a 1. 3 mm diameter ampoule and the induced voltage V / dM dH Á dH dt ¼ dM dt. Numerical integration with respect to time, t, yields M. In order to ensure accurate values of M, the sample can be moved in and out of the coil, allowing empty coil data to be subtracted from sample data under identical conditions. Samples are placed within a  3  He cryostat to allow low-temperature studies down to approximately 0. 4 K to be performed. A ten-turn coil accompanies the susceptometer into the cryostat to measure the applied field, B. The voltage induced in this coil is V / dB dt and numerical integration yields B. Calibration of the instruments is accomplished using the de Haas-van Alphen oscillations of the belly orbits of the copper of the susceptometer. The method has been calibrated using all-ferric [Fe 8 ] 0 compounds (ESI, S5 †).
e. , the analytical form for symmetric cases in eqn (  1 ) is no longer appropriate). E + -E -in this case can be obtained from the separation of the Kohn-Sham eigenvalues in the neutral (all-ferric) BS6 state. The full set of spin Hamiltonian parameters that emerges from this analysis is (further details are given in ESI, S2 †): J 1 ¼ À5:9 cm À1 ; J 2 ¼ À10:1 cm À1 ; J 3 ¼ À55:1 cm À1 ; Δ ¼ 0:7 cm À1 ; t 1 ¼ À1438 cm À1 ; t 2 ¼ 1784 cm À1 The J 1 and J 3 values, corresponding to the Fe c III -Fe c III and Fe c III -Fe o III interactions, are very similar to those reported previously for the all-ferric analogue, both from DFT (-6. 3 cm -1 , -52. 8 cm -1 ) and from best fit to the SQUID magnetometry data (-2. 1 cm -1 , -50. 6 cm -1 ). 27 lemente-Juan et al. have noted that the exchange part of the H(8) Hamiltonian (eqn (2)) can be made independent of the position of the itinerant electron by assuming the isotropic exchange to be independent of the oxidation state (i.
The M mol (B) curves measured using either SQUID or pulsed high-field magnetometry (Fig. 3 ) offer a rather different perspective: the agreement between experiment and the curve generated using t 2 = 1784 cm -1 is strikingly poor. In the lowfield region sampled by both the pulsed high-field measurements and SQUID magnetometry (up to 7 T, see the expanded plot), M mol is rather insensitive to the value of t 2 in the range 0-1500 cm -1 because only the ground doublet state is appreciably populated under these conditions (the curves for values of 1250 cm -1 and smaller lie directly below the red curve at 1500 cm -1 ). Based on the low-field data alone, therefore, we can do no more than place an upper limit of ∼1500 cm -1 on t 2 at 2 K. The high-field data, however, pinpoint the crossover field B c , the point where the S = 3/2 state becomes the ground state, at ∼63 T, a value that is consistent only with t 2 values approaching zero (where B c = 60.
In a case study on presenting restaurant information, we show that an optimized IP strategy trained on Wizard-of-Oz data outperforms a baseline mimicking the wizard behavior in terms of total reward gained. The policy is then also tested with real users, and improves on a conventional hand-coded IP strategy used in a deployed SDS in terms of overall task success. The evaluation found that the trained IP strategy significantly improves dialogue task completion for real users, with up to a 8. 2% increase in task success. This methodology also provides new insights into the nature of the IP problem, which has previously been treated as a module following dialogue management with no access to lower-level context features (e. g. from a surface realizer and/or speech synthesizer). Index Terms-Information presentation, natural language generation, natural language user interfaces, reinforcement learning.
II. PREVIOUS WORK: INFORMATION PRESENTATION IN SDS Work on evaluating SDS suggests that the information presentation (IP) phase is the primary contributor to dialogue duration  [27] , and as such, is a central aspect of SDS design. During this phase the system returns a set of items ("hits") from a database, which match the user's current search constraints. An inherent problem in this task is the trade-off between presenting "enough" information to the user (for example helping them to feel confident that they have a good overview of the search results) versus keeping the utterances short and understandable, see for example  [28] ,  [29]. Broadly speaking, IP for SDS can be divided into two main steps: 1) IP strategy selection and 2) Content or attribute selection, which again can be divided into attribute ranking (according to a user model) and the number of attributes to be selected.
In this work we will concentrate on strategy selection as well as selecting the number of attributes from a list of ranked attributes, which we assume to be already provided by a given user model. Prior work has presented a variety of IP strategies for structuring information (see examples in Table  I ). For example, the SUMMARY strategy is used to guide the user's attention to a relevant subset of search results, i. e. it draws the user's attention to relevant attributes by grouping the current results from the database into clusters, e. g. [1] ,  [2]. Other studies investigate a COMPARE strategy, e. g. [30] ,  [31] , while much work in SDS uses a RECOMMEND strategy, e. g. [4] ,  [32]. In a previous proof-of-concept study  [19]  we showed that each of these strategies has its own strengths and drawbacks, dependent on the particular context in which information needs to be presented to a user.
A. Worked Example The input to the NLG planning module is a Communicative Goal supplied by the Dialogue Manager. The CG consists of a Dialogue Act to be generated, for example ( ), and a System Goal ( ) which is the desired user reaction, e. g. to make the user choose one of the presented items ( ). The RL-NLG module must plan a sequence of lower-level NLG actions that achieve the goal (at lowest cost) in the current context. The context consists of a user (who may remain silent, supply more constraints, choose an item, or quit), and, in the case of fully stochastic systems, variation from a stochastic sentence realizer, e. g. systems such as SPaRKy  [39]. Now, let us walk through one simple utterance plan for IP strategy selection only, as carried out by this model, as shown in Table  II  and Fig.
IV. WIZARD-OF-OZ DATA COLLECTION In a previous proof-of-concept study  [19]  using two data sets made available by the MATCH project, see  [40]  and  [41]  we showed that each of the IP strategies in Table  I  has its own strengths and drawbacks, dependent on the particular context in which information needs to be presented to a user. In the following we explore possible combinations of the strategies in a WoZ data collection. In this WoZ study, we asked humans (our "wizards") to produce appropriate IP actions in different dialogue contexts, when interacting with other humans (the "users"), who were told that they were talking to an automated SDS. The wizards were experienced researchers in SDS and were familiar with the search domain (i. e. restaurants in Edinburgh). They were instructed to select IP structures and attributes for NLG so as to most efficiently allow users to find a restaurant matching their search constraints, where we used a database of 340 Edinburgh restaurants 2  provided by TheList 3.
The task for the wizards was to decide which IP structure to use next (see Section IV-B for a list of IP strategies to choose from), how many attributes to mention (e. g. cuisine, price range, location, food quality, and/or service quality), and whether to stop generating. Our hypothesis is that these choices depend on varying numbers of database matches, varying prompt realizations, and varying user behavior. Wizard utterances were synthesized using the Cereproc text-to-speech engine  4. The user speech input was delivered to the wizard using Voice Over IP. In order to make the wizards' decisions comparable with real system decisions, the wizard only sees what the systems would have as input, i. e. the experimenter will listen to the user's utterance and transcribe it into a simple semantic representation of slot-value pairs.
We also experimented with introducing noise to simulate speech recognition errors, following a similar method to  [49]. However, noise did not seem to have a significant effect on either wizard or user behavior  [50]. Fig. 3  shows the web-based interface for the wizard. The wizard GUI contains 5 main panels: A: The wizard receives the user's query as attribute values. The experimenter has a similar input panel. There are 5 searchable attributes in total, which can also be negative ("not expensive"). B: The retrieved database items are presented in an ordered list. We use a user modelling approach for ranking the restaurants, where attributes are ranked according to user preferences. We assume a default user who cares about cheap food with high quality and good service. C: The wizard then chooses which strategy and which attributes to generate next, by clicking radio buttons.
A. Experimental Setup and Data collection We collected 213 dialogues with 18 subjects and 2 wizards  [51]. Each user performed a total of 12 tasks, where no task set was seen twice by any one wizard. Note that 3 dialogues were discharged due to technical difficulties. The majority of users were from a range of backgrounds in a higher education institute, in the age range 20-30, native speakers of English, and none had any prior experience of spoken dialogue systems. After each task, the user answered a questionnaire on a 6 point Likert scale, regarding the perceived generation quality in that task. The wizards' IP strategies were highly ranked by the users on average (4. 7), and users were able to select a restaurant in 98. 6% of the cases. Also, no significant difference between the wizards was observed.
", "The system's utterances had the right length. ", "The system gave me a good overview of all the available options. ". A full list of annotated features is described in  [50]. 5 The full corpus and a list of features is available from the CLASSiC project data repository, see http://www. macs. hw. ac. uk/iLabArchive/CLASSiCProject/ Data/myaccount. php.
C. Supervised Baseline strategy We analyzed the WoZ data to explore the strategies that human wizards explored for this task. Note that while the wizards showed significantly different behavior in terms of overall frequency, however no significant difference in user ratings was detected  [50]. We therefore hypothesize that wizards select different strategies according to different contexts. Observing significantly different strategies which are not significantly different in terms of user satisfaction, we conjecture that the wizards converged on strategies which were appropriate in certain contexts. We also observed that there was a wide spread user ratings for different dialogues. We therefore took the top rated 50% of the IP instances ( ) to build a baseline model which reflects "good" wizard behavior.
We used a variety of supervised learning methods to create a model of the highly rated wizard behavior. Please see  [50]  for further details. The best performing method was Rule Induction (JRip). 7 The model achieved an accuracy of 43. 19% which is significantly ( ) better than the majority baseline of always choosing SUMMARY (34. 65%). 8 The resulting rule set is shown in Fig. 4 , where the determining features are number of database hits retrieved and the IP strategy realized in the previous turn : The features selected by this model were only "high-level" features, i. e. the features that an IP module receives as input from a Dialogue Manager. We further analyzed the importance of different features using feature ranking and selection methods  [50] , finding that the human wizards (in this specific setup) did not pay significant attention to any lower level features, e.
6) The user closes the interaction. We built user simulations using n-gram models of system ( ) and user ( ) acts, as first introduced by  [56]. In order to account for data sparsity, we apply different discounting ("smoothing") techniques including back-off, using the CMU Statistical Language Modelling toolkit  [57] , see Table  III. We construct a bi-gram model for the users' reactions to the system's IP structure decisions ( ), and an extended bi-gram (i. e. IP structure attribute choice) model for predicting user reactions to the system's combined IP structure and attribute selection decisions: , where is the predicted next user action at time , was the system's information presentation action at , and is the attributes selected by the system at. We evaluated the performance of these models by measuring dialogue similarity to the original data, based on the Kullback-Leibler (KL) divergence, as also used by, e.
B. Database Matches and Attention Modelling An important task of information presentation is to support the user in choosing between all the available items (and ultimately in selecting the most suitable one) by structuring the current information returned from the database, as explained in Section II. We therefore represent the items brought to the user's attention as a feature in our learning experiments. In particular, attention modelling is used to determine the number of database hits (# DBhits) used in the reward function, see Section V-C. This feature reflects how the different IP strategies structure information with different numbers of attributes, similar to the clustering approach discussed in  [5]. We implement this shift of the user's attention focus analogously to discovering the user's goal in dialogue management: every time the predicted next user act is to add information ( ), we infer that the user is therefore only interested in a subset of the previously presented results and so the system will focus on this new subset of database items in the rest of the generated utterance.
) ( ). The chosen features indicate that users' ratings are influenced by higher level and lower dialogue level features: Users like to be focused on a small set of database hits (where # DBhits ranges over 1-100), which will enable them to choose an item (valueUserReaction), while keeping the IP utterances short (where # sentence is in the range  [2] -  [18] ): (2) The reward is calculated at the end of an episode, which for this task constitutes a full NLG utterance, also see Section III-A. Note that the worst possible reward for a NLG move is therefore. This is achieved by presenting 100 items to the user in 18 sentences  10  , in such a way that the user immediately ends the conversation unsuccessfully. The top possible reward is achieved in the rare cases where the system can immediately present one item to the user using a minimum of two sentences, and the user then selects that item, i.