dataesr/scientific-paragraphs-categorization huggingface dataset:
H (A, B) the number of modifications between A and B. More precisely, we extend the Hamming distance to a distance between two sets by defining d H (A, B) as d H (A, B) = |(A B) ∪ (B A)|. d H (A, B) is the number of elements present in only one dataset, A or B. In other words, it is the minimal number of addition and deletions to transform A into B (and vice-versa). Theorem 5: Let B be a biased dataset, D a dataset representative of P such as D ⊆ B ∪ S and G = Algo1(B). If |B| = |D|, then d H (B, D) ≥ d H (B, G) = 2 max( k n + B (k), k n - B (k)). So Algorithm 1 outputs a representative dataset with minimal distance to B. Proof: Let A = D B be the set of elements added to B and R = B D be the set of elements removed from B. So D = (B R) ∪ A and d H (B, D) = |A| + |R|. Due to the hypothesis |B| = |D|, we know that |A| = |R| (there are as many additions as removals).
For any class k that is overrepresented in B, n + B (k) is the minimal number of removal of elements of class k such as k is not overrepresented anymore. n - B (k) = max(0, -|B|( T p B,k + δ)) is similar for underrepresented classes. To make a representative dataset from B, one must removeat least k n + B (k) elements and add k n - B (k)), i. e. , |A| ≥ k n + B (k) and |R| ≥ k n - B (k)). However, since |A| = |R|, we conclude that |A| = |R| ≥ max( k n + B (k), k n - B (k)). It means that d H (B, D) ≥ 2 max( k n + B (k), k n - B (k)). Let now show that d H (B, G) = 2 max( k n + B (k), k n - B (k)). At each iteration of the algorithm, if k n + G (k) > 0, then an overrepresented class has one of its elements removed, so k n + G (k) < k n + G (k). For the same reason, at each iteration, k n - G (k) < k n - G (k).
When k n + G (k) = k n - G (k) = 0, then the algorithm halts. So there are max( k n + G (k) = k n - G (k)) iterations. Since each iteration makes two modifications of the dataset, we can conclude that d H (B, G) = 2 max( k n + B (k), k n - B (k)). So d H (B, G) ≤ d H (B, D). Proof of theorem 1: Let us first prove that at each step of the algorithm, d is equal to the number of additions and deletions between B and Algo1(G). This number can be decomposed as the sum of the number of additions and deletions between B and G and between G and Algo1(G). The number of modifications between G and B is simply the difference of their size, i. e. , | |B| -|G| |. The number of modification between G andAlgo1(G) is 2 max( k n + G (k), k n - G (k)) (cf. the previous proof algorithm). So the number of modifications between B and Algo1(G) is| |B| -|G| | + 2 max( k n + G (k), k n - G (k)).
First, let us show that the algorithm always halts. This algorithm halts either if d min = ∞ (so both loops have a finite number of iterations) or if the source dataset lacks some element (lines 16 to 16). If a solution G exists, then at some point Algorithm 2 will call Algorithm 1 with a dataset whose size is the same as G (either at line 2 if |G| = |B|, at line 10 if |G| < |B| or at line 26 if |G| > |B|) and Algorithm 1 will find a solution (due to Theorem 5). If at some point it is not possible to add an element to a underrepresented class because there is no such element in the source dataset, then the algorithm halts (lines 5 to 9). Second, we show that doing d min -1 iterations per loop is sufficient (lines 6 and 15). Let G best be the best dataset found so far, associated to a distance d min , and a better solution G associated to a distance d such as d < d min.
Remark that at each step, i = | |B| -|G| | as one element is modified per iteration. Denote j the iteration at which G could be found. Since d = | |B| -|G| | + 2 max( k n + G (k), k n - G (k)) ≥ j and d < d min , we conclude that j ≤ d min and therefore that G will be found by the algorithm. Theorem 6: The worst-case temporal complexity of Algorithm 2 is O ( 1 δ + |B|) 2 |K|. Proof: Remark that for all datasets A and B, the number of additions and deletions required to obtain B from A is lower than or equal to |A| + |B|. Theorem 3 shows that there exists a representative dataset D of size 1 δ. So the minimal distance d min to reach a representative dataset D from the initial base dataset B is bounded by 1 δ + |B|. So Algo. 2 will do at most 2( 1 δ + |B|) iterations. Each iteration calls Algo.
In supervised learning methods, researchers learn a model that classifies inputs into a set of classes. Two datasets are used in this setting: a training dataset and a test dataset. The training set is used during the learning phase, and the test set is used to evaluate the learned model on new data. Both datasets should be labeled. The training set needs to be selected carefully to avoid biases, as discussed in more detail in Section III. The test set should be representative, as for the Algorithm evaluation scenario.
DATASETS FOR MACHINE LEARNING TRAINING AND EVALUATIONTABLE IV MEAN AND MAX AUC OF MACHINE LEARNING CLASSIFIERS DEPENDING ON THEIR TRAINING SET AND THEIR TEST SET WHEN THE PIVOT YEAR IS 2017. IN BOLD FACE: BEST AUC ON A TEST SET TABLE V MEAN AND MAX AUC OF MACHINE LEARNING CLASSIFIERS DEPENDING ON THEIR TRAINING SET AND THEIR TEST SET WHEN THE PIVOT YEAR IS 2014. IN BOLD FACE: BEST AUC ON A TEST SET.
4 G5 G ← B 6 for i from 1 to d min -1 do Search smaller datasets 7 if G = ∅ then break; 8 k 9 Remove one random element of class k h from G 10 dthen 17 K 21 r 22 Remove r from S and add it to G 23 d 47891017212223 best ← Algo1(B, T, S, δ) h ← arg max k∈K ( p G,kp T,k ) ← | |B| -|G| | + 2 max( k n + G (k), k n - G (k))11 if d < d min and Algo1(G, T, S, δ) = "Impossible" then 12 d min ← d; G best ← Algo1(G, T, S, δ) 13 G ← B 14 for i from 1 to d min -1 do Search larger datasets 15k l ← arg min k∈K ( p G,kp B,k ) 16 if |S(k l )| = 0 S = {k ∈ K | |S(k)| > 0} 18 if p G,k l ≥ p T,k -δ and K S = ∅ then 19 k l ← arg min k∈K S p G,kp ← one element of S(k l ) chosen at random ← | |B| -|G| | + 2 max( k n + G (k), k n - G (k))24 if d < d min and Algo1(G, T, S, δ) = "Impossible" then 25 d min ← d; G best ← Algo1(G, T, S, δ) 26 return G best Theorem 1: Let B, S, T be three datasets and δ > 0.
In supervised learning methods, researchers learn a model that classifies inputs into a set of classes. Two datasets are used in this setting: a training dataset and a test dataset. The training set is used during the learning phase, and the test set is used to evaluate the learned model on new data. Both datasets should be labeled. The training set needs to be selected carefully to avoid biases, as discussed in more detail in Section III. The test set should be representative, as for the Algorithm evaluation scenario. 1 http://googlemobile. blogspot. com/2012/02/android-and-security. html 2 https://www. blog. google/products/android/google-play-protect/ 3 https://www. apkmirror. com/ 4 https://apkpure. com 3) Machine Learning:.
Snow cover models support avalanche forecasting. However, despite much development to derive snow instability from snow cover model output, snow cover models presently do not provide information on avalanche problem typesan essential element to describe avalanche danger. We present an approach to detect, track and assess weak layers in snow cover model output data and assess the related avalanche problem types. We classify different avalanche situations into new snow, wind slab, persistent weak layer and wet-snow problems for either natural or artificial avalanche release. We demonstrate the applicability of this approach with both SNOWPACK and Crocus model output for several winter seasons at Weissfluhjoch, Switzerland, and two sites in Western Canada. At Weissfluhjoch, simulation results from both models coincided reasonably well with avalanche activity recorded with a seismic avalanche detection system.
(2017). Mass and heat exchange with the atmosphere and soil were simulated with ISBA  (Decharme et al. , 2011)  using Neumann boundary conditions. Events of drifting snow were estimated when the SNOWPACK variable "snow_transport_24_h" exceeded 0. 4 kg/cm. For Crocus simulations, we estimated snow transport based on a threshold wind speed of 7. 5 m/s  (Li and Pomeroy, 1997). We did not utilize snow transport modules, but calculated a simple index which does not account for the feedback of snow transport on the snow cover  (Lehning and Fierz, 2008). Both snow cover models, SNOWPACK and Crocus, ran until 31 May outputting snow profiles at 6:00 and 15:00 UTC. For the simulations in Canada, we used numerical weather prediction model (NWP) data extracted from the 2. 5 km version of the GEM model  (Milbrandt et al.
70; FP = 0. 33) that we can use to assess natural release and eventually the corresponding avalanche problem types in Crocus simulations (Table  1 ). As expected, time to failure and critical crack length are used in combination, threshold values differ from typical values when the indicators are used alone, such as natural stability index S n = 1. We compare the natural avalanche activity during the winter season of 2015-16 to simulated avalanche problems that are related to natural release. Fig. 7  shows the number of avalanches per day from the seismic avalanche catalogue. Data were recorded from 1 January 2016 until 30 April 2016 without interruption (period presented in the figure). The first part of the winter season shows several avalanche cycles with up to three avalanches per day, whereas from the beginning of April daily avalanche activity was higher with up to 13 avalanches per day.
Avalanche observations For evaluating our model, avalanche observations would ideally be separated by weak layer type or avalanche problem type and release times were accurate to the hour. However, crown profiles and direct visual observations are rare. Observers can often separate the trigger, but a limited number of observations hampers spatial coverage and even when using cameras for better coverage a correct estimate is limited by visibility  (Helbig et al. , 2015). We therefore used an avalanche catalogue obtained from seismic monitoring. Such data have high temporal resolution, which is particularly relevant for natural dry-snow avalanche release. The area covered by seismic monitoring systems (about 30 km 2 ) is much smaller than the typical size of an avalanche forecasting region (> 200 km 2 ).
Nevertheless, such systems can generally be used to identify the main avalanche periods  (Heck et al. , 2019) , and in our case we combined data from two systems about 15 km apart. The seismic avalanche data allowed us to calibrate snow instability indicators and evaluate simulated avalanche problems regarding natural release. We found the combination of the expected time to failure and the critical crack length provided a satisfying tradeoff in predicting avalanche days (POD >0. 77 for SNOWPACK and Crocus) and detecting non-avalanche days (PON > 0. 66 for both models). Comparing days with identified avalanche problems and days with natural release (Table  4 ) suggests that the model overestimates the number of avalanche days. On one hand, as our snow cover simulations can at best describe the situation around Weissfluhjoch, they are not necessarily indicative of the situation in the region aroundwhich can be more variable than is reflected by the single, albeit representative measurement location.
Snow climates We detected avalanche problem types at locations in Western Canada, at Whistler and Rogers Pass, and in the Swiss Alps, at Weissfluhjoch. Based on numerical weather prediction data and SNOWPACK simulations we identified twice as many new snow than persistent weak layer problems in Whistler. At Rogers Pass avalanche problems were 5 times more often related to persistent weak layers than to new snow. In other words, the avalanche problem type new snow was twice as common in Whistler and persistent weak layers dominated at Rogers Pass -6 times more common than in Whistler. The results confirm our expectation of an intermountain (Rogers Pass) and a maritime snow climate (Whistler). More importantly, these results provide us more confidence that the presented approach for avalanche problem type assessment works in different snow climates.
Including a time derivative such as the expected time to failure allowed to account for the meteorological history and the dynamic evolution on the scale of hours, that are important for the likelihood of natural release. For artificial release we simulated the propensity to failure initiation and crack propagation as the current knowledge of avalanche release suggests and applied commonly used thresholds for snow instability, as data to determine the values are rare. We used the algorithm with two different snow cover models and applied the algorithm to three different locations and several winter seasons. Assessing avalanche problem types in SNOWPACK and Crocus simulations yielded reasonable agreement with field data at Weissfluhjoch, Davos, Switzerland and Whistler and Rogers Pass, BC, Canada.
Times and frequencies of avalanche problem types were similar and coincided with highly resolved natural avalanche activity data obtained with seismic monitoring, although some discrepancies remainedin particular related to snow transport. We also show a time series of 17 winter seasons of avalanche problems between 1999 and 2016 at Weissfluhjoch, Davos. Simulated avalanche problem types were in 41 or 47% of the cases related to new snow or wind slabs, 37 or 31% of the cases related to persistent weak layers and in 22 and 21% related to wet-snow, for SNOWPACK and Crocus, respectively. The onset date of the wet-snow avalanche cycle in spring shifted at Weissfluhjoch by about 10-14 days during the 16-year period, consistent with other research that wet-snow avalanches release increasingly earlier in the last decades.
Task and Models Task. The GuessWhat?! game (de Vries et al. 2017) is a cooperative two-player game based on a referential communication task where two players collaborate to identify a referent object in an image. This setting has been extensively used in human-human collaborative dialogue  (Clark 1996; Yule 2013). GuessWhat?! is an asymmetric game involving two human participants who see a real-world image containing multiple objects from the MSCOCO dataset  (Lin et al. 2014). One of the participants (the Oracle) is secretly assigned a target object within the image and the other participant (the Questioner agent) has to guess the target by asking binary (Yes/No) questions to the Oracle.
Datasets We are interested in studying how modifying part of the human data in the training set affects the linguistic output and the model's accuracy on the GuessWhat?! game at test time. More specifically, we aim at building a training set in which part of the dialogues collected with human annotators are replaced with dialogues generated by the GDSE-SL questioner model while playing with the baseline Oracle model on the same games being replaced, i. e. , the final training set size is always the same. In this way, we build Oracle. The images used in GuessWhat?! are taken from the MS-COCO dataset  (Lin et al. 2014). Each image contains at least three and at most twenty objects. More than ten thousand people in total participated in the dataset collection procedure. Human annotators were instructed to ask only binary yes/no questions to the Oracle, and they could stop asking questions at any time, so the length of the dialogues is not fixed.
Humans used a vocabulary of 17657 words to play GuessWhat?!: 10469 of these words appear at least three times, and thus make up the vocabulary given to the models. For our experiments, we considered only those games in which humans succeeded in identifying the target object in less than 20 dialogue turns, following a common practice from previous work. Mixed Batches. We let the GDSE-SL model play with the baseline Oracle on the same games of the human training dataset. This produces automatically generated data for the whole training set. The model uses less than 3000 words out of a vocabulary of more than 10000 words. We built new training sets according to two criteria: the proportion of human and machine-generated data (50%-50% or 75%-25% respectively) and the length of the generated dialogue.
Either we always keep a fixed dialogue length (5 turns, as the average length in the dataset) or we take the same number of turns that the human Questioner used while playing the game we are replacing. Table  1  reports some statics of different training sets. Human dialogues have a very low mutual overlap and a much larger vocabulary than both the generated (100-0 setting, where the first number refers to the percentage of human-generated dialogues and the second one to the machine-generated percentage) and mixed batches datasets. Looking at the number of games with at least one repeated question in the training set (GRQ column in Table  1 ), it can be observed that human annotators never produce dialogues with repetitions. The 75-25 dataset configuration contains less than 3% of dialogues with repeated questions and this percentage rises to around 5% for the 50-50 configuration and to around 10% for the machine-generated dialogues (0-100).
Implementation Details In order to obtain a dataset of machine-generated data (Step 1 in Figure  1 ), we trained the GDSE-SL model for 100 epochs as described in  Shekhar et al. (2019b). At the end of the training, we used GDSE to play the game with the Oracle on the whole training set, saving all the dialogues. We generate these dialogues with the model trained for all the 100 epochs since it generates fewer repetitions, although it is not the best-performing on the validation set. The dialogues generated by GDSE while playing with the Oracle are noisy: they may contain duplicated questions, wrong answers, etc. See Figure  2  for an example of human and machine-generated dialogues for the same game. We design different training sets as described in Section 5 and train the GDSE-SL model on these datasets (Step 2 in Figure  1 ).
Results To have an overall comparison of the various settings, we computed the harmonic mean of all the metrics described above -after normalization and inverting the 'lower is better' metrics. The best balance is obtained with the 50-50 variable-length configuration. Below we zoom into the details of this coarse evaluation so as to highlight how the behaviour of the model changes across the settings. Table  2  reports the results of the GDSE model trained on different training sets. By looking at the results on the test set, we can see how even a small number of machinegenerated dialogues affects the generation phase at test time, when the model generates 5-turns dialogues and, at the end of the game, it guesses the target object. First of all, it can be noticed that the accuracy of GDSE-SL trained on the new datasets outperforms the one trained on the original training set: in particular, the accuracy of GDSE trained on 50% human dialogues and 50% 5-turns generated dialogues is almost 2% higher (in absolute terms) than the model trained only on human dialogues.
09. The percentage of vocabulary used (global recall), on the other hand, remains stable. Interestingly, the only metric that seems to suffer from the model being trained on mixed datasets is the number of novel questions: being trained on noisy data does not seem to improve the "creativity" of the model, measured as the ability to generate new questions compared to ones seen at training time. The impact of the training data on the decoder's creativity and hence on the acquisition of human-like language generation ability deserves further investigation. However, it is out of the scope of our main research question, which instead focuses on understanding the quality of the generated dialogues rather than the creativity of the questioner. Overall, our results show an interesting phenomenon: replacing part of the Guess-What?! training set with machine-generated noisy dialogues, and training the GDSE-SL questioner model on this new dataset, is found to improve both the accuracy of the guessing game and the linguistic quality of the generated dialogues.
Implementational Details In the following, we investigate more in detail the effect of the two surface-level features of machine-generated dialogues that are easier to inject in the training data: repetitions and wrong answers coming from the Oracle. We check whether one of these two features alone accounts for the performance improvement described in the previous Section. To isolate their effect, we took the GuessWhat?! training set containing only ground-truth human-generated dialogues and we randomly swap the answers in the dialogue history or, alternatively, repeat one random question in the dialogue history. Considering that: the accuracy of the Oracle agent in answering questions is around 80%, and that the datasets used in the previous experiment contain on average 2. 6% to 5. 4% of games with repeated questions, we inject repetitions/wrong answers in the training set with a frequency of 5% or 20%.
Results As we can see from Table  3  (top), swapping 5% or 20% of the answers in groundtruth human dialogue from the training set does not lead to the improvement we have presented in Section 6, but instead, it seriously degrades both the accuracy of the guessing task and the quality of the generated dialogues. More specifically, the accuracy drops by up to -14. 2% compared to the model trained on the original GuessWhat?! dataset, and the percentage of games with at least one repeated question increases up to +11. 9%. Table  3  (bottom) shows the results of GDSE-SL when trained on a dataset of human dialogues containing a variable percentage of games with one duplicated question in the dialogue history. Compared to the dataset with swapped answers, in this case the drop in accuracy is very limited (up to -2.
3%). On the other hand, the percentage of games with at least one repeated question at test time increases dramatically (up to +28. 4% compared to the model trained on a dataset without repetitions), thus none of the parameters we consider improves. Overall, these results demonstrate that the presence of repetitions or inconsistent answers alone in the training data do not account for the high performance illustrated in Section 6 using a training set composed of both human and machine-generated dialogues. This means that there are other features in the dialogue structure of generated dialogues that, when mixed with the human ground-truth data, create a training environment that, being closer to the inference setting, leads to more natural and effective dialogues.
Human Annotation Setup We asked human annotators to read machine-generated dialogues and: 1. judge their grammaticality (using a scale from 1 to 5), 2. select the target object from a list of candidate objects, i. e. , we asked human annotators to play the role of the Guesser module, 3. express their confidence on a scale from 1 to 5, their confidence in selecting the target object, where 5 means 'I am highly confident about my choice' and 1 means 'My target selection was a random guess'. We anticipate that all dialogues were evaluated as perfectly grammatical regardless of the training procedure, thus we do not discuss these results in the following. We extracted 140 games from the GuessWhat validation set with fewer than 6 candidate objects in the image. For each game, we took the dialogue generated in the 100-0 and in the 50-50 paradigm.
e. using clean human-generated data while at test/inference time they are left "in the wild", and they have to solve the task by interacting one with the other without having access to ground-truth human-generated dialogues. Following this observation, in this paper we modify the GuessWhat?! training by replacing part of the dialogues produced by human annotators with machine-generated "noisy" dialogues, so that the training and testing conditions become more similar to each other. We show that a state-of-the-art model benefits from being trained on this new "mixed" dataset: being exposed to a small number of "noisy" dialogues at training time improves the quality of the output without deteriorating its accuracy on the task. Our results show an absolute improvement in the accuracy of +1. 8% and a drop in the number of dialogues containing duplicated questions of around -14%.