dataesr/scientific-paragraphs-categorization huggingface dataset:
We believe that the path toward integration could be the key in further perfecting and improving coordination models and languages, by complementing data-driven models elegance and flexibility with control-driven models finegrained control and predictability. Families marriage. Fig. 6  enriches the family tree just described with relationships indicating differences (red and green arrows) and similarities (blue arrows) in features provided-notice that w. r. t. Fig. 5  Piccola, CiAN, and JErlang have been removed because they are so unique that no clear relationship may be found with other technologies. As already mentioned for Fig. 5 , Linda has been taken as the common ground for many technologies which are instead very heterogeneous in the aim pursued: if ACLT , TuCSoN, and LINC have a Linda core enriched with many other features -such as programmability, transactionality, and novel primitives -, the Klaim family and the Lime one diverge more, by changing the way in which primitives behave -as in the case of localities in Klaim -, or the way in which the interacting processes see each others' tuple spaces-as for Lime transient federation.
In fact, TuCSoN / ACLT , LINC, and Moses all provide means to somewhat program the coordinative behaviour of the coordination medium, thus aim at making it configurable, adaptable, malleable, even at run-time, and/or provide additional coordination primitives to expand the expressive reach of the coordination technology. The Klaim family, the Reo family, and the Lime family instead, are geared toward some forms of space-awareness, be it by promoting mobility or by providing location-sensitive primitives. Besides these, two more main goals can be devised, peculiar to specific technologies: (iii) supporting humans-in-the-loop, in the case of CiAN, and (iv) provide a semantic representation of data items, in the case of TripCom. About the application scenarios explicitly declared as of particular interest for the technology, the most prominent one is service composition, which is especially interesting for Piccola, JErlang, the Reo family, the Klaim family, and TripCom-besides being naturally applicable to all other technologies too.
net/projects/statelets SecOS [22] IIC [77] 1999 2012 Linda Reo [6] no implementation found http://github. com/joseproenca/ip-constraints Messengers [95] LINC [58] 1999 2015 CmPS + mobility [52] Linda [44] http://osl. cs. illinois. edu/software/ no mention of "Messengers" implementation not available for commercial reasons see http:// bag-era. fr/ index en. html#about MJada [82] RepliKlaim [3] 1999 2015 OO Linda Klaim [19] no reference to MJada http://www. cs. unibo. it/cianca/wwwpages/macondo/ http://sysma. imtlucca. it/wp-content/uploads/2015/03 Logic Fragments [30] 2014 Sapere [97] no implementation found STL++ [87] 1999 ECM [87] no implementation found Clam [86] 1999 IWIM [5] no implementation found TuCSoN [28] 1999 novel (many extensions to Linda) http://tucson. unibo. it/ Truce [53] 1999 novel (protocols + roles) no implementation found.
TripCom [89]  2007 Triple Space Computing [39]  http://sourceforge. net/projects/tripcom CiAN [88] 2008 novel http://mobilab. cse. wustl. edu/Projects/CiAN/Home/Home. shtml Smrl [1] 2008 Pepa [45] http://groups. inf. ed. ac. uk/srmc/download. html CaSPiS [18] 2008 IMC [17] http://sourceforge. net/projects/imc-fi LeanProlog [91] 2008 novel http://www. cse. unt. edu/ ∼ tarau/research/LeanProlog JErlang [75] 2010 Join-Calculus [41] http://github. com/jerlang/jerlang.
, 2003). PBSMT models are well-known to outperform neural models when labeled data is scarce because they merely count occurrences, whereas neural models typically fit hundred of millions of parameters to learn distributed representations, which may generalize better when data is abundant but is prone to overfit when data is scarce. Our PBSMT model is simple, easy to interpret, fast to train and often achieves similar or better results than its NMT counterpart. We report gains of up to +10 BLEU points on widely used benchmarks when using our NMT model, and up to +12 points with our PBSMT model. Furthermore, we apply these methods to distant and low-resource languages, like English-Russian, English-Romanian and English-Urdu, and report competitive performance against both semi-supervised and supervised baselines.
Unsupervised MT systems Equipped with the three principles detailed in Section 2, we now discuss how to effectively combine them in the context of a NMT model (Section 3. 1) and PBSMT model (Section 3. 2). In the reminder of the paper, we denote the space of source and target sentences by S and T , respectively, and the language models trained on source and target monolingual datasets by P s and P t , respectively. Finally, we denote by P s→t and P t→s the translation models from source to target and vice versa. An overview of our approach is given in Algorithm 1.
Unsupervised PBSMT In this section, we discuss how to perform unsupervised machine translation using a Phrase-Based Statistical Machine Translation (PBSMT) system  (Koehn et al. , 2003)  as the underlying backbone model. Note that PBSMT models are known to perform well on low-resource language pairs, and are therefore a potentially good alternative to neural models in the unsupervised setting. When translating from x to y, a PBSMT system scores y according to: arg max y P (y|x) = arg max y P (x|y)P (y), where P (x|y) is derived from so called "phrase tables", and P (y) is the score assigned by a language model. Given a dataset of bitexts, PBSMT first infers an alignment between source and target phrases. It then populates phrase tables, whose entries store the probability that a certain n-gram in the source/target language is mapped to another ngram in the target/source language.
, 2018) , and e(x) is the embedding of x. Language Modeling: Both in the source and target domains we learn smoothed n-gram language models using KenLM  (Heafield, 2011) , although neural models could also be considered. These remain fixed throughout training iterations. Iterative Back-Translation: To jump-start the iterative process, we use the unsupervised phrase tables and the language model on the target side to construct a seed PBSMT. We then use this model to translate the source monolingual corpus into the target language (back-translation step). Once the data has been generated, we train a PBSMT in supervised mode to map the generated data back to the original source sentences. Next, we perform both generation and training process but in the reverse direction. We repeat these steps as many times as desired (see Algorithm 2 in Section A).
Intuitively, many entries in the phrase tables are not correct because the input to the PBSMT at any given point during training is noisy. Despite that, the language model may be able to fix some of these mistakes at generation time. As long as that happens, the translation improves, and with that also the phrase tables at the next round. There will be more entries that correspond to correct phrases, which makes the PBSMT model stronger because it has bigger tables and it enables phrase swaps over longer spans.
Datasets and Methodology We consider five language pairs: English-French, English-German, English-Romanian, English-Russian and English-Urdu. The first two pairs are used to compare to recent work on unsupervised MT  (Artetxe et al. , 2018; Lample et al. , 2018). The last three pairs are instead used to test our PB-SMT unsupervised method on truly low-resource pairs  (Gu et al. , 2018)  or unrelated languages that do not even share the same alphabet. For English, French, German and Russian, we use all available sentences from the WMT monolingual News Crawl datasets from years 2007 through 2017. For Romanian, the News Crawl dataset is only composed of 2. 2 million sentences, so we augment it with the monolingual data from WMT'16, resulting in 2. 9 million sentences. In Urdu, we use the dataset of  Jawaid et al.
(2014) , composed of about 5. 5 million monolingual sentences. We report results on newstest 2014 for en -f r, and newstest 2016 for en -de, en -ro and en -ru. For Urdu, we use the LDC2010T21 and LDC2010T23 corpora each with about 1800 sentences as validation and test sets, respectively. We use Moses scripts  (Koehn et al. , 2007)   codes. PBSMT is trained with true-casing, and by removing diacritics from Romanian on the source side to deal with their inconsistent use across the monolingual dataset  (Sennrich et al. , 2016).
Results The results reported in Table  2  show that our unsupervised NMT and PBSMT systems largely outperform previous unsupervised baselines. We report large gains on all language pairs and directions. For instance, on the en → f r task, our unsupervised PBSMT obtains a BLEU score of 28. 1, outperforming the previous best result by more than 11 BLEU points. Even on a more complex task like en → de, both PBSMT and NMT surpass the baseline score by more than 10 BLEU points. Even before iterative back-translation, the PBSMT model significantly outperforms previous approaches, and can be trained in a few minutes. Table  3  illustrates the quality of the PBSMT model during the iterative training process. For instance, the f r → en model obtains a BLEU score of 17. 5 at iteration 0 -i. e. after the unsupervised phrase table construction -while it achieves a score of 27.
2 at iteration 4. This highlights the importance of multiple back-translation iterations. The last rows of Table  3  also show that we get additional gains by further tuning the NMT model on the data generated by PBSMT (PBSMT + NMT). We simply add the data generated by the unsupervised PBSMT system to the back-translated data produced by the NMT model. By combining PB-SMT and NMT, we achieve BLEU scores of 20. 2 and 25. 2 on the challenging en → de and de → en translation tasks. While we also tried bootstraping the PBSMT model with back-translated data generated by a NMT model (NMT + PB-SMT), this did not improve over PBSMT alone. Next, we compare to fully supervised models. Figure  2  shows the performance of the same architectures trained in a fully supervised way using parallel training sets of varying size.
The unsupervised PBSMT model achieves the same performance as its supervised counterpart trained on more than 100,000 parallel sentences. This is confirmed on low-resource languages. In particular, on ro → en, our unsupervised PB-SMT model obtains a BLEU score of 23. 9, outperforming  Gu et al. (2018) 's method by 1 point, despite its use of 6,000 parallel sentences, a seed dictionary, and a multi-NMT system combining par-allel resources from 5 different languages. On Russian, our unsupervised PBSMT model obtains a BLEU score of 16. 6 on ru → en, showing that this approach works reasonably well on distant languages. Finally we train on ur → en, which is both low resource and distant. In a supervised mode, PBSMT using the noisy and outof-domain 800,000 parallel sentences from Tiedemann (2012) achieves a BLEU score of 9.
Ablation Study In Figure  3  we report results from an ablation study, to better understand the importance of the three principles when training PBSMT. This study shows that more iterations only partially compensate for lower quality phrase table initialization (Left), language models trained over less data (Middle) or less monolingual data (Right). Moreover, the influence of the quality of the language model becomes more prominent as we iterate. These findings suggests that better initialization methods and more powerful language models may further improve our results. We perform a similar ablation study for the NMT system (see Appendix). We find that backtranslation and auto-encoding are critical components, without which the system fails to learn. We also find that initialization of embeddings is very important, and we gain 7 BLEU points compared to prior work  (Artetxe et al.
Conclusions and Future Work In this work, we identify three principles underlying recent successes in fully unsupervised MT and show how to apply these principles to PBSMT and NMT systems. We find that PBSMT systems often outperform NMT systems in the fully unsupervised setting, and that by combining these systems we can greatly outperform previous approaches from the literature. We apply our approach to several popular benchmark language pairs, obtaining state of the art results, and to several low-resource and under-explored language pairs. It's an open question whether there are more effective instantiations of these principles or other principles altogether, and under what conditions our iterative process is guaranteed to converge. Future work may also extend to the semisupervised setting.
A Supplemental Material In this Appendix, we report the detailed algorithm for unsupervised PBSMT, a detailed ablation study using NMT and conclude with some example translations. Algorithm 2: Unsupervised PBSMT 1 Learn bilingual dictionary using  Conneau et al. (2018) ; 2 Populate phrase tables using Eq. 3 and learn a language model to build P In Table  4  we report results from an ablation study we performed for NMT using the Transformer architecture. All results are for the en → f r task. First, we analyze the effect of different initialization methods for the embedding matrices. If we switch from BPE tokens to words, BLEU drops by 4 points. If we used BPE but train embeddings in each language independently and then map them via MUSE  (Conneau et al. , 2018) , BLEU drops by 3 points. Finally, compared to the word aligned procedure used by  Lample et al.
(2018) , based on words and MUSE, the gain is about 7 points. To stress the importance of initialization, we also report performance using random initialization of BPE embeddings. In this case, convergence is much slower and to a much lower accuracy, achieving a BLEU score of 10. 5. The table also demonstrates the critical importance of the auto-encoding and back-translation terms in the loss, and the robustness of our approach to choice of architectures. iterations of the learning algorithm for both NMT and PBSMT models. Prior to the first iteration of back-translation, using only the unsupervised phrase table, the PBSMT translations are similar to word-by-word translations and do not respect the syntax of the target language, yet still contain most of the semantics of the original sentences.
As we increase the number of epochs in NMT and as we iterate for PBSMT, we observe a continuous improvement in the quality of the unsupervised translations. Interestingly, in the second example, both the PBSMT and NMT models fail to adapt to the polysemy of the French word "langue", which can be translated as "tongue" or "language" in English. These translations were both present in the unsupervised phrase table, but the conditional probability of "language" to be the correct translation of "langue" was very high compared to the one of "tongue": P (language|langue) = 0. 92, while P (tongue|langue) = 0. 0005. As a comparison, the phrase table of a Moses model trained in a supervised way contains P (language|langue) = 0. 633, P (tongue|langue) = 0. 0076, giving a higher probability for "langue" to be properly translated.
NMT Epoch 1 The protein that we use in the ice with the language to pH. NMT Epoch 8 The protein we use into the ice responds with language to pH neutral. NMT Epoch 45 The protein we use in ice responds with the language from pH to neutral. PBSMT Iter. 0 The protein that used in the ice responds with the language and pH neutral. PBSMT Iter. 1 The protein that we use in the ice responds with the language to pH neutral. PBSMT Iter. 4  The protein that we use in the ice reacts with the language to a neutral pH.
In this work, we use a rich data set of atmospheric methane observations collected during an intensive airborne campaign to compare different methane emissions estimates from the Barnett Shale oil and natural gas production basin in Texas, USA. We estimate emissions based on a variety of different meteorological and dispersion models. Previous estimates of methane emissions from this region relied on a simple model (a mass balance analysis) as well as on ground-based measurements and statistical data analysis (an inventory). We find that in addition to meteorological model choice, the choice of tracer dispersion model also has a significant impact on the predicted downwind methane concentrations given the same emissions field. The dispersion models tested often underpredicted the observed methane enhancements with significant variability (up to a factor of 3) between different models and between different days.
, 2012). Lauvaux et al. (2012)  also found that there were some cases (a few days per month in that study) in which the Eulerian (WRF-Chem) and the Lagrangian (LPDM) modeled concentrations showed large differences (when run using the same emissions field) and that these cases were associated with stable conditions in the lower atmosphere. These regional studies suggest that tracer dispersion characteristics (and not only the underlying meteorological fields) can and do affect simulated concentrations. In March and October 2013, an aircraft-based campaign was conducted with the goal of quantifying CH 4 emissions from oil and gas operations in the Barnett Shale region of Texas, USA. The airborne data set was used by  Karion et al. (2015)  and  Smith et al. (2015)  to estimate total emissions of CH 4 and ethane in the region using a mass balance estimation (MBE) method.
Acknowledgements. The authors thank  Tim Bonin ,  Mike Hardesty , and  Alan Brewer  at  NOAA/ESRL Chemical Sciences Division  for providing HRDL data products;  Marikate Mountain  ( AER ) for CarbonTracker-Lagrange (CT-L) WRF-STILT model runs;  Eric Kort  ( University of Michigan ),  Paul Shepson  ( Purdue University ), and the teams from  NOAA/ESRL ,  University of Michigan, and Purdue  for collaborating on the Barnett flight campaign and collecting airborne data;  Daniel Zavala-Araiza  and  David Lyon  for providing the Barnett CH 4 inventory; and  Fantine Ngan  for assistance in development of the experimental HYSPLIT parameterization. We also thank two anonymous reviewers for helpful comments and suggestions. Edited by:  Ken Carslaw Reviewed  by: two anonymous referees.
Data availability. Aircraft observations used in this study are available at https://www. esrl. noaa. gov/gmd/dv/data/index. php?site=Barnett (Sweeney et al. , 2019)  or by request from colm. sweeney@noaa. gov. Meteorological (WRF) and tracer dispersion model output is available on the NIST data portal at https://doi. org/10. 18434/T4/1503403 (Karion et al. , 2019). www. atmos-chem-phys. net/19/2561/2019/ Atmos. Chem. Phys. , 19, 2561-2576, 2019.
The number of human genomes being genotyped or sequenced increases exponentially and efficient haplotype estimation methods able to handle this amount of data are now required. Here, we present a new method, SHAPEIT4, which substantially improves upon other methods to process large genotype and high coverage sequencing datasets. It notably exhibits sub-linear scaling with sample size, provides highly accurate haplotypes and allows integrating external phasing information such as large reference panels of haplotypes, collections of pre-phased variants and long sequencing reads. We provide SHAPET4 in an open source format on https://odelaneau. github. io/shapeit4/ and demonstrate its performance in terms of accuracy and running times on two gold standard datasets:.
While these approaches are extremely efficient to resolve the phase between nearby variants, they do not allow full haplotype resolution across entire chromosome suggesting that additional computational steps at the population level are still required. At this point, it becomes clear that haplotype estimation is now facing two main challenges: computational efficiently to accurately process large scale data sets and data integration to exploit large reference panels of haplotypes and long sequencing reads. In this paper, we describe and benchmark a new method for haplotype estimation, SHAPEIT4, which proposes efficient solutions to these two challenges. Specifically, it allows processing either SNP array or sequencing data accurately with running times that are sub-linear with sample size, therefore making it well suited for very large scale data sets.
In this approach, P is the main parameter controlling the trade-off between speed and accuracy and gives a model in which K varies and adapts to the data and region being processed as opposed to previous methods in which K is usually fixed  19, 23. Indeed, K varies depending on the length of the matches found in the PBWT: longer matches involve smaller K which typically occurs as the algorithm converges, as the level of relatedness between individuals increases and more importantly as the number of samples in the dataset increases. The latter implies, and we later show, that SHAPEIT4 scales sub-linearly with sample size. All other methods proposed so far exhibit at best linear or closeto-linear scaling. In other words, the time spent per genome decreases as the total number of genomes being processed increases.
Phasing large scale datasets We assessed the performance of SHAPEIT4 on genotype data sets with large sample sizes and compared it to the following methods: SHAPEIT3 5 , Eagle2 3,4 and Beagle5  6, 7. To do so, we built multiple subsets of the UKB data set comprising up to ~400,000 individuals, 500 of them being trio children for whom haplotypes can be derived with certainty from the family information. For each phasing scenario, we measured the overall running times and the switch error rates on the 500 trio children. Overall, we find that all tested methods provide haplotype estimates with low error rates that substantially decrease as sample size increases (Figure  2A ). A closer look reveals that both Beagle5 and SHAPEIT4-P=4 significantly outperforms all other methods on all tested configurations.
In terms of running times, we find SHAPEIT4 to be substantially faster than all other methods, no matter the sample size (Figure  2B ). For instance, SHAPEIT4-P=4 is 1. 6 to 3. 6 times faster than Beagle5, 1. 9 to 5. 8 times faster than Eagle2 and 4. 1 to 11 times faster than SHAPEIT3 when phasing from 10,000 to 400,000 individuals. The speedup gets better with sample size as a consequence of sub-linear scaling. Indeed, SHAPEIT4 spends less time per genome on larger sample sizes conversely to all other methods (Figure  2C ). The sublinear scaling can also be noted when looking at the variation of the number of conditioning states with sample sizes and iterations (Supplementary Figure  2A ) or when looking at the coefficients of the fitted functions relating running times T to sample size N on this benchmark dataset (Supplementary Figure  2B ).
Phasing from large reference panels We also assessed the performance of SHAPEIT4 when phasing from large reference panels of haplotypes. To do so, we used the most accurate sets of haplotypes estimated for UKB as part of the first benchmark in which we removed the haplotypes of the 500 trio children. This resulted in reference panels containing from ~1,000 to ~800,000 reference haplotypes. We then phased against these candidate reference panels the 500 trio children alone or as part of larger datasets comprising 5,000 to 50,000 individuals, using either SHAPEIT4, Beagle5 or Eagle2. For each phasing scenario, we measured the overall running times and the switch error rates on the 500 trio children. In this second benchmark, we find the same accuracy patterns than in the first benchmark: overall error rates decrease as reference panel size increases and both SHAPEIT4-P=4 and Beagle5 outperform all other methods no matter the amount of data being processed (Figure  3A-C ).