dataesr/scientific-paragraphs-categorization huggingface dataset:
Finding word boundaries in continuous speech is challenging as there is little or no equivalent of a 'space' delimiter between words. Popular Bayesian non-parametric models for text segmentation  (Goldwater et al. , 2006 (Goldwater et al. , , 2009) )  use a Dirichlet process to jointly segment sentences and build a lexicon of word types. We introduce DP-Parse, which uses similar principles but only relies on an instance lexicon of word tokens, avoiding the clustering errors that arise with a lexicon of word types. On the Zero Resource Speech Benchmark 2017, our model sets a new speech segmentation state-of-theart in 5 languages. The algorithm monotonically improves with better input representations, achieving yet higher scores when fed with weakly supervised inputs. Despite lacking a type lexicon, DP-Parse can be pipelined to a language model and learn semantic and syntactic representations as assessed by a new spoken word embedding benchmark.
(2007) 's implementation, with similar or superior performances. Because of our 'no clustering' approach, we cannot evaluate our model using the word typebased metrics of the Zero Resource speech segmentation challenge  (Versteegh et al. , 2016; Dunbar et al. , 2017). Here, we use the two classes of metrics: boundary-related metrics (Token and Boundary F-score) that do not refer to types, and high-level word embedding metrics. The idea is use a downstream language model to learn high level word embeddings and evaluate them on semantic and Part-Of-Speech (POS) dimensions. We rely on a semantic similarity task from  (Dunbar et al. , 2020)  and introduce two new metrics. The combination of our contributions yields an overall system, DP-Parse, that sets a new state-ofthe-art on speech segmentation for five speech corpora by a large margin.
g. Correspondance Auto-Encoder from  Kamper (2018)  where the first element of the pair is compressed and decoded into the second element). Recently, a growing body of work leverage multilingual labelled dataset to build SSEs that can, to an extent, generalize to unseen languages  (Hu et al. , 2020a,b; Jacobs et al. , 2021). In this work, we use a state-of-the-art selfsupervised system SSE model from  Algayres et al. (2022).
Chung and Glass (2018)  showed that gold word boundaries provide sufficient information to learn good quality semantic embeddings from raw au- dio using a skipgram or word2vec objective, as assessed by semantic similarity benchmarks adapted to speech inputs. These datasets were also used to compile the sSIMI benchmark in the Zero Speech Challenge 2021  (Nguyen et al. , 2020) , but turns out to give very low and noisy results (at least for small training sets). Here, we introduce two new metrics evaluating semantic and grammatical similarity that we show to be more stable.
Density estimation from Gaussian-smoothed k-Nearest-Neighbors In text, the exact frequency of a string of letters can be computed by counting how often it occurs in the corpus. In speech, due to acoustic variability, most of the counts would be 1, and clustering methods cause too many errors to get reliable count estimates (more details in appendix A. 2. 1). Here, we follow the method in L 0w ≈ F (w) = k j=1 exp -β Ew-E j 2 2 (1) F is the Parzen-Rosenblatt window method for density estimation  (Parzen, 1962)  and returns a continuous value between 0 and k, estimating L 0w. It rests on two hyper-parameters: k which is set to 1000 and β, the standard deviation of the Gaussian kernel. To set β, we follow the observation from  Algayres et al. (2022) : around 50% of the segments in the development dataset of the Zerospeech Challenge 2017 appear to have a frequency of one.
Here, we assume that a Speech LM has been trained on SSE inputs on a given dataset with a given segmentation and can be used at test time to provide embeddings associated to a set of test words. As our segmentation models work on whole utterances, we present the test words in continuous utterances and then mean pool the HLSEs from the Speech LM that overlap with that word to obtain a single vector. An overview of the building of generic HLSEs and their evaluation is shown in Figure  3. We introduce two zero-shot tasks to evaluate HLSEs that do not require training a further clas-   2013 ) which computes a discrimination score over sets of (A, B, X) triplets. For each triplet, A and X belong to the same category and B plays the role of a distractor. The task for the model is to find the distractor in each triplet based on the cosine distance between the HLSEs of A, B and X.
Datasets and Experimental settings The Zerospeech Challenge 2017  (Dunbar et al. , 2017)  provides five corpora to evaluate speech segmentation systems. These corpora are composed of speech recordings from different languages split into sentences using Voice Activity Detection. Three corpora  (Mandarin, 2h30; English, 45h; French, 24h)  are used for development, and two 'surprise' corpora for testing  (German, 25h; Wolof, 10h). On each corpus, a separate SSE model from Algayres et al. (  2022 ) is trained and a separate run of DP-Parse is performed to produce a full-coverage segmentation. DP-Parse's hyper-parameters from Table  2 , as well as q's formula (Equation  5 ), were grid-searched to maximize token-F1 segmentation scores over the three development datasets. The two remaining test sets are used to show generalization of DP-Parse to new unseen languages.
More details on hyperparameters search in Appendix A. 2. 2. For the ABX and sSIMI tasks, we train a BERT model as a Speech Language Model on the training set of the Librispeech dataset  (Panayotov et al. , 2015) , composed of 960 hours of English recordings. We proceed by first training a SSE model from  Algayres et al. (2022)  on the Librispeech. Then, this latter dataset is segmented using DP-Parse. Each segmented speech sequence is aggregated into a single vector using the pre-trained SSE model. Segmented sentences are used to train a BERT model with masked language modelling and the Noise Contrastive Estimation (NCE) loss  (Gutmann and Hyvärinen, 2010)  (see Figure  4 ). The BERT model is composed of 12 layers, 12 attention heads per layer with 768 neurons each and a FFN size of 3072 neurons.
Results on word-level segmentation Regarding text segmentation, DP-Parse compares favorably to the original Dirichlet Process Unigram model from  Goldwater et al. (2009). It produces a 21 points increase in token F1 compared to the original version (and 3 points increase in boundary F score) for a 10x runtime speed-up (Table  3 ) under the Adaptor Grammar implementation of  Johnson et al. (2007). Regarding speech segmentation, we compare DP-Parse with the three best speech segmentation models submitted at the Zerospeech Challenge 2017  (Bhati et al. , 2020; Kamper et al. , 2017b; Räsänen et al. , 2015). Table  4  reports the token F1 and boundary F1 obtained by these models over the 5 datasets of the Zerospeech challenge 2017. Across all corpora, DP-Parse outperforms its competitors by at least 5 points in both boundary and token F1.
We also introduce a naive baseline that draws boundaries every 120 milliseconds, disregarding the content of the speech signal. It turns out to be surprisingly competitive with all speech segmentation systems except DP-Parse. The latter is the only existing speech segmentation system that beats the naive baseline on all languages. Most speech segmentation systems from the Zerospeech Challenge 2017 rely on off-the-shelf self-supervised representations of speech. The hope is that, without modification, these systems would benefit from future improvements in speech representations and mechanically lead to higher segmentation scores. Yet, such hopes have never been guaranteed by explicit experiments. Here, we test for the ability of DP-Parse to improve with better inputs. For that, we use the weaklysupervised version of the SSE model from Algayres et al.
Conclusion and open questions We introduce a new speech segmentation pipeline that sets the state-of-the-art on the Zerospeech's datasets at 16. 8 token F1. The whole pipeline needs no specific tuning of hyper-parameters, making it ready to use on any new languages. We showed that the problem of speech segmentation can be reduced to the problem of learning discriminative speech representations. Indeed, using different level of supervision, our pipeline reaches up to 35 token F1 score. Therefore, as long as the field of unsupervised representation learning makes headway, this method should automatically produce higher token F1 scores. A first avenue of improvement of the current work is in the SSE component. Here, we took the system described in  Algayres et al. (2022)  out of the box, and we showed good performance in speech segmentation compared to the state of the art, but there was still a large margin of improvement compared to text-based system.
A recent unpublished paper  (Kamper, 2022)  came to our attention based on the non-lexical principle and showed similar or slightly better results than ours on a subset of the ZR17 language. Kamper (2022) also uses a segmentation lattice that resembles ours for inference. Yet, as we have shown, our system is monotonous with input embedding quality, and can therefore reach much better performance if the speech sequence embedding component were better trained. Further work is needed to improve the SSE component based on purely unsupervised methods. Despite our computational speedup, our current system would face challenges to scale up the approach to larger datasets than Librispeech. While it uses FAISS, an optimized search library, it is unclear that storing each instance of a possible   2009 ).
A Appendix A. 1 Construction of triplets for ABX sem and ABX P OS Let's write the series of triplet from our ABX tasks: (A i , B i , X i ) 0≤i<N. For all i, A i is defined as a tuple (R a i , s a i , e a i , t a i ) where R a i is a recording of a whole sentence and s a i and e a i are the temporal boundaries of a word with phonetic transcription t a i. B i and X i are defined identically. The sentences are extracted from the Librispeech dataset, a 960 hours corpus of read English literature. The Speech LM is asked to encode the whole sentences (R a i , R b i , R x i ) and compute the three word embeddings for the words of interest using the provided timestamps. The ABX score is given by the following formula: ABX(T, d) = 1 N N -1 i=0 1 d(f a i ,f x i )<d(f b i ,f x i ) where T is a collection of triplet word embeddings (f a i , f b i , f x i ) 0≤i<N and d is the cosine distance.
ABX sem is composed of 502 pairs (evenly split into a development and test set) of words created using a synonym dictionary. By sampling a list distractor for each pair, we reached 1557 different triplets. For each unique word found in the triplets, we sample 10 occurrences from the sentences of the Librispeech corpus. In total, ABX sem is computed over 1. 5M triplets. ABX P OS is composed of 9997 pairs of words built from the WordNet database  (Fellbaum, 1998). The words in each pair have the same POS tags that can be either noun, verb or adjective. By sampling distractors and 10 occurrences of each word types, we reached 37. 3M triplets.
A. 2. 2 DP-Parse hyper-parameters We provide in Table  7 , DP-Parse speech segmentation performances, as measured by token-F1 scores, over three datasets (Mandarin, English and French) for different hyper-parameters values. Surprisingly, increasing the number of neighbors (k), the number of samples in L 0 (#L 0 ) or the beam size does not improve token-F1 scores. DP-Parse time complexity scales linearly with each Table  7 : Token F1-score on the number of neighbors for the k-NN search (k), the concentration parameter (alpha 0 ), the beam-search size (beam), the number of samples in L 0 (#L 0 ) and the pair (δ,γ) from the penalty function. The default parameter values are k = 100, α 0 = 100, beam = 10, #L 0 = 1M , γ = 1. 8 and δ = 4 (also in bold in the table). of these three parameters, therefore we keep their values as low as possible.
Token and Boundary F1 scores (the higher, the better) for speech and text segmentation models across development and test set from the Zerospeech Challenge 2017. The average scores are computed over the five datasets. *: ours, ×: Räsänen et al. (2015), +: Bhati et al. (2020), ∨: Kamper et al. (2017b), ‡: Goldwater et al. (2009).
Distribution of the Q sample, and the Q sort Participants then undertook an exercise designed to measure and map features of one of landscapes previously captured by UAV and modelled in 3D, the Chale Terraces on the south coast of the Isle of Wight, UK. After completing the tasks, they were guided to undertake the Q sort process which was accessed via a web browser, utilising the opensource HTMLQ software  (Aproxima, 2015). Participants were asked to consider the statements against a condition of instruction which asked them to reflect upon their experience and assess to what extent they agreed or not with the statement in hand. The statements were then sorted onto a grid with a quasi-normal distribution, with the distribution ranging from two poles representing points of most agreement or "most agree" to "most disagree", with each position along this spectrum given a score.
Analysis The data were captured via a database, which was downloaded and processed by the freely available Ken-Q Analysis (v1. 0. 6) software developed by  Banasick (2019). Analysis of the entire data set was undertaken using exploratory factor analysis. As this has long been privileged over allied techniques, such as PCA  (Watts & Stenner, 2014). This analysis resulted in a small of factors that represent common patterns found within the sorting of statements by the participants. The degree to which these factors are associated with each participant's holistic opinion (as made manifest in their sort), is indicated by the factor loading. Finally, a final composite Q sort (or a factor array) represents each factor. This shows how each statement would have been scored purely under that factor.
Acknowledgements I would like to thank  Bruno Duchesne  for his helpful comments on a first version of the paper. I am also grateful to the anonymous referee for finding many additional mistakes in the submitted version and making useful suggestions, thus making the paper much more readable. Research partially supported by  ANR   AGRUME  ( ANR-17-CE40-0026 ) and  ANR AODynG  ( 19-CE40-0008-01 ).
Objective: The aim of this study was to develop an accurate regional forecast algorithm to predict the number of hospitalized patients and to assess the benefit of the Electronic Health Records (EHR) information to perform those predictions. Materials and Methods: Aggregated data from SARS-CoV-2 and weather public database and data warehouse of the Bordeaux hospital were extracted from May 16, 2020 to January 17, 2022. The outcomes were the number of hospitalized patients in the Bordeaux Hospital at 7 and 14 days. We compared the performance of different data sources, feature engineering, and machine learning models. Results: During the period of 88 weeks, 2561 hospitalizations due to COVID-19 were recorded at the Bordeaux Hospital. The model achieving the best performance was an elastic-net penalized linear regression using all available data with a median relative error at 7 and 14 days of 0.
136 [0. 063; 0. 223] and 0. 198 [0. 105; 0. 302] hospitalizations, respectively. Electronic health records (EHRs) from the hospital data warehouse improved median relative error at 7 and 14 days by 10. 9% and 19. 8%, respectively. Graphical evaluation showed remaining forecast error was mainly due to delay in slope shift detection. Discussion: Forecast model showed overall good performance both at 7 and 14 days which were improved by the addition of the data from Bordeaux Hospital data warehouse. Conclusions: The development of hospital data warehouse might help to get more specific and faster information than traditional surveillance system, which in turn will help to improve epidemic forecasting at a larger and finer scale.
LAY SUMMARY The objective of this work was to develop a forecast algorithm to predict the number of hospitalized patients at Bordeaux Hospital. In addition, we assessed the benefit of the Electronic Health Records (EHRs) information to perform those predictions. To perform this task, we used data between May 16, 2020, and January 17, 2022, from national database on SARS-CoV-2 epidemics, public database on weather and the data warehouse of the Bordeaux hospital. The outcomes were the number of hospitalized patients in the Bordeaux Hospital at 7 and 14 days. During the period of 88 weeks, 2561 hospitalizations due to COVID-19 were recorded at the Bordeaux Hospital. The best model had an error of 13. 6% at 7 days and 19. 8% at 14 days. EHRs from the hospital data warehouse improved the performance by 10% at 7 days and 20% at 14 days.
Because many factors change over time (eg, population behavior, government policies, vaccine coverage, virus strain), long-term forecast of COVID-19 is impossible. In France, several approaches to short-term epidemic forecast have been proposed relying on linear algebra,  9  ensemble methods  10  or neural networks. 11, 12 All models aimed to forecast hospitalizations (among other things), except the one proposed by Carvalho et al,  11  which focused on cases, ICU, and deaths. The data used were hospitalizations, Reverse Transcriptase Polymerase Chain Reaction (RT-PCR), Intensive Care Units (ICU), weather, mobility, vaccination, variants of concern, and mask wearing policy data. The approach proposed by Mohimont et al, based on Convolutional Neural Networks seems to achieve the best results with a normalized root mean square deviation at 14 days of 3.
21, 22 hanks to those previous experiences, the Bordeaux hospital data warehouse was used, during the pandemic, to describe the current state of the epidemic at the hospital level on a daily basis. Those data were then used in the forecast model including: hospitalizations, hospital and ICU admission and discharge, ambulance service notes, and emergency unit notes. Concepts related to COVID-19 were extracted from notes by dictionary-based approaches (eg, cough, dyspnea, COVID-19). Dictionaries were manually created based on manual chart review to identify terms used by practitioners. Then, the number and proportion of ambulance service calls or hospitalization in emergency units mentioning concepts related to COVID-19 were extracted. Detail of features is available in Supplementary Table  S1.
Modeling strategy To train the model, the primary analysis used a matrix where each row corresponds to a day and each column to a feature from Gironde (ie, the Bordeaux department) open source and Bordeaux Hospital EHR data. Other department data were not added to limit the number of features. Prediction performance were evaluated depending on the data used for the forecast, initially using only hospitalization and RT-PCR and progressively adding SARS-CoV-2 incidence in Gironde, weather, EHRs, Vaccine, and Variant data. Because of the elastic-net penalization,  24  each day the model might select different features by shrinking beta coefficient of unimportant features to zero. An additional analysis considered the Bordeaux Hospital as an additional unit among the other French departments. The advantage was to leverage the information from all the departments; the drawback was the impossibility to include specific information only available for the Bordeaux Hospital.
Model evaluation The models were evaluated every day on the data available from December 1, 2020 to January 17, 2022 (ie, data from May 16, 2020 to December 1, 2020 were used for training only). The models were trained using all prior data available at a given date d and the forecast of the number of hospitalizations at both 7 and 14 days after d was evaluated. Prediction performances were evaluated according to median absolute error (MAE) and median relative error (MRE). Median was chosen over mean because it is less sensitive to extreme values which are frequent when the observed outcome is low (ie, a small absolute error could result in a huge relative error). Bootstrapped 95% confidence interval was provided with 500 samples (more details in the Supplementary Material). Graphical evaluation was also performed.
2C  supports those findings and shows that using EHR data anticipates the April peak, better forecasts the 2021-2022 winter increase and is more robust to the hospital cluster of December 2020. Last, we evaluated the information added by the number of vaccinated people and the variants distribution. Those additional data provided similar forecast performance and vaccine-RT-PCR interaction improved forecast at 7 days but decreased it at 14 days. Similar results were obtained with the additional analysis considering the Bordeaux Hospital as an additional department; results are available at Supplementary Table  S2. Overall, we considered the model including all the data but without the vaccine-RT-PCR interaction as the best model. Information retrieval from data warehouse was done with a 1day delay whereas it was a 5-day delay for hospitalizations and RT-PCR at the department level.
Therefore, forecasting using data from Bordeaux Hospital data warehouse gives a 4-day advantage over usual open data. To evaluate its consequences, we compared 7-day forecast with 11-day forecast and 14-day forecast with 18-day forecast. As shown in Table  1 , this resulted in a MAE and MRE decrease of respectively 2. 08% and 4. 9% at 7 days and of respectively 2. 6% and 3. 7% at 14 days.
We also observed that RT-PCR-variant interaction deteriorated the model performance at 14 days. This was explained by a spurious positive correlation between the interaction term and the hospitalizations. ] [27]  The moderate impact of the of Gironde data to the model might be explained by: (1) the information already being captured by the local data from the Bordeaux Hospital and (2) the data collection being slower at the department level than in the hospital. Data consolidation (ie, the update of prior data thanks to new information) was not taken into account because of lack of availability of data versioning. For instance, hospitalization related to Sars-Cov-2 is sometimes identified in the EHR through billing codes in International Classification of Disease, 10th revision, which are only available at the end of the sojourn.
ACKNOWLEDGMENTS Preprint of this paper is available at SSRN: https://ssrn. com/abstract=4071506 or http://dx. doi. org/10. 2139/ssrn. 4071506. Code of this paper is available on https://github. com/thomasferte/PredictCovi-dOpen. git. Computer time for this study was provided by the computing facilities of the  MCIA (M esocentre de Calcul Intensif Aquitain). This work has been partly supported by  Inria ,  Mission COVID19 ,  GESTEPID  project, and  Nouvelle Aquitaine  regional funding (Prediction territorial  COVID N 1333140 ). We thank  Louis CAPITAINE  and  Robin GENUER  for their help with Fr echet random forest. We thank the  Bordeaux University Hospital  Covid-19 crisis task force for their help with the data curation and the interpretation:  Isabelle FAURE ,  Philippe REVEL ,  Eric TENTILLIER, Jean-Michel DIN-DART ,  Didier GRUSON ,  Olivier JOANNES-BOYAU ,  Jean-Marie Denis MALVY ,  Thierry PISTONE ,  Didier NEAU ,  Duc NGUYEN ,  Marie-Edith LAFON ,  Mathieu MOLIMARD ,  Thierry SCHAEVER-BEKE ,  Nicolas GRENIER ,  Nathalie SALLES ,  Francois ROUANET.
DATA AVAILABILITY Open data used for this study are available on data. gouv. fr  14 and ncei. noaa. gov. 15 EHR data used for this study are available upon request. Data available from the Dryad Digital Repository: https:// doi. org/10. 5061/dryad. hhmgqnkkx. Code available from github: https://doi. org/10. 5281/zenodo. 6595011.