dataesr/scientific-paragraphs-categorization huggingface dataset:
Weighting Schemes Given a corpus of documents D = {d 1 , d 2 ,. , d N }, where N = |D| is the total number of documents in the dataset and n the number of documents where some term t appears. The TF-IDF weight is computed by multiplying the augmented term frequency T F (t, d) = K + (1 -K) • f t,d max t ∈d (f t ,d ) ) by the inverted document frequency IDF (t, D) = 1 + log N n , i. e. , T F IDF (t, d, D) = T F (t, d) • IDF (t, D). The augmented form of T F prevents a bias towards long tweets when the free parameter K is set to 0. 5  [27]. It uses the number of co-occurrences f t,d of a word in a document, normalized with the frequency of the most frequent term t , i. e. , max t ∈d (f t ,d ). The Okapi BM25 weight is given in Equation (  1   [28, 29, 30]. Okapi(t, d, D) = T F IDF (t, d, D) • (k 1 + 1) T F (t, d) + k 1 • (1 -b + b • ||d|| avg d ∈D (||d ||) ) (1) To extract top-k keywords, the overall relevance of a term t for a given corpus D is computed as the sum of all the TF-IDF (Equation (  2 )) or Okapi BM25 (Equation (  3 )) weights for that term.
c 8 is the JOIN condition between the documents and vocabulary entities. c 9 is the JOIN condition between the vocabulary and words entities. c 10 is the JOIN condition between the documents and geo location. Function f w is used to compute the weighting schema using nested queries:  words : [ { " t f " : 1 , " count " : 1 , "word" : "amanda" } , { " t f " : 1 , " count " : 1 , "word" : " c a r " } , { " t f " : 1 , " count " : 1 , "word" : " headache " } ] , d a t e : ISODate ( "2015-09-17T23 : 3 9 : 1 1 Z" ) } Queries. In DODBMSs, user-defined functions written in JavaScript are used to compute top-k keywords, regardless of the schema. For T 2 K 2 , the TF-IDF weight can take advantage of both native database aggregation (aggregation pipeline -AP) and MapReduce (MR). However, due to the multitude of parameters involvedand the calculations needed for the Okapi BM25 weighting scheme, the NA method is usually difficult to develop.
Document-ortiented implementation Dataset. To implement T 2 K 2 D 2 's star schema in a DODBMS each dimension becomes a nested document, thus a document contains multiple nested docu-ments  [31] , one for each dimension. Figure  6  presents a document sample for implementing a multidimensional model in a DODBMS. " g e n d e r " : " male " } , " t i m e d i m e n s i o n " : { " minute " : 3 9 , " hour " : 2 3 , " day " : 1 7 , "month" : 9 , " y e a r " : 2 0 1 5 , " f u l l d a t e " : ISODate ( "2015-09-17T23 : 3 9 : 1 1 Z" ) } , " l o c a t i o n d i m e n s i o n " : { "X" : 3 2 , "Y" : 79 } , " words " : [ { " t f " : 1 , " count " : 1 , "word" : "amanda" } , { " t f " : 1 , " count " : 1 , "word" : " c a r " } , { " t f " : 1 , " count " : 1 , "word" : " headache " } ] } MongoDB's performance is tested both in a single-instance and a distributed environment.
The distributed environment is composed of a configuration node and 5 slave nodes; each slave becoming a worker in the MR algorithm. The database is distributed using horizontal scaling to test whether weighting scheme computing time decreases as the number of nodes increases. In MongoDB, horizontal scaling is achieved using sharding, and each individual data partition is refered as a shard  [32]. The relational database implementations are tested only on a single-instance environment, because the Oracle Real Application Cluster that is needed to distribute Oracle databases is not free; and because PostgreSQL does not distribute natively, requiring third party software. Besides indexes on primary keys, no other indexes are used in our tests. The sharding key in the MongoDB distributed environment is the tweet's unique identifier.
Dataset Experiments are done on a 2 500 000 tweets corpus. The initial corpus is split into 5 different datasets that all keep an equal balance between the number of tweets for both genders, location and date. These datasets contain 500 000, 1 000 000, 1 500 000, 2 000 000 and 2 500 000 tweets, respectively. They allow scaling experiments and are associated to a scale factor (SF ) parameter, where SF ∈ {0. 5, 1, 1. 5, 2, 2. 5}, for conciseness sake.
r. t. TF-IDF. In PostgreSQL, the biggest performance difference between TF-IDF and Okapi BM25 is obtained for Q1 (Figure  7d ) due to the large number of rows returned when joining all tables. Figure  8  presents a comparison by database system and weighting scheme of query response time for retrieving top-k documents w. r. t. scale factor SF. In MongoDB, computing TF-IDF is faster than computing Okapi BM25 for all queries, regardless of SF (Figure  8a ). The same results are observed in the distributed environment (Figure  8b ). In this case the performance of MongoDB is stable, regardless of the environment (single node or distributed), the performance curve increases linearly (Figures  8a  and 8b ). Such results are an expected consequence of the variation of query selectivity and complexity, as seletivity decreases and complexity increases from Q1 to Q2.
Furthermore, as SF increases, so does seletivity in all queries. The execution times of Q1 and Q3 for Oracle are getting worse when SF increases, since selectivity increases with the number of documents. In Oracle, the execution time for computing TF-IDF is half of the execution time for computing Okapi BM25 (Figure  8c ) regardless of SF. This is a direct consequence of query complexity. The complexity of queries that use the TF-IDF weighting is indeed lower than those that use Okapi BM25 (Table  4 ). Moreover, in Oracle, the execution time is unstable as the standard deviation fluctuates w. r. t SF due to automatic cache cleaning when query results are too big to be stored in the database's internal buffers. In PostgreSQL, the biggest performance difference between TF-IDF and Okapi BM25 is obtained for all the queries as the SF increases (Figure  8d ).
MongoDB in a distributed environment has the best performance time regardless of the weighting schema. Query execution performance in MongoDB is improved because the workload is distributed between shards and results are aggregated after each shard finishes a task. 5. 3. 2. T 2 K 2 D 2 Weighting schema comparison. For MongoDB the same patterns in performance appear as for the T 2 K 2 test scenarios. The time performance is almost non-existent between the queries run on T 2 K 2 (Figures  7a  and 7b ) and the ones executed on T 2 K 2 D 2 (Figures  13a  and 13b ). This pattern emerges because MongoDB is a schemaless database and the data modeling does not really influence performance. The same results are optained for the top-k documents (Figures  14a  and 14b ). Using a multidimensional model improves performance with both Oracle and PostgreSQL.
Using such a model, the indexes used for primary and foreign keys to associate entities make query execution plan optimization easy, because a join does not have to pass through multiple tables as in the normalized model. 5  and 6 ), plays an important role. In Oracle and PostgreSQL, Okapi BM25 weights are computed through nested queries, but in MongoDB, separate queries are needed. To improve MongoDB query performance, an indexing and sharding key analysis should be made, but such a extensive study is outside the scope of this paper. MongoDB in a single environment has the overall worst performance between the tested databases for top-k keywords (Figures  15  and 16 ), while PostgreSQL for top-k documents (Figures  17  and 16 ). conceptual level, and thus does not include any bridge tables at the logical level for expressing many-to-many relationships, as in T 2 K 2.
Conclusion Jim Gray defined four primary criteria to specify a "good" benchmark  [36] : • Relevance: The benchmark must deal with aspects of performance that appeal to the largest number of users. Considering the wide usage of top-k keywords and documents queries in various text analytics tasks, we think T 2 K 2 and T 2 K 2 D 2 fulfills this criterion. We also show in Section 5 that our benchmark achieves what it is designed for. • Portability: The benchmark must be reusable to test the performances of different database systems. We successfully used T 2 K 2 and T 2 K 2 D 2 to compare two types of database systems, namely relational and documentoriented systems. • Simplicity: The benchmark must be feasible and must not require too many resources. We designed T 2 K 2 and T 2 K 2 D 2 with this criterion in mind (Section 3), which is particularly important for reproducibility.
We notably made up parameters that are easy to setup. • Scalability: The benchmark must adapt to small or large computer architectures. By introducing scale factor SF , we allow users to simply parameterize T 2 K 2 and T 2 K 2 D 2 and achieve some scaling, though it could be pushed further in terms of data volume. Regarding experimental results, for T 2 K 2 's top-k keywords the best TF- When using a multidimensional model with T 2 K 2 D 2 , MongoDB registers the same performance or both top-k keywords and top-k documents as with T 2 K 2. Oracle in a single instance has the overall best performance for both tasks. For top-k keywords MongoDB in a single instance has the worst performance while for top-k documents, PostgreSQL has the worst performance. In future work, we plan to expand T 2 K 2 and T 2 K 2 D 2 dataset significantly to aim at big data-scale volume (scalability).
We also plan to adapt our benchmarks so that they run in the Hadoop and Spark environments. Moreover, we intend to further our proof of concept and validation efforts by benchmarking other NoSQL database systems and gain insight regarding their capabilities and shortcomings (relevance). Eventually, we considered in this paper that TF-IDF and Okapi BM25 were enough for machine learning tasks. Yet, the next version of our benchmarks should include other weighting schemes, such as KL-divergence  [37] , also to improve their relevance.
id,words. word,fw ( σ c1∧c2∧c3∧c4 ( documents c5 documents authors c6 authors c7 genders c8 vocabulary c9 words c10 geo location))) 4. 2. 2. Document-ortiented implementation Dataset. In a Document Oriented Database Management System (DODBMS), all information is typically stored in a single collection. An example of DODBMS document is presented in Figure 4. The many-to-many Vocabulary relationship from Figure1is modeled as a nested document for each record. The information about user and date become single fields in a document, while the location becomes an array.
How physical dimensions govern children's perception, language acquisition and cognition is an important question in developmental science. Here we use the psychophysical technique of Maximum Likelihood Conjoint Measurement (MLCM) as a novel approach to investigate how infants combine information distributed along two or more dimensions. MLCM is based on a signal detection model of decision that allows testing of several models of how observers integrate information to make choices. We tested 6-month-old infants' preferential looking to "green" stimuli that covaried in lightness and chroma and analysed infant preferences using MLCM. The findings show that infant looking is driven primarily by lightness, with darker stimuli having a greater preference than lighter, plus a small but significant positive contribution of chroma.
To test whether MLCM can be applied to infants' preferential looking and to further understand infant colour perception, we conducted an experiment using a forced-choice preferential looking method with 6-month-olds and adult observers  (Teller, 1979). The method involves eye-tracking observers' responses to pairs of stimuli, and coding which of the pair they look at first. For infants, the pairs of stimuli were randomly selected from a 3-by-3 matrix of green stimuli in which lightness and chroma independently covary. All stimulus levels were above threshold for infants at 6 months according to threshold data from  Knoblauch et al. (2001). The adult experiment used the same range of lightness and chroma, but the range was divided into four stimulus levels rather than three. We were able to use more stimulus levels with adults as they have better colour discrimination and can tolerate a longer experimental procedure.
We analysed the data using MLCM to estimate the relative contributions of lightness and chroma to the observers' decisions (first look). The outcome variable of our MLCM analysis is parameterised to be on the scale of d' (units of the standard deviation for each scale value) from Signal Detection Theory (e. g. Gerardin, Devnick, Dojat, & Knoblauch, 2014; Green & Swets, 1966; Ho, Landy, & Maloney, 2008; Rogers, Knoblauch, & Franklin, 2016; Stanislaw & Todorov, 1999). Here, we take d' to reflect the perceptual salience of a stimulus, not necessarily its discriminability, by measuring which of two stimuli observers look at first. We follow Kaldy and Blaser's definition of salience as, "the visual system's real-time assessment of the behavioural relevance (current importance) of information in the scene-a prioritization that drives attention allocation and consequent eye movements"  (2009, p.
Data analysis For each participant and on each trial, the stimulus looked at first was recorded as the choice. The data were analysed using Maximum Likelihood Conjoint Measurement; this analysis is described in full elsewhere  (Knoblauch & Maloney, 2012; Rogers et al. , 2016) , and we provide a brief summary here. On each trial, a pair of stimuli are randomly selected from the stimulus matrix (Figure  1 ) for presentation. The pair can be indexed in terms of their ordinal chroma levels (w, y) and lightness levels (x, z). For example, in the pair C3L1 (bottom right stimulus) plus C2L2 (centre stimulus), w = 3, y = 2, x = 1 and z = 2. It is assumed that when viewing the stimulus pair, the observer forms the noise-contaminated decision variable: ∆ = 𝛿 + 𝜀 = 𝛹(𝜑 𝑦 𝐶 , 𝜑 𝑧 𝐿 ) -𝛹(𝜑 𝑤 𝐶 , 𝜑 𝑥 𝐿 ) + 𝜀, Equation 1 where  = Ψ(   ,    ) -Ψ(   ,    ) and the Ψ terms are internal responses that depend on the contributions of the stimulus lightness and chroma to the perceptual salience.
The mixed-effects models were fit using the glmer function in the lme4 package  (Bates et al. , 2014)  within the software R (R Core Team, 2017). We can use the same framework to fit the independence model by dropping one of the terms, covariates, dC or dL, and a saturated model by including terms that are the product of the two covariates. The three models are then evaluated using nested likelihood ratio tests. Likelihood ratio tests indicated that the additive model fit the data significantly better than the independence models for lightness ( 2 (1)= 19. 79, p <<. 001) and for chroma ( 2 (1)= 5. 90, p =. 015). However, the saturated model was not an improvement over the additive model ( 2 (1)= 0. 289, p =. 591). This suggests that both chroma and lightness contribute to infants' looking preference with no interaction between the two dimensions.
Influence of sampling constraints in infant data The adults were tested with 16 stimuli generating 240 ordered pairs, while the infants were tested with only 9 stimuli generating 72 ordered pairs. In addition, every adult completed all 240 trials while only about one quarter of the infants completed the full set of trials. The adult and infant data sets differ both quantitatively and qualitatively. To what extent could the differences in sampling account for these differences? To address this question, we repeatedly simulated reduced and variable size data sets by sampling from the 12 adult data sets, to mimic the sampling characteristics of the infant data sets. We analysed each simulated data set with a GLMM to assess whether the sampling influenced the magnitude and the slopes of the estimated chroma and lightness functions.
Acknowledgements We thank the adult participants, and the infant participants and their parents. This research was funded by a  European Research Council  Grant project, " CATEGORIES ",  283605  to AF. KK was supported by the following grants:  LABEX CORTEX  ( ANR-11-LABX-0042 ) of  Université de Lyon  ( ANR-11-IDEX-0007 ) operated by the  French National Research Agency (ANR) ,  ANR-11-BSV4-501 ,  CORE-NETS ,  ANR-14-CE13-0033 ,  ARCHI-CORE ,  ANR-15-CE32-0016 ,  CORNET ,  ANR-17-NEUC-0004 ,  A2P2MC ,  ANR-17-HBPR-0003 , CORTICITY. There are no conflicts of interest.
Most social platforms offer mechanisms allowing users to delete their posts, and a significant fraction of users exercise this right to be forgotten. However, ironically, users' attempt to reduce attention to sensitive posts via deletion, in practice, attracts unwanted attention from stalkers specifically to those (deleted) posts. Thus, deletions may leave users more vulnerable to attacks on their privacy in general. Users hoping to make their posts forgotten face a "damned if I do, damned if I don't" dilemma. Many are shifting towards ephemeral social platform like Snapchat, which will deprive us of important user-data archival. In the form of intermittent withdrawals, we present, Lethe, a novel solution to this problem of (really) forgetting the forgotten. If the next-generation social platforms are willing to give up the uninterrupted availability of non-deleted posts by a very small fraction, Lethe provides privacy to the deleted posts over long durations.
However, the system administrator can reduce the availability of the system by a small fraction and set it to 90%, which drops the adversary's precision back to 20%. For a largescale system such as Twitter, with trillions of tweets, even precision of 80% can result in a significant overhead for the adversary (investigating 20 million non-deleted tweets falsely marked as deleted each day). Finally, we evaluate the effect of our scheme on Twitter dataset to show the feasibility of Lethe in a realworld scenario. We show that our proposal, while maintaining a trade-off between availability and privacy, also allows interactions in the system without much interruption. Specifically, leveraging real-world interaction data from Twitter we show that, by applying Lethe the utility (i. e. user interactions with posts) remains above 99% even when content availability is 85%.
Collecting a utility dataset: We need to ensure that Lethe preserves utility for all normal users of our system. To create a collective random sample of such users, we first take all the users who appeared in the 1% random Twitter sample collected in the first week of November 2011. Then we divide the users into five exponential buckets based on their number of followers (i. e. by their popularity) and randomly sampled 500 users from each bucket. We did this subsampling in mid-February 2016. Thus we end up with 2,500 random users. We collected all the tweets posted by these users (respecting Twitter's limit of 3200 most recent tweets per user) and all the retweets of those tweets on end of February 2016. Out of 2,500, 6 users have made their account private between the time of subsampling and the time of all-tweets collection.
Setup for measuring utility in presence of Lethe: We have experimented with setting the platform availability to 85%, 90% and 95%. We again set the mean down time to 1 hour and set mean up times to satisfy the availability requirements. The up and down distributions are geometric and negative binomial respectively. Recall that the negative binomial distribution needs a shape parameter along with the mean. Although we are not considering the adversary in the utility experiment, to be consistent with the privacy analysis, we repeat the experiment for the shape parameters from Table  3. Specifically, we simulate Lethe for each of the posts in our utility dataset. Note that, an original retweet happening in a down duration (i. e. , when the tweet is hidden) is essentially missed and thus platform utility is affected.
B Modeling Deletion Distribution Twitter provides 1% random sample of all the tweets posted in real time via their streaming API  [12]. We collect this random sample continuously for more than one year (From October 2015 to February 2017) and leverage all the tweets provided in this 16 months random sample. There is a total of 834, 750, 752 tweets (not including retweets) in our dataset. Additionally, when a tweet is deleted Twitter provides a deletion notification to the data collectors. We leverage these deletion notifications in the same way as in  [16]  and for each tweet, we detect whether the tweet is deleted at some time (along with the timestamp of deletion). We observed in Section 4. 2 that, for the Bayesian analysis we need to have a prior distribution for the posts deletion times.
We leverage our deletion notifications to create the creation and deletion history for each tweet. We found that 115, 785, 407 tweets in our dataset are deleted. Using the creation and deletion timestamps of these tweets we also ascertain how long did it take for each tweet to be deleted, i. e. the deletion times for each tweet. Figure  10  shows the probability density of time taken for deletions with bins of one minute. We further chose to fit a closed form probability distribution for this empirical deletion time distribution. The reason is: a fitted distribution enables a researcher to study our system even when the data is much more than 16 months (our data collection period in this work). Based on our study on more than 110 million tweet deletions over the period of 16 months, we found that the deletions are well modeled by a generalized Pareto distribution with a shape parameter of k = 10 and scale factor of σ = 11.
E An Overview of Lethe Implementation In section 5 we presented the basic steps of Lethe. However in our paper we considered that Lethe should be applied to each single post. So a very pratical question is: How to efficiently implement Lethe in a platform? Here we provide a brief implementation sketch. Basic setup for a platform. We assume a generic archival platform where each post is stored as an Active Store Object (ASO)  [26]. ASOs are simply key-value pairs with some (optional) code to run on values. Traditionally this ASO code is written in terms of handlers (e. g. , code to handle deletion). Each post ASO will have an unique post id as key, the user generated post content as value, identification of the owner (as authentication token) and some metadata (e. g. , the real state flag for a post).
We further assume that there is an internal trusted time server, which is used throughout the platform for synchronizing operations. The platform internally does not use any other timestamps. Any mention of timestamps in this section refers to this internal timestamp. Extending this set-up to traditional databases is simple and left to future work. We use an architecture similar to Comet  [26] , where the platform operator as well as platform users (including adversary) have some specific Application Programmer Interfaces (API) to access/create/delete the posts. However note that in our adversary model, the adversary can just query the posts and can not change them in any way. Thus, unlike Comet in Lethe post ASO objects are immutable from the point of view of an adversary.
We prepared total RNA for sequencing using the Illumina TruSeq methodology (TruSeq Stranded Total RNA LT Sample Prep Kit with Ribo-Zero Gold). The first step involves the removal of ribosomal RNA (rRNA) using biotinylated, targetspecific oligos, combined with Ribo-Zero rRNA removal beads. After individual samples were bar-coded they were pooled and sequencing was done on two 2 Â 75 bp Illumina NextSeq 2500 lanes. All raw data have been submitted to the NCBI's Sequence Read Archive (http://www. ncbi. nlm. nih. gov/sra) under accession number SRP075217.