dataesr/scientific-paragraphs-categorization huggingface dataset:
A smaller data_hash_size would greatly improve performance, but would increase the probability of collisions, or in other words that two variants are represented by the same hash. (3) From h(i) bits he extracts x bits that will map to an index. The hash, h(i) bits , is then stored in another file (encoded VCF) in this new position. A larger x means that the server's database will have more entries (2 x ) since the number of possible mapped indexes is bigger. A smaller x reduces the number of possible indexes and, as a consequence, increases the number of collisions in the new encoded VCF file. If a collision occurs, the variant is concatenated to pre-existent ones. (4) All rows in the encoded VCF file, even if empty, are padded so that they have the same length. (5) Finally, each element in a row is symmetrically encrypted.
Querying phase The querying phase (Fig. 4 ) is mostly the same as standard PIR, repeated for each variant and/or VCF file queried: (1) The client specifies the query and calculates the hash for the variant j being searched using the same key, U, as before, h(j) bits. He then maps it to its respective index, pos, using the first x bits of that hash. (2) The client issues a PIR query for that position. In other words, the client sends an array, arr, with the same length as the queried VCF file in the database, composed of homomorphic encrypted 0s and a 1 on the position of the desired element → ∀l ∈ arr : HE R (1) if l = pos HE R (0) otherwise Note that the encryption scheme is probabilistic (there are many encryptions of 0 and of a single 1) and that the encryptions of 0 and 1 are indistinguishable for an attacker.
Most of the time, these additional data do not raise any security or privacy concerns as they belong to the client himself. However, if we consider an honest-but-curious querier (e. g. , an outside entity, such as a medical researcher, who is authorised to query the database), retrieving more information than intended can pose privacy problems. Therefore, we propose an extension to our querying phase to be performed after the PIR reply-generation denoted subtraction step, depicted in Fig. 5:  (3a) In addition to the PIR query, the client sends a "subtract query", sub, which contains the symmetrically encrypted hash of the variant being searched, E S (h(j) bits ), replicated multiple times so that it has the same length as a row in the queried file. (3b) The server then subtracts its PIR reply with sub.
Implementation In our solution, we make use of a very recently improved private information retrieval implementation, XPIR  [17] , Fig. 5  Subtraction step. The server subtracts the encrypted hash of the variant with its PIR reply and then randomises the result, thus hiding all other variants except for the one being queried combined with AES_CTR256 for symmetric encryption and HMAC_SHA256 for hashing. The XPIR library has been modified to include an optimized Fan and Vercauteren (FV)  [18]  homomorphic encryption scheme. There are a number of reasons for choosing these particular techniques. First, for the protection of data at rest, we opted for AES_CTR256 because it is considered safe for today's systems, but it also dramatically reduces the storage cost at the server side. In fact, the size of each ciphertext is the same as the corresponding plaintext value and the data owner can simply store locally the 64-bit nonce needed to generate the initialization vectors (IVs).
Parametrization For the keys of HMAC and AES, we chose sizes that are considered to be standard and secure. The remaining parameters had to be empirically fine-tuned. Table  1  contains the different parameters of our system. We first analyze the data_hash_size parameter. For this particular variable, more bits will reduce the chance of having false positives (one variant with the exact same hash as another one), but will also increase the overall database size. For example, storing 48 bits of each variants' hash in a five million VCF file would lead to a false positive probability of roughly 5, 000, 000/2 48 ≈ 1 2 25. We consider this to be acceptable, as the error rate in DNA sequencing is well above this probability  [22]. Nevertheless, we could increase the stored hashes to 96 bits so as to have a cryptographically low probability of false positives, thus doubling the database size, as well as the subtract query size, and multiplying the response time by slightly less than two (as the PIR query size is unchanged).
Varying the number of bits for mapping x affects multiple dimensions of our solution. The entries of each encoded VCF are indexed by the x first bits of the hash of each variant. Hence, each entry is a list of encrypted variants that map to the same index i. e. , the first bits of their hash are the same. Choosing a small x significantly increases the average number of collisions and reduces the number of entries, num_entries. Subsequently, the number of collisions defines the amount of padding needed, or in other words the row_size. For example, if the maximum number of collisions in one single index, among the different VCF files, is 100, then we can set our row_size to the same number and, with that, homogenize the size of each VCF file in the database. On the contrary, a higher x means a lengthier encoded VCF file, as more bits are extracted during our mapping phase.
Both aggregation and dimensionality are associated with the PIR scheme. The first parameter enables the packing of data, that is, enables multiple rows to be concatenated into a single row. Thus, during the PIR protocol, we can reduce the number of rows of a file in the database, as well as the size of PIR query, in exchange for a bigger PIR reply. The second parameter, dimensionality, enables the recursive execution of multiple PIR queries simultaneously (Fig. 6 ), thus reducing query transmission time. However, the reply size grows exponentially in the number of dimensions, hence we must keep the dimension small (<4). Both dimensionality and aggregation can be chosen in a way that ensures maximum performance.
Generic performance evaluation Table  2  lists five different settings, and Table  3  showcases the results. All different setups will run a single variant search on a specific VCF file with five million variants. We generated this test file by using the two datasets provided during the iDASH genome privacy challenge. For more information on each of the parameters, please refer to subsection -Parametrization. We evaluate four setups, of which Setup default uses the default settings for our strategy. We fine-tuned these parameters to offer the best possible performance. For storage, we decided to represent each variant with 48 bits of its hash, ensuring a small file size and a negligible probability of error. Furthermore, we also decided in favor of a compact database that enables us to reduce storage complexity and improve PIR query-generation/querysending time.
We ended with 8192 entries per file (x = 13), as further aggregating the database would make the PIR reply-generation/reply-sending too cumbersome. For the encryption mode and complementary parameters, we chose those provided by the XPIR optimizer -FV:80:1024:62:14 with aggregation 3 and dimension 2. To  The table is divided into three sections: initialization phase, querying phase and overall querying performance. The results are for one single variant search on a specific VCF file with five million variants. The data in boldface represents the round trip time, which provides an overall measure of the efficiency of our solution in the different setups obtain these values, the XPIR optimizer empirically tests, for a specific file, different combinations of parameters and chooses those that provide the better overall PIR performance and guarantee at least 80-bits security.
We opted to include a pipeline execution and pre-importation. Consequently, both query and reply are sent as they are created, and some data are stored in RAM to enable a much faster computation of all PIR operations. It is worth mentioning that the pre-importation is not viable if the number of patients is too high, roughly 100 for this machine, as there is not enough space in the RAM. Finally, to provide data confidentiality and privacy, we selected a 256-bit key for AES encryption, a secure standard, and a row_size of 716. This is sufficient for hiding the average maximum amount of collisions on any given VCF file up to five million variants, without having too much storage and time overhead and still keeping the size of each entry at the minimum possible. The three remaining setups enable us to compare the default strategy (compact database) with an alternative one (sparse database), to evaluate the performance of our subtraction step and to assess the security scalability of our solution.
For Setup sparse, we decided on a lengthier database, an alternative solution, with fewer collisions and a higher recursion to mitigate the effect of having more entries. Setup no_info removes the extra information retrieved for each query by adding our subtraction step. Finally, Setup security is similar to the previous one but, this time, the ciphertext size chosen is larger to provide much stronger security (>128 bits). We evaluate each setup based on three performance parameters (Table  3 ) in decreasing order of importance: performance, storage complexity and communication cost. The first metric is assessed by looking at the round-trip time (RTT), the PIR query and reply generation, as well as the reply extraction. Both data preparation and importation have a low impact in the overall complexity, as they are executed only once.
The iDash Challenge In the iDash Challenge  [13]  there were a number of requirements to consider. Accepted solutions had to (1) hide the data, query and access patterns from the cloud server, (2) employ homomorphic encryption, and (3) retrieve/reveal fewer than 20 variants to the data owner during each single variant search. However, for this competition it was not mandatory to hide the number of variants in each VCF file. Therefore, we decided to slightly change the padding scheme and significantly reduce query runtime. In this case, we add only dummy data to homogenize all rows inside a specific VCF. This padding is weaker than our previous one because the server can still distinguish different files by looking at their sizes. Therefore, the cloud knows if a VCF has more or less variants than another one in the database and can consequently infer an approximation of the total number of variants.
Discussion In this section, we analyze the results reported in Tables 3 and 5 and discuss the pros and cons of our solution. The data in boldface represents the round trip time, which provides an overall measure of the efficiency of our solution in the different setups We also unveil some alternative strategies for potentially addressing some of the limitations that our system can incur. From the results of the previous section, we are confident that our solution enables the execution of a query in a short amount of time and is scalable no matter what the number of elements in the file are. Furthermore, under the right parameters, e. g, Setup default, and under the same encoding strategy, our solution is faster than downloading the entire database (approximately 3. 5 seconds with a five million VCF file) and provides stronger security and privacy.
Limitations There are a few limitations that we can identify in our strategy. The first is the error rate associated with our hashing scheme. In the next section, we propose a way to mitigate this problem, and we still provide a fast querying time. Second, if the number of VCF files in the database is relatively large, then we can no longer store pre-imported data into RAM. This can be solved by storing pre-imported data into a disk, but will result in a slow down. As reply generation is more rapid than sending the reply, the overhead due to the disk reading will not affect RTT, except for very slow disks. Finally, our default strategy still suffers from some scalability issues, especially if multiple variants or files are queried -O(n) complexity, with n being the number of variants or files queried.
Other possible strategies We now list a couple of other alternative strategies and mechanisms for addressing some of the previously listed shortcomings of our protocol. These ideas have both pros and cons, some of which we detail in the following paragraphs. The first idea is to store the encoding of a variant instead of part of its hash. This would remove the risk of having false positives and enable the possible execution of other operations on the data. However, in addition to the normal padding, we would also need to hide each variant's length by means of another padding. With this particular encoding, variant size varies from a SNP (37 bits, see example above) to an insertion/deletion (variable, e. g. , 81 bits). Thus, padding each element to the maximum encoding length in the database would increase the database size by a factor of 2 with the proposed example.
Larger databases significantly degrade performance (see Table  6 ) because they increase importation time and PIR reply-generation/reply-sending. The second idea, which we briefly discussed in the previous subsection, is to have a much lengthier database with no collisions, thus removing the need to perform the subtraction step. This would also enable a much faster AND operation by means of a point-value polynomial representation. This way, we could batch several queried elements into one single PIR query; instead of one query per variant, and then search, in the reply, for the value corresponding to the addition of the variants. This approach, however, would require a huge database Table  6  Quantitative results relative to our alternative encoding strategy. We use the same settings as Setup default but with data_hash_size = 81 The data in boldface represents the round trip time, which provides an overall measure of the efficiency of our solution in the different setups and much dummy data, thus severely degrading the performance of the PIR protocol.
One naive way to avoid collisions could be to separate each variant into a collection of SNPs to be encoded (37 bits) and directly place them in a database with 2 37 entries. Cuckoo hashing  [26]  offers a smarter alternative, by replicating the database and using two hashes/positions for each single element. However, for our algorithm to work, we require one of the two positions to remain empty, something that cannot be efficiently done with the standard greedy insertion algorithm. Finally, the third alternative would be to not use PIR and instead employ a fully-homomorphic encryption (FHE) scheme. For this protocol, the database would contain encrypted multi-nucleotide variants unordered. To perform a query, we would use FHE to test the equality of each database element to, for example, a 48-bit homomorphically encrypted hash as shown in Fig.
7. We would do this by sending 48 queries that correspond to the 48 bits of the encrypted variant's hash we want to verify. Then, because the noise increases significantly with multiplicative depth, we would apply a binary tree on each entry of the database. Notation: a i is the i th bit we send; b i is the i th bit of an element in the database; t i represents the equality test of the bits i. We can do this by working modulo 2 with: 1 + (a i + b i ). We list some of the pros and cons for the FHE-only alternative strategy.
Conclusion We have described a new efficient strategy that uses private information retrieval to search genomic variants on a cloud database. This scheme makes use of a new enhanced PIR protocol that we adapt to fit our purpose. All the security and privacy requirements were met by introducing certain modifications in the PIR protocol, such as the need to homogenize each client's VCF file and the symmetric encryption of the data to guarantee confidentiality and privacy. We have also listed some other alternative mechanisms that can be useful, depending on the setting, e. g. , sparse database and cuckoo hashing to reduce AND complexity, or encoding to enable other operations. Finally, results show that, although not as effective as a simple search through an unencrypted database, this strategy exhibits a good performance and could be realistically deployed, for example, in clinics or hospitals.
Stop Scatter search is implemented in C and requires two third-party libraries: SUNDIALS (http:// computation. llnl. gov/projects/sundials) and GSL (https://www. gnu. org/software/gsl) [44]. Source code is available on GitHub at https://github. com/amirmasoudabdol/flyOpt. We compiled the code with GCC 4. 7. 2 (https://gcc. gnu. org)) using the -O2 optimization flag. We have run scatter search simulations on the Lisa system of SURFsara (https://userinfo. surfsara. nl/systems/lisa) using search parameters as listed in Table  1. We saturated single nodes (Intel Xeon CPUs at 2. 6 GHz, 16 cores) with the maximum of 16 runs to suppress hardware parallelism. We note that these tests were run on a HPC facility simply for convenience. The software does not use any parallel computation features and the hardware of current-day workstations is often comparable to that of the Lisa system.
For interpretation, we may also express the cost as a Root Mean Square (RMS), which ignores the weights v and is a normalization with respect to the total number of data points (N RN A = 1804 and N protein = 1976). After optimization, the resulting solutions are judged on RMS, numerical stability, and visual inspection of model output, as described in detail elsewhere  [13, 14]. Analysis and plotting is performed using the previously developed SuperFIT package (https://app. assembla. com/spaces/superfit/ subversion/source)  [16].
Acknowledgments: We thank  Johannes Jaeger  for critical feedback and scientific advice. We thankfully acknowledge the computer resources, technical expertise and assistance provided by the  Barcelona Supercomputing Center , which is part of the  Red Española de Supercomputación. We thank SURFsara (www. surfsara. nl)for the support in using the  Lisa Compute Cluster. The Centre for Genomic Regulation (CRG)  acknowledges support from the  Spanish Ministry of Economy and Competitiveness, 'Centro de Excelencia Severo Ochoa   2013-2017', SEV-2012-0208. AC kindly acknowledges  Fondation Bettencourt Schueller.
Context The amount of text data on the web, or stored by companies is growing continuously. In order to exploit this wealth, it is essential to extract knowledge from such data. The discipline dealing with this type of data is called "Text Mining" includes several issues such as search indexing of documents, summary generation, creation of bots, etc. The work done during our project is part of the enrichment of the Palo IT textual and data analysis research. It aims to create a semiintelligent bot. For this project an internal R & D was launched. This project "PetiText" (translated SmallText in English; petit=small) is based on the analysis and reasoning on short sentences to detect new facts and answer questions. It involves an analysis of data from text corpora which allows to: • extract targeted information, sorted and added value for companies using algorithms • search for similarities and identifying causal relationships between different facts.
Issues Faced with the growing demand of Palo IT customers to extract knowledge from their textual data, the PetiText R & D project was launched. Indeed, these customers possess documents and tools for collecting reviews and customer complaints or employees. Hence the need to design and implement a tool for analysing this type of data. Text data poorly used by most companies, represent a wealth of information. Their analysis is a means of decision support and a strategic asset for companies. Study of Text Mining existing products shows a major flaw for processing text data written in French. This defect consists of the almost total absence of "open source" libraries incorporating the semantics of the French language. Indeed, unlike the English, we found that most libraries and tools used globally (as Clips  [1] , NLTK  [2] , etc.
Technological choices and human resources The "petiText" is a project that is part of Data Science Palo IT activities leaded by three PhDs: a data science expert as supervisor Mr. Patrick LAFFITTE, then Mrs. Raja HADDAD and Mr. Yassin CHABEB. Thanks to the wealth of existing libraries in python dedicated to machine learning, the choice of that language was obvious. Regarding data storage, we used ZODB (Zope Object DataBase) which is a hierarchical and object-oriented database. It can store the data as python objects. We used Gensim  [3]  and Scikit-learn  [4]  are two python libraries that implement various machine learning methods and facilitating the statistical treatment of data. These methods of learning and statistical data computations require considerable material resources: due to the volume of data to process and especially the time computations, therefore, two remote OVH machines were rented.
CoreNLP The construction of logic models from the input sentences needs first level of understanding from a syntactic and grammatical analysis. For this we used a natural language processing tool (NLP) entitled CoreNLP  [5]  of Stanford their basic form from plural or conjugation or grammatical declinations. It also explains the overall structure of the sentence analysing subject, verb, etc. It also acts as a parser adapting to the turn of phrases. Figure  1  shows an example of analysis of a sentence using CoreNLP. When edges represent the overall structure of the sentence, then the blue tags represent lemmas of different words, and those in red, give the grammatical class of a wor.
• A node (name or type). • One or more predicates (optional). These definitions and synonyms have served as a learning base to allow the bot to learn the meaning of words and phrases in different contexts The construction of logic models from the input sentences needs first level of understanding from a syntactic and grammatical analysis. For this we used a natural language processing tool (NLP) entitled CoreNLP  [5]  of Stanford University. CoreNLP fetches lemmas of words, it identifies their basic form from plural or conjugation or grammatical declinations. It also explains the overall structure of the sentence analysing subject, verb, etc. It also acts as a parser adapting to he turn of phrases. Figure  1  shows an example of analysis of a sentence using CoreNLP. When edges represent the overall structure of the sentence, then the blue tags represent lemmas of different words, and those in red, give the grammatical class of a word.
One or more predicates (optional). definitions and synonyms have served as a learning base to allow the bot to learn the meaning of words and phrases in different contexts. The construction of logic models from the input sentences needs first level of understanding from a syntactic and grammatical analysis. For this we used a natural language processing tool (NLP) University. CoreNLP fetches lemmas of words, it identifies their basic form from plural or conjugation or grammatical declinations. It also explains the overall structure of the sentence analysing subject, verb, etc. It also acts as a parser adapting to he turn of phrases. Figure  1  shows an example of analysis of a sentence using CoreNLP. When edges represent the overall structure of the sentence, then the blue tags represent lemmas of Web Scraping is a set of techniques to extract the contents of a source (a website or other).
Word embeddings To build a learning model based on the word embeddings every sentence is converted into vector of real values. The application of a model based on the succession of several layers neural networks to detect semantics, contexts and relationships between them, and the classification of new texts through an unsupervised learning. To apply the word embeddings on a textual corpus we used different python libraries like: Gensim (Word embeddings) and Scikit-learn. They offer all necessary methods (Doc2Vec, word2Vec. ).