dataesr/scientific-paragraphs-categorization huggingface dataset:
Online content has shifted from static and documentoriented to dynamic and discussion-oriented, leading users to spend an increasing amount of time navigating online discussions in order to participate in their social network. Recent work on emotional contagion in social networks has shown that information is not neutral and affects its receiver. In this work, we present an approach to detect the emotional impact of news, using a dataset extracted from the Facebook pages of a major news provider. The results of our approach significantly outperform our selected baselines.
Introduction With the rise of the social web, a majority of online content has shifted from being static and document-oriented to being highly dynamic and discussionoriented. With this shift, users have been spending more time navigating online discussions in order to stay informed with their social network. Recent work on emotion contagion in social networks  [2]  suggests that information is not neutral, and the way it is presented has an impact on the emotional state of its consumers. This demonstrates the importance of providing users with a way to control this content. In this work, we present a technique to predict the emotional impact of news on its consumers, using a dataset extracted from the Facebook pages of the New York Times, a major news network. We highlight the novelty of our work with respect to existing research on textual emotion detection, before formalizing our problem and explaining our methodology.
Dataset We used a dataset crawled from the comments on the Facebook page of the New York Times. As detailed in Table  1  emotions are not uniformly distributed in the dataset itself, but the distribution of emotions in the Facebook posts is strongly correlated with the distribution of emotions in the reactions (R = 0. 8814 on a Pearson test). We also note that the coverage of our emotion lexicon is close from the coverage of the Word2Vec embedding despite the word embedding being computed on a general purpose resource.
Stateful Window-Based Processing At its basis, an infinite data stream is a set of events or tuples that grows indefinitely in time  [16]. An infinite data stream is divided (based on event timestamp or other attributes) into finite slices called windows  [4]. The properties of a window are determined by a window assigner: it specifies how the elements of the stream are divided into windows. The main categories are: • global windows: each element is assigned to one single per-key global window; • tumbling windows: elements are assigned to fixed length, non-overlapping windows of a specified window size; • sliding windows: elements are assigned to overlapping windows of fixed length equal to the window size, the size of the overlap is defined by the window slide; and • session windows: windows are defined by features of the data themselves and window boundaries are adjusting to incoming data.
Serialized Objects State in Off-Heap Similar to the heap object state, Flink offers an option to configure an operator state to off-heap and it implements an embedded key-value store state interface (i. e. RocksDB). The main difference is that objects are serialized before they are persisted in the off-heap state and every time objects are accessed the cost of deserialization adds to the processing latency of corresponding operator's user defined function. Another difference consists in the fact that the RocksDB database is using local task manager data directories and as such the state size is limited by the amount of disk space available.
Methodology For every experiment we follow a similar cycle. We install Redis 3. 2. 4 [6]  and we configured a standalone Flink to use it as the state backend. We start the event generator as a Java socket program that listens to a configured port. First, it generates strings (events) until they fill the window state. Then, it generates strings for five iterations, each of one minute, keeping the same rate of new events. At the same time, the Flink application uses the socketTextStream method of the StreamExecutionEnvironment in order to create a new data stream that contains the strings received from the configured socket. We parse each event with a flatMap operation, assigning it a timestamp. After applying a userdefined function (as mentioned in the previous section), the timestamp is used to obtain the event latency, defined as the time seen at the end of the aggregation minus the initial time of the last event of a window.
We collect each aggregated value and write it as text with the operator writeAsText. We measure only the event latencies corresponding to the five iterations and we compute the aggregated upper bound of latencies experienced by 99% of events. We make sure to clear the OS buffer cache and temporary data or logs before a new execution starts. After each execution ends we clear the Redis cache and we collect logs that hold the event latency percentiles. For each experiment we fix the values for the following parameters (while keeping all other parameters at their default): (1) window size -is the size of the window for which we execute an user defined function aggregation, window slide is fixed to 1 (in order to put pressure on window evaluation); (2) window keys -gives the number of windows that are processed in parallel, equals the number of cores; (3) events rate -is the rate of new events that are streamed by the socket program each second according to the distribution mentioned in the previous section.
While for heap state we do not have differences between observed latencies, for shared state we observe an almost constant gap between each execution. This gap is related to the increased pressure on the key-value store, trying to access the same data concurrently. As we can see in Figure  5 , this experiment evaluates the same problem (events rate of 120 per second, window size of 4096 events) while increasing the parallelism from one to four (each core will process an equal number of elements) for the same four benchmarks like in the previous section. It is clear that with larger windows and increased throughput, relying on an external key-value store for memory deduplication of window-based processing may be unfeasible due to large overheads of serialization. Although there is significant performance overhead when using deduplication (as detailed in the previous section), such a technique may lead to large memory savings.
SBT-SK SBT-ALSO   2. Query wall-clock run times and maximum memory usage, for batches of different sizes. For the batch of 1,000 queries, we used the same 1,000 queries as in Figure  3. For the batch of 100 queries, we generated three replicate sets, where each set contains 100 randomly sampled transcripts without replacement from the 1,000 queries set. For the batch of 10 queries, we generated 10 replicate sets by partitioning one of the 100 query sets into 10 sets of 10 queries. For the batch of 1 query, we generated 50 replicate sets by sampling 50 random queries from Gencode set. The shown running times are the averages of these replicates. For θ, we used the default value in the SBT software (θ = 0. 9). The AllSome representation, without the clustering construction, also gives the benefit of allowing earlier query resolution, but the effect only becomes pronounced for queries that hit a lot of leaves.
Discussion In this paper we present an alternate implementation of the SBT that provides substantial improvements in query and construction time. We are especially effective for large batches of queries (6 times faster) or for large queries (155 times faster). Solomon and Kingsford make a convincing case that an efficient SBT implementation translates to an efficient and accurate solution to the broader problem of identifying RNA-seq samples that express a transcript of interest. They study the best parameter values of SBT (θ, k, b, p) to achieve accuracy and speed for the broader problem. The focus of this paper is on improving resource performance, and hence we do not revisit these questions, however, a more thorough exploration of the biological questions that the SBT can answer will be important moving forward.
We employ the method of lines to obtain a large set of nonlinear ODEs x are (n -1) × (n + 1) finite-difference differentiation matrices for the operators ∂ x and ∂ xx , respectively. In all PDE computations we use n = 4000 gridpoints. Time stepping is performed using MATLAB's ode23s with default parameters and explicitly formed Jacobian. For the stochastic time stepping we use a standard Euler-Maruyama time-stepping scheme  [27]  with s = 10 -3. Steady states are computed using MATLAB's fsolve with default parameter values and continued in parameter space using the routines developed in  [32]. Stability of a steady state A * = {A * (x i )} is studied by computing the first 40 eigenvalues of the problem M ψ = J(A * )ψ with the smallest absolute value, where M = diag(0, 1,. , 1, 0) ∈ R (n+1)×(n+1) , J =   e 1 D 2 x -diag(1 -αA 2 * )D x + diag(2αA * D x A * + εµ -3 √ ε A 2 * ) e n+1   ∈ R (n+1)×(n+1) and the {e i } are row vectors forming the canonical basis of R n+1.
Reproducibility has been shown to be limited in many scientific fields. This question is a fundamental tenet of the scientific activity, but the related issues of reusability of scientific data are poorly documented. Here, we present a case study of our attempt to reproduce a promising bioinformatics method [1] and illustrate the challenges to use a published method for which code and data were available. First, we tried to re-run the analysis with the code and data provided by the authors. Second, we reimplemented the method in Python to avoid dependency on a MATLAB licence and ease the execution of the code on HPCC (High-Performance Computing Cluster). Third, we assessed reusability of our reimplementation and the quality of our documentation. Then, we experimented with our own software and tested how easy it would be to start from our implementation to reproduce the results, hence attempting to estimate the robustness of the reproducibility.
If we used different data but with the same code, it is Replicability. lastly, using different data and different code is referred as Generalisability. Here we will primarily elaborate on Reproducibility and Robustness. Indeed, it takes great efforts and competence to overcome all the obstacles on the path to reproduction. The process is costly in resources, both in time and funding. In computational science, there are also many technical barriers ranging from unavailable data to hardware infrastructure  [9]. Even when authors provide data and code, the outcome can vary either marginally or fundamentally  [10]. Tackling irreproducibility in bioinformatics thus requires considerable effort beyond code and data availability. In most cases, there is a significant gap between apparent executable work (.
We are very grateful to the main authors who kindly provided online all the related data and code, and gave us invaluable input upon request. The authors of this study thus should not be blamed for the difficulty that we experienced in attempting to reproduce and to make more robust their study, as they did more to help reproduce their results than is generally done. Despite their help we experienced a number of difficulties that we document here, hoping that this report will help future researchers to improve the reproducibility of results and reusability of research products. The first step of our project was to execute the original method code with the given data: reproducibility. The programming code was written in MATLAB, an interpreted language originally developed for linear algebra computations which is easier and faster to write as well as more readable than compiled language such as C, making our reproducibility attempt easier.
To improve execution speed, the original authors used a library for MATLAB using executable compiled code MEX file  [13]  callable from MATLAB: MTIMESX  [14] , a library with compiled code allowing acceleration of large matrix multiplication. MEX files however are specific to the architecture and have to be recompiled for each Operating System (OS). The original MEX file was initially developed for Linux. Since our lab was using Mac OS X Sierra, the compilation of this MEX file into a mac64 binary required a new version of MTIMESX. It was also necessary to install and to configure properly OpenMP [15], a development library for parallel computing. After this, the original MATLAB code was successfully run in our environment. These issues are classic, but may not be overcome by researchers with little experience in compilation or installation issues.
Robustness: from MATLAB to Python, language and organization To fully master the method, adapt it to our data, and ease its reuse, we developed a complete open source toolkit of genomic stratification in Python  [2]. Python is also an interpreted programming language, but contrary to MATLAB is free of use and has a GPL-compatible license  [16]. This is particularly interesting for both robustness and generalizability. Recoding in another language in a different environment will lead to be some unavoidable problems such as variation in low level libraries (e. g. glibc): it is likely that the outcomes will vary even if the same algorithm is implemented. In addition, we rely on Python packages to perform visualization or linear algebra computations (e. g. Matplotlib, SciPy, NumPy), and results may depend on these packages versions.
Metadata and file formats Even if the original code could be run, we had to handle several file formats to check and understand the structure of the original data. For instance the data on patients with cancer data was provided by The Cancer Genome (TCGA)  [17]  and made available in a MATLAB. mat file format. Thanks to SciPy, Python can load all versions through v7. 2 MATLAB files. To read v7. 3. mat files, we however needed an HDF5 Python library. Moreover, the original authors had denoted download dates of patients' data of TCGA, thereby clarifying source of data. But in the absence of structural metadata, it was not always obvious how to interpret patients' dataset variables (e. g. patient ID, gene ID, phenotype). Fig  2  shows an analogy between robustness issues and road transport: driving in a different environment (e.
Codes and parameters Once the environment and file format issues were resolved, the code was finally executable with genetic data. Unfortunately, several attempts produced error messages. Alternatively, "unexpected" results were obtained: e. g. during the application of hierarchical clustering, we used the clustering tools of SciPy. Both SciPy and MATLAB (MathWorks) functions offer seven linkage methods, however, SciPy's default option (single method)  [18]  differs from MATLAB's default option (UPGMA or average method)  [19] , which was used in the original study. Another key example is the value of one of the most important parameters of the method, the graph regulator factor, which was not clarified in the original paper. We believed that this factor had a constant value of 1. 0 until we found in the code that during iterations, its value was changing and converged to a high optimal value (~1800).
Therefore, we obtained different results from the original NBS at the beginning (Fig  3 ). We observed heterogeneous subgroups instead of obtaining homogeneous clusters. No or little explanation on the parameter choices can explain variability in the results as we explored the possible parameters. Moreover, during our attempts to run the original code to understand the causes of the errors, we realized that some parts of the code were not run anymore (e. g. discarded work, remaining traces of debugging) which made the attempt to understand the implementation harder. Readers are proposed to reproduce those two confusion matrices (Fig  3 ) using different practical tools: GitHub, Docker and Jupyter/IPython notebook, which are further described below.
Jupyter/IPython During the re-cording process, we used an enhanced Python interpreter to debug: IPython, an interactive shell supporting both Python 2 and 3. Since the dataset is large and the execution takes a significant amount of time, we used IPython to re-run interactively some sub-sections of the script, which is one of the most helpful features. IPython can be integrated in the web interface Jupyter Notebook, offering an advanced structure for mixing code and documentation. For instance, verifying intermediate results by plotting helped us to better understand the original code. While the Jupyter/IPython notebook was therefore initially convenient, it does not scale well and is not well adapted to versioning. However, ability of mixing code with document text is very useful for tutorials: a user of the code can read documentation (docstring), text explanations, and see how to run the code, explore parameters and visualize results in the browser.
Reproducibility of Robustness: from Python to Python Besides Jupyter/IPython notebook, we used versioning tools like the git code version control system (VCS) to document the development of our Python code. Git is arguably one of the most powerful VCS, allowing easy development of branches and helping us to work together as a distributed team (Paris, Berkeley) on the same project. This project, StratiPy, is hosted on GitHub, a web-based Git repository hosting service  [2]. While the original code was not available on GitHub, the main authors shared their code on a website. This should be sufficient for our purpose, but makes it less easy to collaborate on code. While working on our GitHub repository, several researchers from all over the world contacted us about our robustness experiment. Not only GitHub supports a better organization of projects, it also facilitates the collaboration of open-source software projects, thanks to several social network functions  [21].
We tried to comply with open source coding standards and to learn how to efficiently use Git and GitHub. Both required considerable efforts on the short-term but brought clear benefits on the long-term, especially regarding collaboration and debugging. We then attempted to re-run and reproduce the results we obtained on another platform. While the Python code was developed under Mac OS X Sierra (10. 12) we used an Ubuntu 16. 04. 1 (Xenial) computer to test the Python implementation. Some additional issues emerged. First, our initial documentation was not complete enough to know which packages were required and how to launch the code. Second, the code was very slow to the extent that it was impractical to run it on a laptop because the Numpy package had not been compiled with BLAS (Basic Linear Algebra Subprograms), low-level routines performing basic vector and matrix operations.
Environment Container technologies such as Docker  [24] , Vagrant  [25] , or Singularity  [26, 27]  (easily works in cluster environments) are becoming a standard solution to installation issues. These rely however on competencies that we think few biologists possess today. Also, while the container will encapsulate everything needed for the software execution, it could be hard to develop in a container. For instance, running Jupyter/IPython notebooks in Docker's container requires certain knowledge of computer science (e. g. advanced port forwarding), which can become a discouraging task. Therefore, we decided to propose two options in our example implementation of reproducibility: 1) a step-by-step process to follow in a Jupyter/IPython notebook; or 2) a Docker container ready to be built and run.
Write readable code Anyone who has spent time to understand someone else's code would advise some simple basic rules to help make the code readable and understandable. First, the structure of the program should be clear and easily accessible. Second, good concise code documentation and naming convention will help readability. Third, the code should not contain left-overs of previously tested solutions. When a solution takes a long time to compute, an option to store it locally can be proposed. Using standard coding and documentation conventions (e. g. PEP 8 and PEP 257 in Python  [30, 31] ) with detailed comments and references of papers makes the code more accessible. When an algorithm from another paper is used, any modification should be explained and discussed in the paper as well as in the code.
Test the code To check if the code is yielding a correct answer, software developers associate test suites (unit tests or integration tests) with their software. While we developed only a few tests in this project, we realize that this has a number of advantages, such as checking if the software installation seems correct, check if updates in the operating system impact the results, etc. This does not in general validate the method, but at least provides a basic check. In our case, we propose to check for the integrity of the data and for the results of some key processing.
The training in coding is still too limited for biologists. Often, it is self-training, from searching answers on Stack Overflow or equivalent. Despite efforts by organizations such as software  [33]  or data carpentry  [34]  and the growing demand for 'data scientists' in life science, university training on coding practices is still not enough generalized. The difficulty to access and understand code may lead to applying code blindly without checking the validity of the results: often, scientists may prefer to believe that the results are correct because of the time that would be needed to check the validity of the results. Mastering a package such that results are truly understood can take a long time, as it was the case in our experiment. Academia could instruct young scientists best practices for reproducibility.
Standard consensus dataset and workflow system We propose here that bioinformatics methods publications are systematically accompanied with a test dataset, code source and some basic tests. As the method is tested on new datasets, the number of tests of the method would increase in number and cover a wider range of applications. We give a first example with our NBS re-implementation. We develop below how this could generalize and what would be the benefit for the scientific community. In a sense, we propose to use the software development test framework idea but apply it to the scientific context. A schematic overview of workflow system is shown in Fig 4. The core of this system would be a standard consensus dataset used to validate methods. For instance in the field of machine learning, standard image databases are widely used for training and testing (e.
g. MNIST for handwritten digits  [35] ). In the case of our proposal, data could be classified in general categories such as binary, text, image (A, B, C in Fig  4 b ), and with sub-categories to introduce criteria such as size, quantitative/qualitative, discrete/continuous using a tagging system (e. g. A-2, B-1, C-5 in Fig  4 b ). Dataset could be issued from simulations or from acquisition, and would validate a method on a particular component. This workflow system will help scientists that cannot release their data because of privacy issues (Fig 4 a. 1) (although these can often be overcome) but also give access to data and tests to a wide community. Scientists can use only standard data at the beginning of the project. And if there is no appropriate data, they have to suggest a new standard data.