Miracle-dz/UniFAR-FTUs huggingface dataset:
Quasi ML Estimation of the Panel AR(1) Model with Additional Regressors

In this paper we discuss limited information and full information random effects and fixed effects Quasi ML estimators for panel AR(1) models with additional regressors. We also consider related GMM estimators. The models allow for arbitrary initial conditions and heteroskedasticity and are extensions of the models considered in Kruiniger (2013. Quasi ML estimation of the panel AR(1) model with arbitrary initial conditions. Journal of Econometrics 173, 175-188). Among other things, we distinguish between the case where the regressors are strictly exogenous and the case where they are predetermined with respect to the idiosyncratic errors, including the case where they are weakly exogenous; we consider the possibility that the regressors are correlated with the individual effects; and we discuss estimation of models with time-varying individual effects. Finally, we stress the advantages of an LI approach over an FI approach such as robustness, computational ease and in some cases the higher precision of the resulting estimators.
Genetic differentiation following recent domestication events: A study of farmed Nile tilapia (Oreochromis niloticus) populations

Nile tilapia (Oreochromis niloticus) is among the most farmed finfish worldwide, distributed across different environmental conditions. Its wide distribution has mainly been facilitated by several breeding programs and widespread dissemination of genetically improved strains. In the first Nile tilapia study exploiting a whole‐genome pooled sequencing (Poolseq) approach, we identified the genetic structure and signatures of selection in diverse, farmed Nile tilapia populations, with a particular focus on the GIFT strain, developed in the 1980s, and currently managed by WorldFish (GIFTw). We also investigated important farmed strains from The Philippines and Africa. Using both SNP array data and Poolseq SNPs, we characterized the population structure of these samples. We observed the greatest separation between the Asian and African populations and greater admixture in the Asian populations than in the African ones. We also established that the SNP array data were able to successfully resolve relationships between these diverse Nile tilapia populations. The Poolseq data identified genomic regions with high levels of differentiation (FST) between GIFTw and the other populations. Gene ontology terms associated with mesoderm development were significantly enriched in the genes located in these regions. A region on chromosome Oni06 was genetically differentiated in pairwise comparisons between GIFTw and all other populations. This region contains genes associated with muscle‐related traits and overlaps with a previously published QTL for fillet yield, suggesting that these traits may have been direct targets for selection on GIFT. A nearby region was also identified using XP‐EHH to detect genomic differentiation using the SNP array data. Genomic regions with high or extended homozygosity within each population were also identified. This study provides putative genomic landmarks associated with the recent domestication process in several Nile tilapia populations, which could help to inform their genetic management and improvement.
Identifying and addressing methodological incongruence in phylogenomics: A review

The availability of phylogenetic data has greatly expanded in recent years. As a result, a new era in phylogenetic analysis is dawning—one in which the methods we use to analyse and assess our data are the bottleneck to producing valuable phylogenetic hypotheses, rather than the need to acquire more data. This makes the ability to accurately appraise and evaluate new methods of phylogenetic analysis and phylogenetic artefact identification more important than ever. Incongruence in phylogenetic reconstructions based on different datasets may be due to two major sources: biological and methodological. Biological sources comprise processes like horizontal gene transfer, hybridization and incomplete lineage sorting, while methodological ones contain falsely assigned data or violations of the assumptions of the underlying model. While the former provides interesting insights into the evolutionary history of the investigated groups, the latter should be avoided or minimized as best as possible. However, errors introduced by methodology must first be excluded or minimized to be able to conclude that biological sources are the cause. Fortunately, a variety of useful tools exist to help detect such misassignments and model violations and to apply ameliorating measurements. Still, the number of methods and their theoretical underpinning can be overwhelming and opaque. Here, we present a practical and comprehensive review of recent developments in techniques to detect artefacts arising from model violations and poorly assigned data. The advantages and disadvantages of the different methods to detect such misleading signals in phylogenetic reconstructions are also discussed. As there is no one‐size‐fits‐all solution, this review can serve as a guide in choosing the most appropriate detection methods depending on both the actual dataset and the computational power available to the researcher. Ultimately, this informed selection will have a positive impact on the broader field, allowing us to better understand the evolutionary history of the group of interest.
T-Rex: Optimizing Pattern Search on Time Series

Pattern search is an important class of queries for time series data. Time series patterns often match variable-length segments with a large search space, thereby posing a significant performance challenge. The existing pattern search systems, for example, SQL query engines supporting MATCH_RECOGNIZE, are ineffective in pruning the large search space of variable-length segments. In many cases, the issue is due to the use of a restrictive query language modeled on time series points and a computational model that limits search space pruning. We built T-ReX to address this problem using two main building blocks: first, a MATCH_RECOGNIZE language extension that exposes the notion of segment variable and adds new operators, lending itself to better optimization; second, an executor capable of pruning the search space of matches and minimizing total query time using an optimizer. We conducted experiments using 5 real-world datasets and 11 query templates, including those from existing works. T-ReX outperformed an optimized NFA-based pattern search executor by 6x in median query time and an optimized tree-based executor by 19X.
Efficient and Effective Algorithms for Generalized Densest Subgraph Discovery

The densest subgraph problem (DSP) is of great significance due to its wide applications in different domains. Meanwhile, diverse requirements in various applications lead to different density variants for DSP. Unfortunately, existing DSP algorithms cannot be easily extended to handle those variants efficiently and accurately. To fill this gap, we first unify different density metrics into a generalized density definition. We further propose a new model, c-core, to locate the general densest subgraph and show its advantage in accelerating the searching process. Extensive experiments show that our c-core-based optimization can provide up to three orders of magnitude speedup over baselines. Moreover, we study an important variant of DSP under a size constraint, namely the densest-at-least-k-subgraph (DalkS) problem. We propose an algorithm based on graph decomposition, and it is likely to give a solution that is at least 0.8 of the optimal density in our experiments, while the state-of-the-art method can only ensure a solution with density at least 0.5 of the optimal density. Our experiments show that our DalkS algorithm can achieve at least 0.99 of the optimal density for over one-third of all possible size constraints.
Selection Pushdown in Column Stores using Bit Manipulation Instructions

Modern analytical database systems predominantly rely on column-oriented storage, which offers superior compression efficiency due to the nature of the columnar layout. This compression, however, creates challenges in decoding speed during query processing. Previous research has explored predicate pushdown on encoded values to avoid decoding, but these techniques are restricted to specific encoding schemes and predicates, limiting their practical use. In this paper, we propose a generic predicate pushdown approach that supports arbitrary predicates by leveraging selection pushdown to reduce decoding costs. At the core of our approach is a fast select operator capable of directly extracting selected encoded values without decoding, by using Bit Manipulation Instructions, an instruction set extension to the X86 architecture. We empirically evaluate the proposed techniques in the context of Apache Parquet using both micro-benchmarks and the TPC-H benchmark, and show that our techniques improve the query performance of Parquet by up to one order of magnitude with representative scan queries. Further experimentation using Apache Spark demonstrates speed improvements of up to 5.5X even for end-to-end queries involving complex joins.
Generalizing Bulk-Synchronous Parallel Processing for Data Science: From Data to Threads and Agent-Based Simulations

We generalize the bulk-synchronous parallel (BSP) processing model to make it better support agent-based simulations. Such simulations frequently exhibit hierarchical structure in their communication patterns which can be exploited to improve performance. We allow for the creation of temporary artificial network partitions during which agents synchronize only locally within their group in a way that does not compromise the correctness of a simulation. We have built a distributed engine, CloudCity, which uses this idea to improve the locality of computation, communication, and synchronization in such simulations. We experimentally evaluate the performance of our system on a benchmark of simulation workloads and compare it against other popular BSP-like systems, obtaining insights into the impact of various system design choices and optimization on simulation engine performance.
HR-Index: An Effective Index Method for Historical Reachability Queries over Evolving Graphs

Reachability query is a fundamental problem and has been well studied on static graphs. However, in the real world, the graphs are not static but always evolving over time. In this paper, we study the problem of historical reachability query on evolving graphs. We propose a novel index, named HR-Index, which integrates complete and correct historical reachability information of the evolving graph. A historical reachability query on an evolving graph can be converted into a static reachability query on its HR-Index and thus query efficiency can be improved significantly. We also propose two optimization techniques to reduce the size of HR-Index effectively. We confirm the effectiveness and efficiency of our method through conducting extensive experiments on real-life datasets. Experimental results show both vertex and edge size of HR-Index are far smaller than that of the evolving graphs and our method has at least an order of magnitude improvement in time and space efficiency compared to the state-of-the-art method.
Theories and Principles Matter: Towards Visually Appealing and Effective Abstraction of Property Graph Queries

Existing visual abstraction of a property graph query by representing it as a labeled atomic graph (LAG) has great potential to democratize the usage of property graph databases as it enables user-friendly visual query formulation without demanding the need to learn a property graph query language e.g., Cypher. Unfortunately, existing LAG-based query interfaces do not embrace HCI principles and psychology theories to inform their design and as a result may have adverse impact on their usability and aesthetics. In this paper, we depart from the classical theory- and principles-oblivious LAG abstraction to present a novel theory-informed visual abstraction called labeled composite graph (LCG) to address this limitation. It realizes a novel and extensible visual shape definition language called VEDA to create and maintain an LCG systematically, guided by a variety of theories and principles from HCI, visualization and psychology. We build a novel LCG-based visual property graph query interface for Cypher called SIERRA and demonstrate through a user study its superiority to an industrial-strength LAG-based query interface for property graphs w.r. t. usability, aesthetics and efficient query formulation.
Designing a Technology-enhanced Play Environment for Young Children's Science Modeling Practice

In this paper, we built on the Learning through Embodied Activity Framework (LEAF) and designed the GEM-STEP mixed reality environment to support young children's science modeling practice through different play activities. The participants in this study are 16 first and second-grade students who participated in activities within the GEM-STEP environment to explore the mechanism of pollination. We applied interaction analysis to classroom videos to explore how the design of mediators within the MR environment supports students’ collective play and orients students toward science modeling practices. We found that the design of the MR environment provides a pivot for students to engage in science modeling practice, and the two roles that students played with the technology offered them different perspectives to explore and examine their model of pollination. We hope this study provides guidance on how to design for young children's science modeling practice with extended reality in a more playful way.
Big Code Search: A Bibliography

Code search is an essential task in software development. Developers often search the internet and other code databases for necessary source code snippets to ease the development efforts. Code search techniques also help learn programming as novice programmers or students can quickly retrieve (hopefully good) examples already used in actual software projects. Given the recurrence of the code search activity in software development, there is an increasing interest in the research community. To improve the code search experience, the research community suggests many code search tools and techniques. These tools and techniques leverage several different ideas and claim a better code search performance. However, it is still challenging to illustrate a comprehensive view of the field considering that existing studies generally explore narrow and limited subsets of used components. This study aims to devise a grounded approach to understanding the procedure for code search and build an operational taxonomy capturing the critical facets of code search techniques. Additionally, we investigate evaluation methods, benchmarks, and datasets used in the field of code search.
rkHit: Representative Query with Uncertain Preference

A top-k query retrieves the k tuples with highest scores according to a user preference, defined as a scoring function. It is difficult for a user to precisely specify the scoring function. Instead, obtaining the distribution on scoring functions, i.e., the preference distribution, has been extensively explored in many fields. Motivated by this, we introduce the uniform (r,k)-hit (UrkHit) problem. Given a preference distribution, UrkHit aims to select a representative set of r tuples to maximize the probability of containing a tuple attractive to the user. We say a tuple attracts a user, if it is a top-k tuple for the scoring function adopted by the user. Further, we generalize UrkHit and propose the (r,k)-hit (rkHit) problem with an additional penalty function to model the user satisfaction with the tuple ranked i-th. rkHit aims to maximize the expected user satisfaction with the representative set. In 2D space, we design an exact algorithm 2DH for rkHit, indicating rkHit is in P for d=2. We show that rkHit is NP-hard when d\ge3. In 3D space, assuming a uniform preference distribution, we propose a (1-1/e)-approximation algorithm 3DH based on space partitioning. In addition, we propose an approximate algorithm MDH suitable for any dimension and distribution, which creatively combines the ideas of sampling and clustering. It relaxes the approximation guarantee slightly. Comprehensive experiments demonstrate the efficiency and effectiveness of our algorithms.
Future-Time Temporal Path Queries

Most previous research considers processing queries on the current or previous states of a graph. In this paper, we propose processing future-time graph queries, i.e., predicting the output of a query on some future state of the graph. To process future-time queries, we present a generic approach that exploits a predictive model that provides oracles about the future state of the graph. We focus on future-time shortest path queries that given a temporal graph and two nodes return the shortest path between them at some future time. We present two algorithms each invoking a different type of oracle: (a) a link prediction oracle that given two nodes returns the probability of an edge between them, and (b) a connection prediction oracle that given a node u and a future time instance t returns the node υ that u will connect to at t. Finally, we present experimental results using off-the-shelf prediction models that provide such oracles.
Benchmarking Combinations of Learning and Testing Algorithms for Automata Learning

Automata learning enables model-based analysis of black-box systems by automatically constructing models from system observations, which are often collected via testing. The required testing budget to learn adequate models heavily depends on the applied learning and testing techniques. Test cases executed for learning (1) collect behavioural information and (2) falsify learned hypothesis automata. Falsification test-cases are commonly selected through conformance testing. Active learning algorithms additionally implement test-case selection strategies to gain information, whereas passive algorithms derive models solely from given data. In an active setting, such algorithms require external test-case selection, like repeated conformance testing to extend the available data. There exist various approaches to learning and conformance testing, where interdependencies among them affect performance. We investigate the performance of combinations of six learning algorithms, including a passive algorithm, and seven testing algorithms, by performing experiments using 153 benchmark models. We discuss insights regarding the performance of different configurations for various types of systems. Our findings may provide guidance for future users of automata learning. For example, counterexample processing during learning strongly impacts efficiency, which is further affected by testing approach and system type. Testing with the random Wp-method performs best overall, while mutation-based testing performs well on smaller models.
Apache IoTDB: A Time Series Database for IoT Applications

A typical industrial scenario encounters thousands of devices with millions of sensors, consistently generating billions of data points. It poses new requirements of time series data management, not well addressed in existing solutions, including (1) device-defined ever-evolving schema, (2) mostly periodical data collection, (3) strongly correlated series, (4) variously delayed data arrival, and (5) highly concurrent data ingestion. In this paper, we present a time series database management system, Apache IoTDB. It consists of (i) a time series native file format, TsFile, with specially designed data encoding, and (ii) an IoTDB engine for efficiently handling delayed data arrivals and processing queries. The system achieves a throughput of 10 million inserted values per second. Queries such as 1-day data selection of 0.1 million points and 3-year data aggregation over 10 million points can be processed in 100 ms. Comparisons with InfluxDB, TimescaleDB, KairosDB, Parquet and ORC over real world data loads demonstrate the superiority of IoTDB and TsFile.
Machine Vision Based Wireless Link Layer Anomaly Characterization

As the number of wireless end and edge devices increases, so does the volume of data to be monitored in view of predicting or detecting malfunctions. Furthermore, as the networks become more complex, the more context can be provided around a certain anomaly, fault or malfunction, the easier will be to establish mitigation actions in a fast and efficient manner. While detecting and classifying shapes of anomalous link behaviour from time series has already been investigated, the precise localization in time and characterization in duration and amplitude from actual data traces that would provide more context related to the anomaly is outstanding. In this paper, we study the performance of time-series to image transformation techniques combined with machine vision to precisely localize, describe and classify link layer anomalies in wireless networks. Our evaluation shows that the proposed approach is able to characterize an anomaly by precisely localizing its start and end, as well as its amplitude with mean absolute error of less than 1%. Furthermore, the classification scores of the method are comparable with state of the art classifiers, reaching overall F1 scores of 0.97. This is the first attempt at characterizing anomalous shapes that appear in networking time-series data with potential applications in real-time monitoring systems, predictive maintenance where more context may lead to better prioritization and faster mitigation.
ForestTI: A Scalable Inverted-Index-Oriented Timeseries Management System with Flexible Memory Efficiency

Timeseries management systems play an important role in IoT and performance monitoring. As the data volume scales up, absorbing data memory efficiently with high throughput becomes a growing requirement for timeseries management systems. However, the designs of the existing systems, especially the in-memory data structures, suffer from two issues. First, they suffer from the trade-off between memory efficiency and performance. Second, they are not scalable because of lock contention where they cannot benefit from parallel insertion and querying. In this paper, we propose ForestTI, a scalable inverted-index-oriented timeseries management system where the balance point between memory efficiency and performance can be flexibly adjusted under the increasing memory pressure. First, we present a two-level inverted index, which is scalable with optimistic lock coupling, and its internal structure can be gradually converted to more memory efficient representations. Second, we propose a two-level pointer swizzling mechanism to actively swap out the cold posting lists and in-memory timeseries objects as the number of timeseries increases. Finally, we further optimize the on-disk data structures (i.e. write-ahead logs and LSM-tree) to adapt to the high insertion throughput from the in-memory components. We prototype ForestTI with C++ from scratch, and compared to the storage engine of Prometheus, ForestTI achieves 1.79x higher insertion throughput, 52.1% lower query latency, and 56.9% lower memory occupation. We have released the open-source code of ForestTI for public access.
BtrBlocks: Efficient Columnar Compression for Data Lakes

Analytics is moving to the cloud and data is moving into data lakes. These reside on object storage services like S3 and enable seamless data sharing and system interoperability. To support this, many systems build on open storage formats like Apache Parquet. However, these formats are not optimized for remotely-accessed data lakes and today's high-throughput networks. Inefficient decompression makes scans CPU-bound and thus increases query time and cost. With this work we present BtrBlocks, an open columnar storage format designed for data lakes. BtrBlocks uses a set of lightweight encoding schemes, achieving fast and efficient decompression and high compression ratios.