OpenRaiser/Intern-Atlas huggingface dataset:
On the Limitation and Experience Replay for GNNs in Continual Learning
Venue: arXiv cs.LG 2024
Year: 2024

Continual learning seeks to empower models to progressively acquire information from a sequence of tasks. This approach is crucial for many real-world systems, which are dynamic and evolve over time. Recent research has witnessed a surge in the exploration of Graph Neural Networks (GNN) in Node-wise Graph Continual Learning (NGCL), a practical yet challenging paradigm involving the continual training of a GNN on node-related tasks. Despite recent advancements in continual learning strategies for GNNs in NGCL, a thorough theoretical understanding, especially regarding its learnability, is lacking. Learnability concerns the existence of a learning algorithm that can produce a good candidate model from the hypothesis/weight space, which is crucial for model selection in NGCL development. This paper introduces the first theoretical exploration of the learnability of GNN in NGCL, revealing that learnability is heavily influenced by structural shifts due to the interconnected nature of graph data. Specifically, GNNs may not be viable for NGCL under significant structural changes, emphasizing the need to manage structural shifts. To mitigate the impact of structural shifts, we propose a novel experience replay method termed Structure-Evolution-Aware Experience Replay (SEA-ER). SEA-ER features an innovative experience selection strategy that capitalizes on the topological awareness of GNNs, alongside a unique replay strategy that employs structural alignment to effectively counter catastrophic forgetting and diminish the impact of structural shifts on GNNs in NGCL. Our extensive experiments validate our theoretical insights and the effectiveness of SEA-ER.
An Achievable and Analytic Solution to Information Bottleneck for Gaussian Mixtures
Venue: arXiv cs.IT 2024
Year: 2024

In this paper, we study a remote source coding scenario in which binary phase shift keying (BPSK) modulation sources are corrupted by additive white Gaussian noise (AWGN). An intermediate node, such as a relay, receives these observations and performs additional compression to balance complexity and relevance. This problem can be further formulated as an information bottleneck (IB) problem with Bernoulli sources and Gaussian mixture observations. However, no closed-form solution exists for this IB problem. To address this challenge, we propose a unified achievable scheme that employs three different compression/quantization strategies for intermediate node processing by using two-level quantization, multi-level deterministic quantization, and soft quantization with the hyperbolic tangent ($\tanh$) function, respectively. In addition, we extend our analysis to the vector mixture Gaussian observation problem and explore its application in machine learning for binary classification with information leakage. Numerical evaluations show that the proposed scheme has a near-optimal performance over various signal-to-noise ratios (SNRs), compared to the Blahut-Arimoto (BA) algorithm, and has better performance than some existing numerical methods such as the information dropout approach. Furthermore, experiments conducted on the realistic MNIST dataset also validate the superior classification accuracy of our method compared to the information dropout approach.
Joint Metrics for EMF Exposure and Coverage in Real-World Homogeneous and Inhomogeneous Cellular Networks
Venue: arXiv cs.NI 2024
Year: 2024

This paper evaluates the downlink performance of cellular networks in terms of coverage and electromagnetic field exposure (EMFE), in the framework of stochastic geometry. The model is constructed based on datasets for sub-6~GHz macro cellular networks but it is general enough to be applicable to millimeter-wave networks as well. On the one hand, performance metrics are calculated for $β$-Ginibre point processes which are shown to faithfully model a large number of motion-invariant networks. On the other hand, performance metrics are derived for inhomogeneous Poisson point processes with a radial intensity measure, which are shown to be a good approximation for motion-variant networks. For both cases, joint and marginal distributions of the EMFE and the coverage, and the first moments of the EMFE are provided and validated by Monte Carlo simulations using realistic sets of parameters from two sub-6~GHz macro urban cellular networks, i.e., 5G~NR~2100 (Paris, France) and LTE~1800 (Brussels, Belgium) datasets. In addition, this paper includes the analysis of the impact of the network parameters and discusses the achievable trade-off between coverage and EMFE.
Quality Engineering for Agile and DevOps on the Cloud and Edge
Venue: arXiv cs.SE 2024
Year: 2024

Today's software projects include enhancements, fixes, and patches need to be delivered almost on a daily basis to clients. Weekly and daily releases are pretty much the norm and sit alongside larger feature upgrades and quarterly releases. Software delivery has to be more agile now than ever before. Companies that were, in the past, experimenting with agile based delivery models, are now looking to scale it to enterprise grade. This shifts the need from the ability to build and execute tests rapidly, to using different means, technologies and procedures to provide rapid and insightful validation sequences and tests to establish quality withing the manufacturing cycle. This book addresses the need of effectively embedding quality engineering throughout the agile development cycle thus addressing the need for enterprise scale high quality agile development.
Concept Algebra for (Score-Based) Text-Controlled Generative Models
Venue: arXiv cs.CL 2024
Year: 2024

This paper concerns the structure of learned representations in text-guided generative models, focusing on score-based models. A key property of such models is that they can compose disparate concepts in a `disentangled' manner. This suggests these models have internal representations that encode concepts in a `disentangled' manner. Here, we focus on the idea that concepts are encoded as subspaces of some representation space. We formalize what this means, show there's a natural choice for the representation, and develop a simple method for identifying the part of the representation corresponding to a given concept. In particular, this allows us to manipulate the concepts expressed by the model through algebraic manipulation of the representation. We demonstrate the idea with examples using Stable Diffusion. Code in https://github.com/zihao12/concept-algebra-code
Persuading a Behavioral Agent: Approximately Best Responding and Learning
Venue: arXiv cs.GT 2024
Year: 2024

The classic Bayesian persuasion model assumes a Bayesian and best-responding receiver. We study a relaxation of the Bayesian persuasion model where the receiver can approximately best respond to the sender's signaling scheme. We show that, under natural assumptions, (1) the sender can find a signaling scheme that guarantees itself an expected utility almost as good as its optimal utility in the classic model, no matter what approximately best-responding strategy the receiver uses; (2) on the other hand, there is no signaling scheme that gives the sender much more utility than its optimal utility in the classic model, even if the receiver uses the approximately best-responding strategy that is best for the sender. Together, (1) and (2) imply that the approximately best-responding behavior of the receiver does not affect the sender's maximal achievable utility a lot in the Bayesian persuasion problem. The proofs of both results rely on the idea of robustification of a Bayesian persuasion scheme: given a pair of the sender's signaling scheme and the receiver's strategy, we can construct another signaling scheme such that the receiver prefers to use that strategy in the new scheme more than in the original scheme, and the two schemes give the sender similar utilities. As an application of our main result (1), we show that, in a repeated Bayesian persuasion model where the receiver learns to respond to the sender by some algorithms, the sender can do almost as well as in the classic model. Interestingly, unlike (2), with a learning receiver the sender can sometimes do much better than in the classic model.
A Large-Scale Analysis of Persian Tweets Regarding Covid-19 Vaccination
Venue: arXiv cs.CL 2024
Year: 2024

The Covid-19 pandemic had an enormous effect on our lives, especially on people's interactions. By introducing Covid-19 vaccines, both positive and negative opinions were raised over the subject of taking vaccines or not. In this paper, using data gathered from Twitter, including tweets and user profiles, we offer a comprehensive analysis of public opinion in Iran about the Coronavirus vaccines. For this purpose, we applied a search query technique combined with a topic modeling approach to extract vaccine-related tweets. We utilized transformer-based models to classify the content of the tweets and extract themes revolving around vaccination. We also conducted an emotion analysis to evaluate the public happiness and anger around this topic. Our results demonstrate that Covid-19 vaccination has attracted considerable attention from different angles, such as governmental issues, safety or hesitancy, and side effects. Moreover, Coronavirus-relevant phenomena like public vaccination and the rate of infection deeply impacted public emotional status and users' interactions.
Cooperative Open-ended Learning Framework for Zero-shot Coordination
Venue: arXiv cs.AI 2024
Year: 2024

Zero-shot coordination in cooperative artificial intelligence (AI) remains a significant challenge, which means effectively coordinating with a wide range of unseen partners. Previous algorithms have attempted to address this challenge by optimizing fixed objectives within a population to improve strategy or behaviour diversity. However, these approaches can result in a loss of learning and an inability to cooperate with certain strategies within the population, known as cooperative incompatibility. To address this issue, we propose the Cooperative Open-ended LEarning (COLE) framework, which constructs open-ended objectives in cooperative games with two players from the perspective of graph theory to assess and identify the cooperative ability of each strategy. We further specify the framework and propose a practical algorithm that leverages knowledge from game theory and graph theory. Furthermore, an analysis of the learning process of the algorithm shows that it can efficiently overcome cooperative incompatibility. The experimental results in the Overcooked game environment demonstrate that our method outperforms current state-of-the-art methods when coordinating with different-level partners. Our demo is available at https://sites.google.com/view/cole-2023.
Generalized Video Anomaly Event Detection: Systematic Taxonomy and Comparison of Deep Models
Venue: arXiv cs.CV 2024
Year: 2024

Video Anomaly Detection (VAD) serves as a pivotal technology in the intelligent surveillance systems, enabling the temporal or spatial identification of anomalous events within videos. While existing reviews predominantly concentrate on conventional unsupervised methods, they often overlook the emergence of weakly-supervised and fully-unsupervised approaches. To address this gap, this survey extends the conventional scope of VAD beyond unsupervised methods, encompassing a broader spectrum termed Generalized Video Anomaly Event Detection (GVAED). By skillfully incorporating recent advancements rooted in diverse assumptions and learning frameworks, this survey introduces an intuitive taxonomy that seamlessly navigates through unsupervised, weakly-supervised, supervised and fully-unsupervised VAD methodologies, elucidating the distinctions and interconnections within these research trajectories. In addition, this survey facilitates prospective researchers by assembling a compilation of research resources, including public datasets, available codebases, programming tools, and pertinent literature. Furthermore, this survey quantitatively assesses model performance, delves into research challenges and directions, and outlines potential avenues for future exploration.
Interpretable Deep Learning for Forecasting Online Advertising Costs: Insights from the Competitive Bidding Landscape
Venue: arXiv cs.LG 2024
Year: 2024

As advertisers increasingly shift their budgets toward digital advertising, accurately forecasting advertising costs becomes essential for optimizing marketing campaign returns. This paper presents a comprehensive study that employs various time-series forecasting methods to predict daily average CPC in the online advertising market. We evaluate the performance of statistical models, machine learning techniques, and deep learning approaches, including the Temporal Fusion Transformer (TFT). Our findings reveal that incorporating multivariate models, enriched with covariates derived from competitors' CPC patterns through time-series clustering, significantly improves forecasting accuracy. We interpret the results by analyzing feature importance and temporal attention, demonstrating how the models leverage both the advertiser's data and insights from the competitive landscape. Additionally, our method proves robust during major market shifts, such as the COVID-19 pandemic, consistently outperforming models that rely solely on individual advertisers' data. This study introduces a scalable technique for selecting relevant covariates from a broad pool of advertisers, offering more accurate long-term forecasts and strategic insights into budget allocation and competitive dynamics in digital advertising.
Distributional GFlowNets with Quantile Flows
Venue: arXiv cs.LG 2024
Year: 2024

Generative Flow Networks (GFlowNets) are a new family of probabilistic samplers where an agent learns a stochastic policy for generating complex combinatorial structure through a series of decision-making steps. Despite being inspired from reinforcement learning, the current GFlowNet framework is relatively limited in its applicability and cannot handle stochasticity in the reward function. In this work, we adopt a distributional paradigm for GFlowNets, turning each flow function into a distribution, thus providing more informative learning signals during training. By parameterizing each edge flow through their quantile functions, our proposed \textit{quantile matching} GFlowNet learning algorithm is able to learn a risk-sensitive policy, an essential component for handling scenarios with risk uncertainty. Moreover, we find that the distributional approach can achieve substantial improvement on existing benchmarks compared to prior methods due to our enhanced training algorithm, even in settings with deterministic rewards.
Investigation of Enhanced Inertial Navigation Algorithms by Functional Iteration
Venue: arXiv cs.RO 2024
Year: 2024

The defects of the traditional strapdown inertial navigation algorithms become well acknowledged and the corresponding enhanced algorithms have been quite recently proposed trying to mitigate both theoretical and algorithmic defects. In this paper, the analytical accuracy evaluation of both the traditional algorithms and the enhanced algorithms is investigated, against the true reference for the first time enabled by the functional iteration approach having provable convergence. The analyses by the help of MATLAB Symbolic Toolbox show that the resultant error orders of all algorithms under investigation are consistent with those in the existing literatures, and the enhanced attitude algorithm notably reduces error orders of the traditional counterpart, while the impact of the enhanced velocity algorithm on error order reduction is insignificant. Simulation results agree with analyses that the superiority of the enhanced algorithm over the traditional one in the body-frame attitude computation scenario diminishes significantly in the entire inertial navigation computation scenario, while the functional iteration approach possesses significant accuracy superiority even under sustained lowly dynamic conditions.
A Comprehensive Survey on Graph Summarization with Graph Neural Networks
Venue: arXiv cs.LG 2024
Year: 2024

As large-scale graphs become more widespread, more and more computational challenges with extracting, processing, and interpreting large graph data are being exposed. It is therefore natural to search for ways to summarize these expansive graphs while preserving their key characteristics. In the past, most graph summarization techniques sought to capture the most important part of a graph statistically. However, today, the high dimensionality and complexity of modern graph data are making deep learning techniques more popular. Hence, this paper presents a comprehensive survey of progress in deep learning summarization techniques that rely on graph neural networks (GNNs). Our investigation includes a review of the current state-of-the-art approaches, including recurrent GNNs, convolutional GNNs, graph autoencoders, and graph attention networks. A new burgeoning line of research is also discussed where graph reinforcement learning is being used to evaluate and improve the quality of graph summaries. Additionally, the survey provides details of benchmark datasets, evaluation metrics, and open-source tools that are often employed in experimentation settings, along with a detailed comparison, discussion, and takeaways for the research community focused on graph summarization. Finally, the survey concludes with a number of open research challenges to motivate further study in this area.
Revet: A Language and Compiler for Dataflow Threads
Venue: arXiv cs.AR 2024
Year: 2024

Spatial dataflow architectures such as reconfigurable dataflow accelerators (RDA) can provide much higher performance and efficiency than CPUs and GPUs. In particular, vectorized reconfigurable dataflow accelerators (vRDA) in recent literature represent a design point that enhances the efficiency of dataflow architectures with vectorization. Today, vRDAs can be exploited using either hardcoded kernels or MapReduce languages like Spatial, which cannot vectorize data-dependent control flow. In contrast, CPUs and GPUs can be programmed using general-purpose threaded abstractions. The ideal combination would be the generality of a threaded programming model coupled with the efficient execution model of a vRDA. We introduce Revet: a programming model, compiler, and execution model that lets threaded applications run efficiently on vRDAs. The Revet programming language uses threads to support a broader range of applications than Spatial's parallel patterns, and our MLIR-based compiler lowers this language to a generic dataflow backend that operates on streaming tensors. Finally, we show that mapping threads to dataflow outperforms GPUs, the current state-of-the-art for threaded accelerators, by 3.8x.
A Logic for Veracity
Venue: arXiv cs.LO 2024
Year: 2024

This paper shows the initial stages of development, from first principles, of a formal logic to characterise and then explore issues in a broadly defined idea of Veracity, which includes properties of demonstrability, truth, trust and authenticity.
Enhancing SMT-based Weighted Model Integration by Structure Awareness
Venue: arXiv cs.AI 2024
Year: 2024

The development of efficient exact and approximate algorithms for probabilistic inference is a long-standing goal of artificial intelligence research. Whereas substantial progress has been made in dealing with purely discrete or purely continuous domains, adapting the developed solutions to tackle hybrid domains, characterised by discrete and continuous variables and their relationships, is highly non-trivial. Weighted Model Integration (WMI) recently emerged as a unifying formalism for probabilistic inference in hybrid domains. Despite a considerable amount of recent work, allowing WMI algorithms to scale with the complexity of the hybrid problem is still a challenge. In this paper we highlight some substantial limitations of existing state-of-the-art solutions, and develop an algorithm that combines SMT-based enumeration, an efficient technique in formal verification, with an effective encoding of the problem structure. This allows our algorithm to avoid generating redundant models, resulting in drastic computational savings. Additionally, we show how SMT-based approaches can seamlessly deal with different integration techniques, both exact and approximate, significantly expanding the set of problems that can be tackled by WMI technology. An extensive experimental evaluation on both synthetic and real-world datasets confirms the substantial advantage of the proposed solution over existing alternatives. The application potential of this technology is further showcased on a prototypical task aimed at verifying the fairness of probabilistic programs.
Closure Properties of General Grammars -- Formally Verified
Venue: arXiv cs.FL 2024
Year: 2024

We formalized general (i.e., type-0) grammars using the Lean 3 proof assistant. We defined basic notions of rewrite rules and of words derived by a grammar, and used grammars to show closure of the class of type-0 languages under four operations: union, reversal, concatenation, and the Kleene star. The literature mostly focuses on Turing machine arguments, which are possibly more difficult to formalize. For the Kleene star, we could not follow the literature and came up with our own grammar-based construction.
A Deep Learning-based Global and Segmentation-based Semantic Feature Fusion Approach for Indoor Scene Classification
Venue: arXiv cs.CV 2024
Year: 2024

This work proposes a novel approach that uses a semantic segmentation mask to obtain a 2D spatial layout of the segmentation-categories across the scene, designated by segmentation-based semantic features (SSFs). These features represent, per segmentation-category, the pixel count, as well as the 2D average position and respective standard deviation values. Moreover, a two-branch network, GS2F2App, that exploits CNN-based global features extracted from RGB images and the segmentation-based features extracted from the proposed SSFs, is also proposed. GS2F2App was evaluated in two indoor scene benchmark datasets: the SUN RGB-D and the NYU Depth V2, achieving state-of-the-art results on both datasets.
Label-efficient Time Series Representation Learning: A Review
Venue: arXiv cs.LG 2024
Year: 2024

Label-efficient time series representation learning, which aims to learn effective representations with limited labeled data, is crucial for deploying deep learning models in real-world applications. To address the scarcity of labeled time series data, various strategies, e.g., transfer learning, self-supervised learning, and semi-supervised learning, have been developed. In this survey, we introduce a novel taxonomy for the first time, categorizing existing approaches as in-domain or cross-domain, based on their reliance on external data sources or not. Furthermore, we present a review of the recent advances in each strategy, conclude the limitations of current methodologies, and suggest future research directions that promise further improvements in the field.
A Convex Hull Cheapest Insertion Heuristic for the Non-Euclidean TSP
Venue: arXiv cs.AI 2024
Year: 2024

The convex hull cheapest insertion heuristic produces good solutions to the Euclidean Traveling Salesperson Problem, but it has never been extended to the non-Euclidean problem. This paper uses multidimensional scaling to first project the points from a non-Euclidean space into a Euclidean space, enabling the generation of a convex hull that initializes the algorithm. To evaluate the proposed algorithm, non-Euclidean spaces are created by adding separators to the TSPLIB data-set, or by using the L1 norm as a metric.