OpenRaiser/Intern-Atlas huggingface dataset:
Interdependency and cascading failures in co-patenting and shareholding interfirm networks
Venue: arXiv cs.SI 2024
Year: 2024

This work analyses the interdependent link creation of patent and shareholding links in interfirm networks, and how this dynamics affects the resilience of such networks in the face of cascading failures. Using the Orbis dataset, we construct very large co-patenting and shareholding networks, globally as well as in terms of individual countries. Besides, we construct smaller overlap networks from those firm pairs which have both types of links between them, for nine years between 2008-2016. We use information theoretic measures, such as mutual information, active information storage, and transfer entropy, to characterise the topological similarities and shared topological information between the relevant co-patenting and shareholding networks. We then construct a cascading failure model, and use it to analyse the resilience of interdependent interfirm networks in terms of multiple failure characteristics. We find that there is relatively high level of mutual information between co-patenting networks and the shareholding networks from later years, suggesting that the formation of shareholding links is influenced by the existence of patent links in previous years. We also show that this influence becomes most apparent after a delay of four years between the formation of co-patenting links and shareholding links. Analysing the resilience of shareholding networks against cascading failures, we show that in terms of both mean downtime, and failure proportion of firms, certain countries have less resilient shareholding networks compared to other countries with significant economies. Based on our results, we postulate that an interfirm network model which considers multiple types of relationships together could be a to highlight important features of economic systems around the world.
Efficiently Scanning and Resampling Spatio-Temporal Tasks with Irregular Observations
Venue: arXiv cs.LG 2024
Year: 2024

Various works have aimed at combining the inference efficiency of recurrent models and training parallelism of multi-head attention for sequence modeling. However, most of these works focus on tasks with fixed-dimension observation spaces, such as individual tokens in language modeling or pixels in image completion. To handle an observation space of varying size, we propose a novel algorithm that alternates between cross-attention between a 2D latent state and observation, and a discounted cumulative sum over the sequence dimension to efficiently accumulate historical information. We find this resampling cycle is critical for performance. To evaluate efficient sequence modeling in this domain, we introduce two multi-agent intention tasks: simulated agents chasing bouncing particles and micromanagement analysis in professional StarCraft II games. Our algorithm achieves comparable accuracy with a lower parameter count, faster training and inference compared to existing methods.
Chain-of-Restoration: Multi-Task Image Restoration Models are Zero-Shot Step-by-Step Universal Image Restorers
Venue: arXiv cs.CV 2024
Year: 2024

Despite previous image restoration (IR) methods have often concentrated on isolated degradations, recent research has increasingly focused on addressing composite degradations involving a complex combination of multiple isolated degradations. However, current IR methods for composite degradations require building training data that contain an exponential number of possible degradation combinations, which brings in a significant burden. To alleviate this issue, this paper proposes a new task setting, i.e. Universal Image Restoration (UIR). Specifically, UIR doesn't require training on all the degradation combinations but only on a set of degradation bases and then removing any degradation that these bases can potentially compose in a zero-shot manner. Inspired by the Chain-of-Thought that prompts large language models (LLMs) to address problems step-by-step, we propose Chain-of-Restoration (CoR) mechanism, which instructs models to remove unknown composite degradations step-by-step. By integrating a simple Degradation Discriminator into pre-trained multi-task models, CoR facilitates the process where models remove one degradation basis per step, continuing this process until the image is fully restored from the unknown composite degradation. Extensive experiments show that CoR can significantly improve model performance in removing composite degradations, achieving comparable or better results than those state-of-the-art (SoTA) methods trained on all degradations.
Contrastive Knowledge Distillation for Robust Multimodal Sentiment Analysis
Venue: arXiv cs.MM 2024
Year: 2024

Multimodal sentiment analysis (MSA) systems leverage information from different modalities to predict human sentiment intensities. Incomplete modality is an important issue that may cause a significant performance drop in MSA systems. By generative imputation, i.e., recovering the missing data from available data, systems may achieve robust performance but will lead to high computational costs. This paper introduces a knowledge distillation method, called `Multi-Modal Contrastive Knowledge Distillation' (MM-CKD), to address the issue of incomplete modality in video sentiment analysis with lower computation cost, as a novel non-imputation-based method. We employ Multi-view Supervised Contrastive Learning (MVSC) to transfer knowledge from a teacher model to student models. This approach not only leverages cross-modal knowledge but also introduces cross-sample knowledge with supervision, jointly improving the performance of both teacher and student models through online learning. Our method gives competitive results with significantly lower computational costs than state-of-the-art imputation-based methods.
On-Chip Learning via Transformer In-Context Learning
Venue: arXiv cs.NE 2024
Year: 2024

Autoregressive decoder-only transformers have become key components for scalable sequence processing and generation models. However, the transformer's self-attention mechanism requires transferring prior token projections from the main memory at each time step (token), thus severely limiting their performance on conventional processors. Self-attention can be viewed as a dynamic feed-forward layer, whose matrix is input sequence-dependent similarly to the result of local synaptic plasticity. Using this insight, we present a neuromorphic decoder-only transformer model that utilizes an on-chip plasticity processor to compute self-attention. Interestingly, the training of transformers enables them to ``learn'' the input context during inference. We demonstrate this in-context learning ability of transformers on the Loihi 2 processor by solving a few-shot classification problem. With this we emphasize the importance of pretrained models especially their ability to find simple, local, backpropagation free, learning rules enabling on-chip learning and adaptation in a hardware friendly manner.
5G as Enabler for Industrie 4.0 Use Cases: Challenges and Concepts
Venue: arXiv cs.NI 2024
Year: 2024

The increasing demand for highly customized products, as well as flexible production lines, can be seen as trigger for the "fourth industrial revolution", referred to as "Industrie 4.0". Current systems usually rely on wire-line technologies to connect sensors and actuators. To enable a higher flexibility such as moving robots or drones, these connections need to be replaced by wireless technologies in the future. Furthermore, this facilitates the renewal of brownfield deployments to address Industrie 4.0 requirements. This paper proposes representative use cases, which have been examined in the German Tactile Internet 4.0 (TACNET 4.0) research project. In order to analyze these use cases, this paper identifies the main challenges and requirements of communication networks in Industrie 4.0 and discusses the applicability of 5th generation wireless communication systems (5G).
Bad Neighbors: On Understanding VPN Provider Networks
Venue: arXiv cs.CR 2024
Year: 2024

Virtual Private Network (VPN) solutions are used to connect private networks securely over the Internet. Besides their benefits in corporate environments, VPNs are also marketed to privacy-minded users to preserve their privacy, and to bypass geolocation-based content blocking and censorship. This has created a market for turnkey VPN services offering a multitude of vantage points all over the world for a monthly price. While VPN providers are heavily using privacy and security benefits in their marketing, such claims are generally hard to measure and substantiate. While there exist some studies on the VPN ecosystem, all prior works omit a critical part in their analyses: (i) How well do the providers configure and secure their own network infrastructure? and (ii) How well are they protecting their customers from other customers? To answer these questions, we have developed an automated measurement system with which we conduct a large-scale analysis of VPN providers and their thousands of VPN endpoints. Considering the fact that VPNs work internally using non-Internet-routable IP addresses, they might enable access to otherwise inaccessible networks. If not properly secured, this can inadvertently expose internal networks of these providers, or worse, even other clients connected to their services. Our results indicate a widespread lack of traffic filtering towards internally routable networks on the majority of tested VPN service providers, even in cases where no other VPN customers were directly exposed. We have disclosed our findings to the affected providers and other stakeholders, and offered guidance to improve the situation.
Last-iterate Convergence in Regularized Graphon Mean Field Game
Venue: arXiv cs.GT 2024
Year: 2024

To model complex real-world systems, such as traders in stock markets, or the dissemination of contagious diseases, graphon mean-field games (GMFG) have been proposed to model many agents. Despite the empirical success, our understanding of GMFG is limited. Popular algorithms such as mirror descent are deployed but remain unknown for their convergence properties. In this work, we give the first last-iterate convergence rate of mirror descent in regularized monotone GMFG. In tabular monotone GMFG with finite state and action spaces and under bandit feedback, we show a last-iterate convergence rate of $O(T^{-1/4})$. Moreover, when exact knowledge of costs and transitions is available, we improve this convergence rate to $O(T^{-1})$, matching the existing convergence rate observed in strongly convex games. In linear GMFG, our algorithm achieves a last-iterate convergence rate of $O(T^{-1/5})$. Finally, we verify the performance of the studied algorithms by empirically testing them against fictitious play in a variety of tasks.
Enhancing GNNs with Architecture-Agnostic Graph Transformations: A Systematic Analysis
Venue: arXiv cs.LG 2024
Year: 2024

In recent years, a wide variety of graph neural network (GNN) architectures have emerged, each with its own strengths, weaknesses, and complexities. Various techniques, including rewiring, lifting, and node annotation with centrality values, have been employed as pre-processing steps to enhance GNN performance. However, there are no universally accepted best practices, and the impact of architecture and pre-processing on performance often remains opaque. This study systematically explores the impact of various graph transformations as pre-processing steps on the performance of common GNN architectures across standard datasets. The models are evaluated based on their ability to distinguish non-isomorphic graphs, referred to as expressivity. Our findings reveal that certain transformations, particularly those augmenting node features with centrality measures, consistently improve expressivity. However, these gains come with trade-offs, as methods like graph encoding, while enhancing expressivity, introduce numerical inaccuracies widely-used python packages. Additionally, we observe that these pre-processing techniques are limited when addressing complex tasks involving 3-WL and 4-WL indistinguishable graphs.
HpEIS: Learning Hand Pose Embeddings for Multimedia Interactive Systems
Venue: arXiv cs.CV 2024
Year: 2024

We present a novel Hand-pose Embedding Interactive System (HpEIS) as a virtual sensor, which maps users' flexible hand poses to a two-dimensional visual space using a Variational Autoencoder (VAE) trained on a variety of hand poses. HpEIS enables visually interpretable and guidable support for user explorations in multimedia collections, using only a camera as an external hand pose acquisition device. We identify general usability issues associated with system stability and smoothing requirements through pilot experiments with expert and inexperienced users. We then design stability and smoothing improvements, including hand-pose data augmentation, an anti-jitter regularisation term added to loss function, stabilising post-processing for movement turning points and smoothing post-processing based on One Euro Filters. In target selection experiments (n=12), we evaluate HpEIS by measures of task completion time and the final distance to target points, with and without the gesture guidance window condition. Experimental responses indicate that HpEIS provides users with a learnable, flexible, stable and smooth mid-air hand movement interaction experience.
Efficient Differentiable Discovery of Causal Order
Venue: arXiv cs.LG 2024
Year: 2024

In the algorithm Intersort, Chevalley et al. (2024) proposed a score-based method to discover the causal order of variables in a Directed Acyclic Graph (DAG) model, leveraging interventional data to outperform existing methods. However, as a score-based method over the permutahedron, Intersort is computationally expensive and non-differentiable, limiting its ability to be utilised in problems involving large-scale datasets, such as those in genomics and climate models, or to be integrated into end-to-end gradient-based learning frameworks. We address this limitation by reformulating Intersort using differentiable sorting and ranking techniques. Our approach enables scalable and differentiable optimization of causal orderings, allowing the continuous score function to be incorporated as a regularizer in downstream tasks. Empirical results demonstrate that causal discovery algorithms benefit significantly from regularizing on the causal order, underscoring the effectiveness of our method. Our work opens the door to efficiently incorporating regularization for causal order into the training of differentiable models and thereby addresses a long-standing limitation of purely associational supervised learning.
M$^3$-Impute: Mask-guided Representation Learning for Missing Value Imputation
Venue: arXiv cs.LG 2024
Year: 2024

Missing values are a common problem that poses significant challenges to data analysis and machine learning. This problem necessitates the development of an effective imputation method to fill in the missing values accurately, thereby enhancing the overall quality and utility of the datasets. Existing imputation methods, however, fall short of explicitly considering the `missingness' information in the data during the embedding initialization stage and modeling the entangled feature and sample correlations during the learning process, thus leading to inferior performance. We propose M$^3$-Impute, which aims to explicitly leverage the missingness information and such correlations with novel masking schemes. M$^3$-Impute first models the data as a bipartite graph and uses a graph neural network to learn node embeddings, where the refined embedding initialization process directly incorporates the missingness information. They are then optimized through M$^3$-Impute's novel feature correlation unit (FRU) and sample correlation unit (SRU) that effectively captures feature and sample correlations for imputation. Experiment results on 25 benchmark datasets under three different missingness settings show the effectiveness of M$^3$-Impute by achieving 20 best and 4 second-best MAE scores on average.
Enhancing Indonesian Automatic Speech Recognition: Evaluating Multilingual Models with Diverse Speech Variabilities
Venue: arXiv cs.CL 2024
Year: 2024

An ideal speech recognition model has the capability to transcribe speech accurately under various characteristics of speech signals, such as speaking style (read and spontaneous), speech context (formal and informal), and background noise conditions (clean and moderate). Building such a model requires a significant amount of training data with diverse speech characteristics. Currently, Indonesian data is dominated by read, formal, and clean speech, leading to a scarcity of Indonesian data with other speech variabilities. To develop Indonesian automatic speech recognition (ASR), we present our research on state-of-the-art speech recognition models, namely Massively Multilingual Speech (MMS) and Whisper, as well as compiling a dataset comprising Indonesian speech with variabilities to facilitate our study. We further investigate the models' predictive ability to transcribe Indonesian speech data across different variability groups. The best results were achieved by the Whisper fine-tuned model across datasets with various characteristics, as indicated by the decrease in word error rate (WER) and character error rate (CER). Moreover, we found that speaking style variability affected model performance the most.
Adaptive optimization of wave energy conversion in oscillatory wave surge converters via SPH simulation and deep reinforcement learning
Venue: arXiv cs.CE 2024
Year: 2024

The nonlinear damping characteristics of the oscillating wave surge converter (OWSC) significantly impact the performance of the power take-off system. This study presents a framework by integrating deep reinforcement learning (DRL) with numerical simulations of OWSC to identify optimal adaptive damping policy under varying wave conditions, thereby enhancing wave energy harvesting efficiency. Firstly, the open-source multiphysics libraries SPHinXsys and Simbody are employed to establish the numerical environment for wave interaction with OWSCs. Subsequently, a comparative analysis of three DRL algorithms-proximal policy optimization (PPO), twin delayed deep deterministic policy gradient (TD3), and soft actor-critic (SAC)-is conducted using the two-dimensional (2D) numerical study of OWSC interacting with regular waves. The results reveal that artificial neural networks capture the nonlinear characteristics of wave-structure interactions and provide efficient PTO policies. Notably, the SAC algorithm demonstrates exceptional robustness and accuracy, achieving a 10.61% improvement in wave energy harvesting. Furthermore, policies trained in a 2D environment are successfully applied to the three-dimensional (3D) study, with an improvement of 22.54% in energy harvesting. Additionally, the study shows that energy harvesting is improved by 6.42% for complex irregular waves. However, for the complex dual OWSC system, optimizing the damping characteristics alone is insufficient to enhance energy harvesting.
Fragile Giants: Understanding the Susceptibility of Models to Subpopulation Attacks
Venue: arXiv cs.LG 2024
Year: 2024

As machine learning models become increasingly complex, concerns about their robustness and trustworthiness have become more pressing. A critical vulnerability of these models is data poisoning attacks, where adversaries deliberately alter training data to degrade model performance. One particularly stealthy form of these attacks is subpopulation poisoning, which targets distinct subgroups within a dataset while leaving overall performance largely intact. The ability of these attacks to generalize within subpopulations poses a significant risk in real-world settings, as they can be exploited to harm marginalized or underrepresented groups within the dataset. In this work, we investigate how model complexity influences susceptibility to subpopulation poisoning attacks. We introduce a theoretical framework that explains how overparameterized models, due to their large capacity, can inadvertently memorize and misclassify targeted subpopulations. To validate our theory, we conduct extensive experiments on large-scale image and text datasets using popular model architectures. Our results show a clear trend: models with more parameters are significantly more vulnerable to subpopulation poisoning. Moreover, we find that attacks on smaller, human-interpretable subgroups often go undetected by these models. These results highlight the need to develop defenses that specifically address subpopulation vulnerabilities.
An End-to-End Deep Learning Method for Solving Nonlocal Allen-Cahn and Cahn-Hilliard Phase-Field Models
Venue: arXiv cs.LG 2024
Year: 2024

We propose an efficient end-to-end deep learning method for solving nonlocal Allen-Cahn (AC) and Cahn-Hilliard (CH) phase-field models. One motivation for this effort emanates from the fact that discretized partial differential equation-based AC or CH phase-field models result in diffuse interfaces between phases, with the only recourse for remediation is to severely refine the spatial grids in the vicinity of the true moving sharp interface whose width is determined by a grid-independent parameter that is substantially larger than the local grid size. In this work, we introduce non-mass conserving nonlocal AC or CH phase-field models with regular, logarithmic, or obstacle double-well potentials. Because of non-locality, some of these models feature totally sharp interfaces separating phases. The discretization of such models can lead to a transition between phases whose width is only a single grid cell wide. Another motivation is to use deep learning approaches to ameliorate the otherwise high cost of solving discretized nonlocal phase-field models. To this end, loss functions of the customized neural networks are defined using the residual of the fully discrete approximations of the AC or CH models, which results from applying a Fourier collocation method and a temporal semi-implicit approximation. To address the long-range interactions in the models, we tailor the architecture of the neural network by incorporating a nonlocal kernel as an input channel to the neural network model. We then provide the results of extensive computational experiments to illustrate the accuracy, structure-preserving properties, predictive capabilities, and cost reductions of the proposed method.
Wikimedia data for AI: a review of Wikimedia datasets for NLP tasks and AI-assisted editing
Venue: arXiv cs.CY 2024
Year: 2024

Wikimedia content is used extensively by the AI community and within the language modeling community in particular. In this paper, we provide a review of the different ways in which Wikimedia data is curated to use in NLP tasks across pre-training, post-training, and model evaluations. We point to opportunities for greater use of Wikimedia content but also identify ways in which the language modeling community could better center the needs of Wikimedia editors. In particular, we call for incorporating additional sources of Wikimedia data, a greater focus on benchmarks for LLMs that encode Wikimedia principles, and greater multilingualism in Wikimedia-derived datasets.