OpenRaiser/Intern-Atlas huggingface dataset:
Integrating AI Planning with Natural Language Processing: A Combination of Explicit and Tacit Knowledge
Venue: arXiv cs.AI 2023
Year: 2023

Natural language processing (NLP) aims at investigating the interactions between agents and humans, processing and analyzing large amounts of natural language data. Large-scale language models play an important role in current natural language processing. However, the challenges of explainability and complexity come along with the developments of language models. One way is to introduce logical relations and rules into natural language processing models, such as making use of Automated Planning. Automated planning (AI planning) focuses on building symbolic domain models and synthesizing plans to transit initial states to goals based on domain models. Recently, there have been plenty of works related to these two fields, which have the abilities to generate explicit knowledge, e.g., preconditions and effects of action models, and learn from tacit knowledge, e.g., neural models, respectively. Integrating AI planning and natural language processing effectively improves the communication between human and intelligent agents. This paper outlines the commons and relations between AI planning and natural language processing, argues that each of them can effectively impact on the other one by five areas: (1) planning-based text understanding, (2) planning-based natural language processing, (3) planning-based explainability, (4) text-based human-robot interaction, and (5) applications. We also explore some potential future issues between AI planning and natural language processing. To the best of our knowledge, this survey is the first work that addresses the deep connections between AI planning and Natural language processing.
Personalized Prompt Learning for Explainable Recommendation
Venue: arXiv cs.IR 2023
Year: 2023

Providing user-understandable explanations to justify recommendations could help users better understand the recommended items, increase the system's ease of use, and gain users' trust. A typical approach to realize it is natural language generation. However, previous works mostly adopt recurrent neural networks to meet the ends, leaving the potentially more effective pre-trained Transformer models under-explored. In fact, user and item IDs, as important identifiers in recommender systems, are inherently in different semantic space as words that pre-trained models were already trained on. Thus, how to effectively fuse IDs into such models becomes a critical issue. Inspired by recent advancement in prompt learning, we come up with two solutions: find alternative words to represent IDs (called discrete prompt learning), and directly input ID vectors to a pre-trained model (termed continuous prompt learning). In the latter case, ID vectors are randomly initialized but the model is trained in advance on large corpora, so they are actually in different learning stages. To bridge the gap, we further propose two training strategies: sequential tuning and recommendation as regularization. Extensive experiments show that our continuous prompt learning approach equipped with the training strategies consistently outperforms strong baselines on three datasets of explainable recommendation.
Efficient Content Delivery in User-Centric and Cache-Enabled Vehicular Edge Networks with Deadline-Constrained Heterogeneous Demands
Venue: arXiv cs.NI 2023
Year: 2023

Modern connected vehicles (CVs) frequently require diverse types of content for mission-critical decision-making and onboard users' entertainment. These contents are required to be fully delivered to the requester CVs within stringent deadlines that the existing radio access technology (RAT) solutions may fail to ensure. Motivated by the above consideration, this paper exploits content caching in vehicular edge networks (VENs) with a software-defined user-centric virtual cell (VC) based RAT solution for delivering the requested contents from a proximity edge server. Moreover, to capture the heterogeneous demands of the CVs, we introduce a preference-popularity tradeoff in their content request model. To that end, we formulate a joint optimization problem for content placement, CV scheduling, VC configuration, VC-CV association and radio resource allocation to minimize long-term content delivery delay. However, the joint problem is highly complex and cannot be solved efficiently in polynomial time. As such, we decompose the original problem into a cache placement problem and a content delivery delay minimization problem given the cache placement policy. We use deep reinforcement learning (DRL) as a learning solution for the first sub-problem. Furthermore, we transform the delay minimization problem into a priority-based weighted sum rate (WSR) maximization problem, which is solved leveraging maximum bipartite matching (MWBM) and a simple linear search algorithm. Our extensive simulation results demonstrate the effectiveness of the proposed method compared to existing baselines in terms of cache hit ratio (CHR), deadline violation and content delivery delay.
Knowledge Distillation with Deep Supervision
Venue: arXiv cs.LG 2023
Year: 2023

Knowledge distillation aims to enhance the performance of a lightweight student model by exploiting the knowledge from a pre-trained cumbersome teacher model. However, in the traditional knowledge distillation, teacher predictions are only used to provide the supervisory signal for the last layer of the student model, which may result in those shallow student layers lacking accurate training guidance in the layer-by-layer back propagation and thus hinders effective knowledge transfer. To address this issue, we propose Deeply-Supervised Knowledge Distillation (DSKD), which fully utilizes class predictions and feature maps of the teacher model to supervise the training of shallow student layers. A loss-based weight allocation strategy is developed in DSKD to adaptively balance the learning process of each shallow layer, so as to further improve the student performance. Extensive experiments on CIFAR-100 and TinyImageNet with various teacher-student models show significantly performance, confirming the effectiveness of our proposed method. Code is available at: $\href{https://github.com/luoshiya/DSKD}{https://github.com/luoshiya/DSKD}$
Alliance Makes Difference? Maximizing Social Welfare in Cross-Silo Federated Learning
Venue: arXiv cs.GT 2023
Year: 2023

As one of the typical settings of Federated Learning (FL), cross-silo FL allows organizations to jointly train an optimal Machine Learning (ML) model. In this case, some organizations may try to obtain the global model without contributing their local training power, lowering the social welfare. In this paper, we model the interactions among organizations in cross-silo FL as a public goods game and theoretically prove that there exists a social dilemma where the maximum social welfare is not achieved in Nash equilibrium. To overcome this dilemma, we employ the Multi-player Multi-action Zero-Determinant (MMZD) strategy to maximize the social welfare. With the help of the MMZD, an individual organization can unilaterally control the social welfare without extra cost. Since the MMZD strategy can be adopted by all organizations, we further study the case of multiple organizations jointly adopting the MMZD strategy to form an MMZD Alliance (MMZDA). We prove that the MMZDA strategy can strengthen the control of the maximum social welfare. Experimental results validate that the MMZD strategy is effective in obtaining the maximum social welfare and the MMZDA can achieve a larger maximum value.
CryptSan: Leveraging ARM Pointer Authentication for Memory Safety in C/C++
Venue: arXiv cs.CR 2023
Year: 2023

Memory safety bugs remain in the top ranks of security vulnerabilities, even after decades of research on their detection and prevention. Various mitigations have been proposed for C/C++, ranging from language dialects to instrumentation. Among these, compiler-based instrumentation is particularly promising, not requiring manual code modifications and being able to achieve precise memory safety. Unfortunately, existing compiler-based solutions compromise in many areas, including performance but also usability and memory safety guarantees. New developments in hardware can help improve performance and security of compiler-based memory safety. ARM Pointer Authentication, added in the ARMv8.3 architecture, is intended to enable hardware-assisted Control Flow Integrity (CFI). But since its operations are generic, it also enables other, more comprehensive hardware-supported runtime integrity approaches. As such, we propose CryptSan, a memory safety approach based on ARM Pointer Authentication. CryptSan uses pointer signatures to retrofit memory safety to C/C++ programs, protecting heap, stack, and globals against temporal and spatial vulnerabilities. We present a full LLVM-based prototype implementation, running on an M1 MacBook Pro, i.e., on actual ARMv8.3 hardware. Our prototype evaluation shows that the system outperforms similar approaches under real-world conditions. This, together with its interoperability with uninstrumented libraries and cryptographic protection against attacks on metadata, makes CryptSan a viable solution for retrofitting memory safety to C/C++ programs.
Grammar-Based Grounded Lexicon Learning
Venue: arXiv cs.CL 2023
Year: 2023

We present Grammar-Based Grounded Lexicon Learning (G2L2), a lexicalist approach toward learning a compositional and grounded meaning representation of language from grounded data, such as paired images and texts. At the core of G2L2 is a collection of lexicon entries, which map each word to a tuple of a syntactic type and a neuro-symbolic semantic program. For example, the word shiny has a syntactic type of adjective; its neuro-symbolic semantic program has the symbolic form λx. filter(x, SHINY), where the concept SHINY is associated with a neural network embedding, which will be used to classify shiny objects. Given an input sentence, G2L2 first looks up the lexicon entries associated with each token. It then derives the meaning of the sentence as an executable neuro-symbolic program by composing lexical meanings based on syntax. The recovered meaning programs can be executed on grounded inputs. To facilitate learning in an exponentially-growing compositional space, we introduce a joint parsing and expected execution algorithm, which does local marginalization over derivations to reduce the training time. We evaluate G2L2 on two domains: visual reasoning and language-driven navigation. Results show that G2L2 can generalize from small amounts of data to novel compositions of words.
Stratified Multivariate Multiscale Dispersion Entropy for Physiological Signal Analysis
Venue: arXiv cs.IT 2023
Year: 2023

Multivariate entropy quantification algorithms are becoming a prominent tool for the extraction of information from multi-channel physiological time-series. However, in the analysis of physiological signals from heterogeneous organ systems, certain channels may overshadow the patterns of others, resulting in information loss. Here, we introduce the framework of Stratified Entropy to prioritize each channels' dynamics based on their allocation to respective strata, leading to a richer description of the multi-channel time-series. As an implementation of the framework, three algorithmic variations of the Stratified Multivariate Multiscale Dispersion Entropy are introduced. These variations and the original algorithm are applied to synthetic time-series, waveform physiological time-series, and derivative physiological data. Based on the synthetic time-series experiments, the variations successfully prioritize channels following their strata allocation while maintaining the low computation time of the original algorithm. In experiments on waveform physiological time-series and derivative physiological data, increased discrimination capacity was noted for multiple strata allocations in the variations when benchmarked to the original algorithm. This suggests improved physiological state monitoring by the variations. Furthermore, our variations can be modified to utilize a priori knowledge for the stratification of channels. Thus, our research provides a novel approach for the extraction of previously inaccessible information from multi-channel time series acquired from heterogeneous systems.
Assessment of Cyber-Physical Intrusion Detection and Classification for Industrial Control Systems
Venue: arXiv cs.CR 2023
Year: 2023

The increasing interaction of industrial control systems (ICSs) with public networks and digital devices introduces new cyber threats to power systems and other critical infrastructure. Recent cyber-physical attacks such as Stuxnet and Irongate revealed unexpected ICS vulnerabilities and a need for improved security measures. Intrusion detection systems constitute a key security technology, which typically monitors cyber network data for detecting malicious activities. However, a central characteristic of modern ICSs is the increasing interdependency of physical and cyber network processes. Thus, the integration of network and physical process data is seen as a promising approach to improve predictability in real-time intrusion detection for ICSs by accounting for physical constraints and underlying process patterns. This work systematically assesses machine learning-based cyber-physical intrusion detection and multi-class classification through a comparison to its purely network data-based counterpart and evaluation of misclassifications and detection delay. Multiple supervised detection and classification pipelines are applied on a recent cyber-physical dataset, which describes various cyber attacks and physical faults on a generic ICS. A key finding is that the integration of physical process data improves detection and classification of all considered attack types. In addition, it enables simultaneous processing of attacks and faults, paving the way for holistic cross-domain root cause identification.
ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!
Venue: arXiv cs.LG 2023
Year: 2023

We introduce ProxSkip -- a surprisingly simple and provably efficient method for minimizing the sum of a smooth ($f$) and an expensive nonsmooth proximable ($ψ$) function. The canonical approach to solving such problems is via the proximal gradient descent (ProxGD) algorithm, which is based on the evaluation of the gradient of $f$ and the prox operator of $ψ$ in each iteration. In this work we are specifically interested in the regime in which the evaluation of prox is costly relative to the evaluation of the gradient, which is the case in many applications. ProxSkip allows for the expensive prox operator to be skipped in most iterations: while its iteration complexity is $\mathcal{O}\left(κ\log \frac{1}{\varepsilon}\right)$, where $κ$ is the condition number of $f$, the number of prox evaluations is $\mathcal{O}\left(\sqrtκ \log \frac{1}{\varepsilon}\right)$ only. Our main motivation comes from federated learning, where evaluation of the gradient operator corresponds to taking a local GD step independently on all devices, and evaluation of prox corresponds to (expensive) communication in the form of gradient averaging. In this context, ProxSkip offers an effective acceleration of communication complexity. Unlike other local gradient-type methods, such as FedAvg, SCAFFOLD, S-Local-GD and FedLin, whose theoretical communication complexity is worse than, or at best matching, that of vanilla GD in the heterogeneous data regime, we obtain a provable and large improvement without any heterogeneity-bounding assumptions.
Distributed Out-of-Memory NMF on CPU/GPU Architectures
Venue: arXiv cs.DC 2023
Year: 2023

We propose an efficient distributed out-of-memory implementation of the Non-negative Matrix Factorization (NMF) algorithm for heterogeneous high-performance-computing (HPC) systems. The proposed implementation is based on prior work on NMFk, which can perform automatic model selection and extract latent variables and patterns from data. In this work, we extend NMFk by adding support for dense and sparse matrix operation on multi-node, multi-GPU systems. The resulting algorithm is optimized for out-of-memory (OOM) problems where the memory required to factorize a given matrix is greater than the available GPU memory. Memory complexity is reduced by batching/tiling strategies, and sparse and dense matrix operations are significantly accelerated with GPU cores (or tensor cores when available). Input/Output (I/O) latency associated with batch copies between host and device is hidden using CUDA streams to overlap data transfers and compute asynchronously, and latency associated with collective communications (both intra-node and inter-node) is reduced using optimized NVIDIA Collective Communication Library NCCL based communicators. Benchmark results show significant improvement, from 32X to 76x speedup, with the new implementation using GPUs over the CPU-based NMFk. Good weak scaling was demonstrated on up to 4096 multi-GPU cluster nodes with approximately 25,000 GPUs when decomposing a dense 340 Terabyte-size matrix and an 11 Exabyte-size sparse matrix of density 10e-6.
Implementing Boolean Functions with switching lattice networks
Venue: arXiv cs.ET 2023
Year: 2023

Four terminal switching network is an alternative structure to realize the logic functions in electronic circuit modeling. This network can be used to implement a Boolean function with less number of switches than the two terminal based CMOS switch. Each switch of the network is driven by a Boolean literal. Any switch is connected to its four neighbors if a literal takes the value 1 , else it is disconnected. In our work, we aimed to develop a technique by which we can find out if any Boolean function can be implemented with a given four-terminal network. It is done using the path of any given lattice network. First, we developed a synthesis tool by which we can create a library of Boolean functions with a given four-terminal switching network and random Boolean literals. This tool can be used to check the output of any lattice network which can also function as a lattice network solver. In the next step, we used the library functions to develop and test our MAPPING tool where the functions were given as input and from the output, we can get the implemented function in four terminal lattice network. Finally, we have proposed a systematic procedure to implement any Boolean function with a efficient way by any given one type of lattice network.
Priming Cross-Session Motor Imagery Classification with A Universal Deep Domain Adaptation Framework
Venue: arXiv cs.CV 2023
Year: 2023

Motor imagery (MI) is a common brain computer interface (BCI) paradigm. EEG is non-stationary with low signal-to-noise, classifying motor imagery tasks of the same participant from different EEG recording sessions is generally challenging, as EEG data distribution may vary tremendously among different acquisition sessions. Although it is intuitive to consider the cross-session MI classification as a domain adaptation problem, the rationale and feasible approach is not elucidated. In this paper, we propose a Siamese deep domain adaptation (SDDA) framework for cross-session MI classification based on mathematical models in domain adaptation theory. The proposed framework can be easily applied to most existing artificial neural networks without altering the network structure, which facilitates our method with great flexibility and transferability. In the proposed framework, domain invariants were firstly constructed jointly with channel normalization and Euclidean alignment. Then, embedding features from source and target domain were mapped into the Reproducing Kernel Hilbert Space (RKHS) and aligned accordingly. A cosine-based center loss was also integrated into the framework to improve the generalizability of the SDDA. The proposed framework was validated with two classic and popular convolutional neural networks from BCI research field (EEGNet and ConvNet) in two MI-EEG public datasets (BCI Competition IV IIA, IIB). Compared to the vanilla EEGNet and ConvNet, the proposed SDDA framework was able to boost the MI classification accuracy by 15.2%, 10.2% respectively in IIA dataset, and 5.5%, 4.2% in IIB dataset. The final MI classification accuracy reached 82.01% in IIA dataset and 87.52% in IIB, which outperformed the state-of-the-art methods in the literature.
Finite-Time Analysis of Natural Actor-Critic for POMDPs
Venue: arXiv cs.LG 2023
Year: 2023

We consider the reinforcement learning problem for partially observed Markov decision processes (POMDPs) with large or even countably infinite state spaces, where the controller has access to only noisy observations of the underlying controlled Markov chain. We consider a natural actor-critic method that employs a finite internal memory for policy parameterization, and a multi-step temporal difference learning algorithm for policy evaluation. We establish, to the best of our knowledge, the first non-asymptotic global convergence of actor-critic methods for partially observed systems under function approximation. In particular, in addition to the function approximation and statistical errors that also arise in MDPs, we explicitly characterize the error due to the use of finite-state controllers. This additional error is stated in terms of the total variation distance between the traditional belief state in POMDPs and the posterior distribution of the hidden state when using a finite-state controller. Further, we show that this error can be made small in the case of sliding-block controllers by using larger block sizes.
Improved Security Proof for the Camenisch-Lysyanskaya Signature-Based Synchronized Aggregate Signature Scheme
Venue: arXiv cs.CR 2023
Year: 2023

The Camenisch-Lysyanskaya signature scheme in CRYPTO 2004 is a useful building block to construct privacy-preserving schemes such as anonymous credentials, group signatures or ring signatures. However, the security of this signature scheme relies on the interactive assumption called the LRSW assumption. Even if the interactive assumptions are proven in the generic group model or bilinear group model, the concerns about these assumptions arise in a cryptographic community. This fact caused a barrier to the use of cryptographic schemes whose security relies on these assumptions. Recently, Pointcheval and Sanders proposed the modified Camenisch-Lysyanskaya signature scheme in CT-RSA 2018. This scheme satisfies the EUF-CMA security under the new q-type assumption called the Modified-q-Strong Diffie-Hellman-2 (q-MSDH-2) assumption. However, the size of a q- type assumptions grows dynamically and this fact leads to inefficiency of schemes. In this work, we revisit the Camenisch-Lysyanskaya signature-based synchronized aggregate signature scheme in FC 2013. This scheme is one of the most efficient synchronized aggregate signature schemes with bilinear groups. However, the security of this synchronized aggregate scheme was proven under the one-time LRSW assumption in the random oracle model. We give the new security proof for this synchronized aggregate scheme under the 1-MSDH-2 (static) assumption in the random oracle model with little loss of efficiency.