OpenRaiser/Intern-Atlas huggingface dataset:
Wiretap Secret Key Agreement Via Secure Omniscience
Venue: arXiv cs.IT 2023
Year: 2023

In this paper, we explore the connection between secret key agreement and secure omniscience within the setting of the multiterminal source model with a wiretapper who has side information. While the secret key agreement problem considers the generation of a maximum-rate secret key through public discussion, the secure omniscience problem is concerned with communication protocols for omniscience that minimize the rate of information leakage to the wiretapper. The starting point of our work is a lower bound on the minimum leakage rate for omniscience, $R_{\mathop{\mathrm{L}}}$, in terms of the wiretap secret key capacity, $C_{\mathop{\mathrm{W}}}$. Our interest is in identifying broad classes of sources for which this lower bound is met with equality, in which case we say that there is a duality between secure omniscience and secret key agreement. We show that this duality holds in the case of certain finite linear source (FLS) models, such as two-terminal FLS models and pairwise independent network models on trees with a linear wiretapper. Duality also holds for any FLS model in which $C_{\mathop{\mathrm{W}}}$ is achieved by a perfect linear secret key agreement scheme. We conjecture that the duality in fact holds unconditionally for any FLS model. On the negative side, we give an example of a (non-FLS) source model for which duality does not hold if we limit ourselves to communication-for-omniscience protocols with at most two (interactive) communications. We also address the secure function computation problem and explore the connection between the minimum leakage rate for computing a function and the wiretap secret key capacity.
MutualFormer: Multi-Modality Representation Learning via Cross-Diffusion Attention
Venue: arXiv cs.CV 2023
Year: 2023

Aggregating multi-modality data to obtain reliable data representation attracts more and more attention. Recent studies demonstrate that Transformer models usually work well for multi-modality tasks. Existing Transformers generally either adopt the Cross-Attention (CA) mechanism or simple concatenation to achieve the information interaction among different modalities which generally ignore the issue of modality gap. In this work, we re-think Transformer and extend it to MutualFormer for multi-modality data representation. Rather than CA in Transformer, MutualFormer employs our new design of Cross-Diffusion Attention (CDA) to conduct the information communication among different modalities. Comparing with CA, the main advantages of the proposed CDA are three aspects. First, the crossaffinities in CDA are defined based on the individual modality affinities in the metric space which thus can naturally avoid the issue of modality/domain gap in feature based CA definition. Second, CDA provides a general scheme which can either be used for multimodality representation or serve as the post-optimization for existing CA models. Third, CDA is implemented efficiently. We successfully apply the MutualFormer on different multi-modality learning tasks (i.e., RGB-Depth SOD, RGB-NIR object ReID). Extensive experiments demonstrate the effectiveness of the proposed MutualFormer.
Towards Interactive Reinforcement Learning with Intrinsic Feedback
Venue: arXiv cs.AI 2023
Year: 2023

Reinforcement learning (RL) and brain-computer interfaces (BCI) have experienced significant growth over the past decade. With rising interest in human-in-the-loop (HITL), incorporating human input with RL algorithms has given rise to the sub-field of interactive RL. Adjacently, the field of BCI has long been interested in extracting informative brain signals from neural activity for use in human-computer interactions. A key link between these fields lies in the interpretation of neural activity as feedback such that interactive RL approaches can be employed. We denote this new and emerging medium of feedback as intrinsic feedback. Despite intrinsic feedback's ability to be conveyed automatically and even unconsciously, proper exploration surrounding this key link has largely gone unaddressed by both communities. Thus, to help facilitate a deeper understanding and a more effective utilization, we provide a tutorial-style review covering the motivations, approaches, and open problems of intrinsic feedback and its foundational concepts.
Parallel and Distributed Hybrid Beamforming for Multicell Millimeter Wave MIMO Full Duplex
Venue: arXiv cs.IT 2023
Year: 2023

Full duplex (FD) is an auspicious wireless technology that holds the potential to double data rates through simultaneous transmission and reception. This paper proposes two innovative designs of hybrid beamforming (HYBF) for a multicell massive multiple-input-multiple-output (mMIMO) millimeter wave (mmWave) FD system. Initially, we introduce a novel centralized HYBF (C-HYBF) scheme, which employs the minorization-maximization (MM) method. However, while centralized beamforming designs offer superior performance, they suffer from high computational complexity, substantial communication overhead, and demand expensive computational resources. To surmount these challenges, we present a framework that facilitates per-link parallel and distributed HYBF (P&D- HYBF) in the mmWave frequency band. This cooperative approach enables each base station (BS) to independently solve its local, low-complexity sub-problems in parallel, resulting in a substantial reduction in communication overhead and computational complexity. Simulation results demonstrate that P&D-HYBF achieves comparable performance to C-HYBF, and with only a few radio-frequency (RF) chains, both designs surpass the capabilities of fully digital half duplex (HD) systems.
Computing Voronoi Diagrams in the Polar-Coordinate Model of the Hyperbolic Plane
Venue: arXiv cs.CG 2023
Year: 2023

A Voronoi diagram is a basic geometric structure that partitions the space into regions associated with a given set of sites, such that all points in a region are closer to the corresponding site than to all other sites. While being thoroughly studied in Euclidean space, they are also of interest in hyperbolic space. In fact, there are several algorithms for computing hyperbolic Voronoi diagrams that work with the various models used to describe hyperbolic geometry. However, the polar-coordinate model has not been considered before, despite its popularity in the network science community. While Voronoi diagrams have the potential to advance this field, the model is geometrically not as approachable as other models, which impedes the development of geometric algorithms. In this paper, we present an algorithm for computing Voronoi diagrams natively in the polar-coordinate model of the hyperbolic plane. The approach is based on Fortune's sweep line algorithm for Euclidean Voronoi diagrams. We characterize the hyperbolic counterparts of the concepts it utilizes and introduce adaptations necessary to account for the differences. We implemented our algorithm and compared it with the corresponding CGAL implementation. While not being as numerically stable, our method has proven to be useful as a reference, which helped resolving fundamental issues in the implementation of the state-of-the-art method.
When the Curious Abandon Honesty: Federated Learning Is Not Private
Venue: arXiv cs.LG 2023
Year: 2023

In federated learning (FL), data does not leave personal devices when they are jointly training a machine learning model. Instead, these devices share gradients, parameters, or other model updates, with a central party (e.g., a company) coordinating the training. Because data never "leaves" personal devices, FL is often presented as privacy-preserving. Yet, recently it was shown that this protection is but a thin facade, as even a passive, honest-but-curious attacker observing gradients can reconstruct data of individual users contributing to the protocol. In this work, we show a novel data reconstruction attack which allows an active and dishonest central party to efficiently extract user data from the received gradients. While prior work on data reconstruction in FL relies on solving computationally expensive optimization problems or on making easily detectable modifications to the shared model's architecture or parameters, in our attack the central party makes inconspicuous changes to the shared model's weights before sending them out to the users. We call the modified weights of our attack trap weights. Our active attacker is able to recover user data perfectly, i.e., with zero error, even when this data stems from the same class. Recovery comes with near-zero costs: the attack requires no complex optimization objectives. Instead, our attacker exploits inherent data leakage from model gradients and simply amplifies this effect by maliciously altering the weights of the shared model through the trap weights. These specificities enable our attack to scale to fully-connected and convolutional deep neural networks trained with large mini-batches of data. For example, for the high-dimensional vision dataset ImageNet, we perfectly reconstruct more than 50% of the training data points from mini-batches as large as 100 data points.
Active Learning for Event Extraction with Memory-based Loss Prediction Model
Venue: arXiv cs.CL 2023
Year: 2023

Event extraction (EE) plays an important role in many industrial application scenarios, and high-quality EE methods require a large amount of manual annotation data to train supervised learning models. However, the cost of obtaining annotation data is very high, especially for annotation of domain events, which requires the participation of experts from corresponding domain. So we introduce active learning (AL) technology to reduce the cost of event annotation. But the existing AL methods have two main problems, which make them not well used for event extraction. Firstly, the existing pool-based selection strategies have limitations in terms of computational cost and sample validity. Secondly, the existing evaluation of sample importance lacks the use of local sample information. In this paper, we present a novel deep AL method for EE. We propose a batch-based selection strategy and a Memory-Based Loss Prediction model (MBLP) to select unlabeled samples efficiently. During the selection process, we use an internal-external sample loss ranking method to evaluate the sample importance by using local information. Finally, we propose a delayed training strategy to train the MBLP model. Extensive experiments are performed on three domain datasets, and our method outperforms other state-of-the-art methods.
HTMOT : Hierarchical Topic Modelling Over Time
Venue: arXiv cs.IR 2023
Year: 2023

Over the years, topic models have provided an efficient way of extracting insights from text. However, while many models have been proposed, none are able to model topic temporality and hierarchy jointly. Modelling time provide more precise topics by separating lexically close but temporally distinct topics while modelling hierarchy provides a more detailed view of the content of a document corpus. In this study, we therefore propose a novel method, HTMOT, to perform Hierarchical Topic Modelling Over Time. We train HTMOT using a new implementation of Gibbs sampling, which is more efficient. Specifically, we show that only applying time modelling to deep sub-topics provides a way to extract specific stories or events while high level topics extract larger themes in the corpus. Our results show that our training procedure is fast and can extract accurate high-level topics and temporally precise sub-topics. We measured our model's performance using the Word Intrusion task and outlined some limitations of this evaluation method, especially for hierarchical models. As a case study, we focused on the various developments in the space industry in 2020.
Universalizing Weak Supervision
Venue: arXiv cs.LG 2023
Year: 2023

Weak supervision (WS) frameworks are a popular way to bypass hand-labeling large datasets for training data-hungry models. These approaches synthesize multiple noisy but cheaply-acquired estimates of labels into a set of high-quality pseudolabels for downstream training. However, the synthesis technique is specific to a particular kind of label, such as binary labels or sequences, and each new label type requires manually designing a new synthesis algorithm. Instead, we propose a universal technique that enables weak supervision over any label type while still offering desirable properties, including practical flexibility, computational efficiency, and theoretical guarantees. We apply this technique to important problems previously not tackled by WS frameworks including learning to rank, regression, and learning in hyperbolic space. Theoretically, our synthesis approach produces a consistent estimators for learning some challenging but important generalizations of the exponential family model. Experimentally, we validate our framework and show improvement over baselines in diverse settings including real-world learning-to-rank and regression problems along with learning on hyperbolic manifolds.
What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods
Venue: arXiv cs.CV 2023
Year: 2023

A multitude of explainability methods and associated fidelity performance metrics have been proposed to help better understand how modern AI systems make decisions. However, much of the current work has remained theoretical -- without much consideration for the human end-user. In particular, it is not yet known (1) how useful current explainability methods are in practice for more real-world scenarios and (2) how well associated performance metrics accurately predict how much knowledge individual explanations contribute to a human end-user trying to understand the inner-workings of the system. To fill this gap, we conducted psychophysics experiments at scale to evaluate the ability of human participants to leverage representative attribution methods for understanding the behavior of different image classifiers representing three real-world scenarios: identifying bias in an AI system, characterizing the visual strategy it uses for tasks that are too difficult for an untrained non-expert human observer as well as understanding its failure cases. Our results demonstrate that the degree to which individual attribution methods help human participants better understand an AI system varied widely across these scenarios. This suggests a critical need for the field to move past quantitative improvements of current attribution methods towards the development of complementary approaches that provide qualitatively different sources of information to human end-users.
Global convergence of ResNets: From finite to infinite width using linear parameterization
Venue: arXiv cs.NE 2023
Year: 2023

Overparametrization is a key factor in the absence of convexity to explain global convergence of gradient descent (GD) for neural networks. Beside the well studied lazy regime, infinite width (mean field) analysis has been developed for shallow networks, using on convex optimization technics. To bridge the gap between the lazy and mean field regimes, we study Residual Networks (ResNets) in which the residual block has linear parametrization while still being nonlinear. Such ResNets admit both infinite depth and width limits, encoding residual blocks in a Reproducing Kernel Hilbert Space (RKHS). In this limit, we prove a local Polyak-Lojasiewicz inequality. Thus, every critical point is a global minimizer and a local convergence result of GD holds, retrieving the lazy regime. In contrast with other mean-field studies, it applies to both parametric and non-parametric cases under an expressivity condition on the residuals. Our analysis leads to a practical and quantified recipe: starting from a universal RKHS, Random Fourier Features are applied to obtain a finite dimensional parameterization satisfying with high-probability our expressivity condition.
CryptoEval: Evaluating the Risk of Cryptographic Misuses in Android Apps with Data-Flow Analysis
Venue: arXiv cs.CR 2023
Year: 2023

The misunderstanding and incorrect configurations of cryptographic primitives have exposed severe security vulnerabilities to attackers. Due to the pervasiveness and diversity of cryptographic misuses, a comprehensive and accurate understanding of how cryptographic misuses can undermine the security of an Android app is critical to the subsequent mitigation strategies but also challenging. Although various approaches have been proposed to detect cryptographic misuse in Android apps, studies have yet to focus on estimating the security risks of cryptographic misuse. To address this problem, we present an extensible framework for deciding the threat level of cryptographic misuse in Android apps. Firstly, we propose a general and unified specification for representing cryptographic misuses to make our framework extensible and develop adapters to unify the detection results of the state-of-the-art cryptographic misuse detectors, resulting in an adapter-based detection tool chain for a more comprehensive list of cryptographic misuses. Secondly, we employ a misuse-originating data-flow analysis to connect each cryptographic misuse to a set of data-flow sinks in an app, based on which we propose a quantitative data-flow-driven metric for assessing the overall risk of the app introduced by cryptographic misuses. To make the per-app assessment more useful for app vetting at the app-store level, we apply unsupervised learning to predict and classify the top risky threats to guide more efficient subsequent mitigation. In the experiments on an instantiated implementation of the framework, we evaluate the accuracy of our detection and the effect of data-flow-driven risk assessment of our framework. Our empirical study on over 40,000 apps and the analysis of popular apps reveal important security observations on the real threats of cryptographic misuse in Android apps.
Stochastic differential equations for limiting description of UCB rule for Gaussian multi-armed bandits
Venue: arXiv cs.LG 2023
Year: 2023

We consider the upper confidence bound strategy for Gaussian multi-armed bandits with known control horizon sizes $N$ and build its limiting description with a system of stochastic differential equations and ordinary differential equations. Rewards for the arms are assumed to have unknown expected values and known variances. A set of Monte-Carlo simulations was performed for the case of close distributions of rewards, when mean rewards differ by the magnitude of order $N^{-1/2}$, as it yields the highest normalized regret, to verify the validity of the obtained description. The minimal size of the control horizon when the normalized regret is not noticeably larger than maximum possible was estimated.
CLIP-Lite: Information Efficient Visual Representation Learning with Language Supervision
Venue: arXiv cs.CV 2023
Year: 2023

We propose CLIP-Lite, an information efficient method for visual representation learning by feature alignment with textual annotations. Compared to the previously proposed CLIP model, CLIP-Lite requires only one negative image-text sample pair for every positive image-text sample during the optimization of its contrastive learning objective. We accomplish this by taking advantage of an information efficient lower-bound to maximize the mutual information between the two input modalities. This allows CLIP-Lite to be trained with significantly reduced amounts of data and batch sizes while obtaining better performance than CLIP at the same scale. We evaluate CLIP-Lite by pretraining on the COCO-Captions dataset and testing transfer learning to other datasets. CLIP-Lite obtains a +14.0% mAP absolute gain in performance on Pascal VOC classification, and a +22.1% top-1 accuracy gain on ImageNet, while being comparable or superior to other, more complex, text-supervised models. CLIP-Lite is also superior to CLIP on image and text retrieval, zero-shot classification, and visual grounding. Finally, we show that CLIP-Lite can leverage language semantics to encourage bias-free visual representations that can be used in downstream tasks. Implementation: https://github.com/4m4n5/CLIP-Lite
Novel Algorithms for Efficient Mining of Connected Induced Subgraphs of a Given Cardinality
Venue: arXiv cs.DS 2023
Year: 2023

Mining subgraphs with interesting structural properties from networks (or graphs) is a computationally challenging task. In this paper, we propose two algorithms for enumerating all connected induced subgraphs of a given cardinality from networks (or connected undirected graphs in networks). The first algorithm is a variant of a previous well-known algorithm. The algorithm enumerates all connected induced subgraphs of cardinality $k$ in a bottom-up manner. The data structures that lead to unit time element checking and linear space are presented. Different from previous algorithms that either work in a bottom-up manner or a reverse search manner, an algorithm that enumerates all connected induced subgraphs of cardinality $k$ in a top-down manner is proposed. The correctness and complexity of the top-down algorithm are theoretically analyzed and proven. In the experiments, we evaluate the efficiency of the algorithms using a set of real-world networks from various fields. Experimental results show that the variant bottom-up algorithm outperforms the state-of-the-art algorithms for enumerating connected induced subgraphs of small cardinality, and the top-down algorithm can achieve an order of magnitude speedup over the state-of-the-art algorithms for enumerating connected induced subgraphs of large cardinality.
Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions
Venue: arXiv cs.LG 2023
Year: 2023

In many numerical simulations stochastic gradient descent (SGD) type optimization methods perform very effectively in the training of deep neural networks (DNNs) but till this day it remains an open problem of research to provide a mathematical convergence analysis which rigorously explains the success of SGD type optimization methods in the training of DNNs. In this work we study SGD type optimization methods in the training of fully-connected feedforward DNNs with rectified linear unit (ReLU) activation. We first establish general regularity properties for the risk functions and their generalized gradient functions appearing in the training of such DNNs and, thereafter, we investigate the plain vanilla SGD optimization method in the training of such DNNs under the assumption that the target function under consideration is a constant function. Specifically, we prove under the assumption that the learning rates (the step sizes of the SGD optimization method) are sufficiently small but not $L^1$-summable and under the assumption that the target function is a constant function that the expectation of the riskof the considered SGD process converges in the training of such DNNs to zero as the number of SGD steps increases to infinity.
One size does not fit all: Investigating strategies for differentially-private learning across NLP tasks
Venue: arXiv cs.CL 2023
Year: 2023

Preserving privacy in contemporary NLP models allows us to work with sensitive data, but unfortunately comes at a price. We know that stricter privacy guarantees in differentially-private stochastic gradient descent (DP-SGD) generally degrade model performance. However, previous research on the efficiency of DP-SGD in NLP is inconclusive or even counter-intuitive. In this short paper, we provide an extensive analysis of different privacy preserving strategies on seven downstream datasets in five different `typical' NLP tasks with varying complexity using modern neural models based on BERT and XtremeDistil architectures. We show that unlike standard non-private approaches to solving NLP tasks, where bigger is usually better, privacy-preserving strategies do not exhibit a winning pattern, and each task and privacy regime requires a special treatment to achieve adequate performance.