OpenRaiser/Intern-Atlas huggingface dataset:
You Only Linearize Once: Tangents Transpose to Gradients
Venue: arXiv cs.PL 2022
Year: 2022

Automatic differentiation (AD) is conventionally understood as a family of distinct algorithms, rooted in two "modes" -- forward and reverse -- which are typically presented (and implemented) separately. Can there be only one? Following up on the AD systems developed in the JAX and Dex projects, we formalize a decomposition of reverse-mode AD into (i) forward-mode AD followed by (ii) unzipping the linear and non-linear parts and then (iii) transposition of the linear part. To that end, we define a (substructurally) linear type system that can prove a class of functions are (algebraically) linear. Our main results are that forward-mode AD produces such linear functions, and that we can unzip and transpose any such linear function, conserving cost, size, and linearity. Composing these three transformations recovers reverse-mode AD. This decomposition also sheds light on checkpointing, which emerges naturally from a free choice in unzipping `let` expressions. As a corollary, checkpointing techniques are applicable to general-purpose partial evaluation, not just AD. We hope that our formalization will lead to a deeper understanding of automatic differentiation and that it will simplify implementations, by separating the concerns of differentiation proper from the concerns of gaining efficiency (namely, separating the derivative computation from the act of running it backward).
Balancing Fairness and Accuracy in Sentiment Detection using Multiple Black Box Models
Venue: arXiv cs.LG 2022
Year: 2022

Sentiment detection is an important building block for multiple information retrieval tasks such as product recommendation, cyberbullying detection, and misinformation detection. Unsurprisingly, multiple commercial APIs, each with different levels of accuracy and fairness, are now available for sentiment detection. While combining inputs from multiple modalities or black-box models for increasing accuracy is commonly studied in multimedia computing literature, there has been little work on combining different modalities for increasing fairness of the resulting decision. In this work, we audit multiple commercial sentiment detection APIs for the gender bias in two actor news headlines settings and report on the level of bias observed. Next, we propose a "Flexible Fair Regression" approach, which ensures satisfactory accuracy and fairness by jointly learning from multiple black-box models. The results pave way for fair yet accurate sentiment detectors for multiple applications.
Noncooperative Herding With Control Barrier Functions: Theory and Experiments
Venue: arXiv cs.RO 2022
Year: 2022

In this paper, we consider the problem of protecting a high-value unit from inadvertent attack by a group of agents using defending robots. Specifically, we develop a control strategy for the defending agents that we call "dog robots" to prevent a flock of "sheep agents" from breaching a protected zone. We take recourse to control barrier functions to pose this problem and exploit the interaction dynamics between the sheep and dogs to find dogs' velocities that result in the sheep getting repelled from the zone. We solve a QP reactively that incorporates the defending constraints to compute the desired velocities for all dogs. Owing to this, our proposed framework is composable \textit{i.e.} it allows for simultaneous inclusion of multiple protected zones in the constraints on dog robots' velocities. We provide a theoretical proof of feasibility of our strategy for the one dog/one sheep case. Additionally, we provide empirical results of two dogs defending the protected zone from upto ten sheep averaged over a hundred simulations and report high success rates. We also demonstrate this algorithm experimentally on non-holonomic robots. Videos of these results are available at https://tinyurl.com/4dj2kjwx.
Long-Range ICN for the IoT: Exploring a LoRa System Design
Venue: arXiv cs.NI 2022
Year: 2022

This paper presents LoRa-ICN, a comprehensive IoT networking system based on a common long-range communication layer (LoRa) combined with Information-Centric Networking (ICN) principles. We have replaced the LoRaWAN MAC layer with an IEEE 802.15.4 Deterministic and Synchronous Multi-Channel Extension (DSME). This multifaceted MAC layer allows for different mappings of ICN message semantics, which we explore to enable new LoRa cenarios. We designed LoRa-ICN from the ground-up to improve reliability and to reduce dependency on centralized components in LoRa IoT scenarios. We have implemented a feature-complete prototype in a common network simulator to validate our approach. Our results show design trade-offs of different mapping alternatives in terms of robustness and efficiency.
Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps
Venue: arXiv cs.LG 2022
Year: 2022

Transformer-based language models significantly advanced the state-of-the-art in many linguistic tasks. As this revolution continues, the ability to explain model predictions has become a major area of interest for the NLP community. In this work, we present Gradient Self-Attention Maps (Grad-SAM) - a novel gradient-based method that analyzes self-attention units and identifies the input elements that explain the model's prediction the best. Extensive evaluations on various benchmarks show that Grad-SAM obtains significant improvements over state-of-the-art alternatives.
Blockchain-Oriented Software Variant Forks: A Preliminary Study
Venue: arXiv cs.SE 2022
Year: 2022

In collaborative social development platforms such as GitHub, forking a repository is a common activity. A variant fork wants to split the development from the original repository and grow towards a different direction. In this preliminary exploratory research, we analyze the possible reasons for creating a variant fork in blockchain-oriented software. By collecting repositories in GitHub, we created a dataset with repositories and their variants, from which we manually analyzed 86 variants. Based on the variants we studied, the main reason to create a variant in blockchain-oriented software is to support a different blockchain platform (65%).
The SCORE normalization, especially for highly heterogeneous network and text data
Venue: arXiv cs.SI 2022
Year: 2022

SCORE was introduced as a spectral approach to network community detection. Since many networks have severe degree heterogeneity, the ordinary spectral clustering (OSC) approach to community detection may perform unsatisfactorily. SCORE alleviates the effect of degree heterogeneity by introducing a new normalization idea in the spectral domain and makes OSC more effective. SCORE is easy to use and computationally fast. It adapts easily to new directions and sees an increasing interest in practice. In this paper, we review the basics of SCORE, the adaption of SCORE to network mixed membership estimation and topic modeling, and the application of SCORE in real data, including two datasets on the publications of statisticians. We also review the theoretical 'ideology' underlying SCORE. We show that in the spectral domain, SCORE converts a simplicial cone to a simplex, and provides a simple and direct link between the simplex and network memberships. SCORE attains an exponential rate and a sharp phase transition in community detection, and achieves optimal rates in mixed membership estimation and topic modeling.
Competitive Physics Informed Networks
Venue: arXiv cs.LG 2022
Year: 2022

Neural networks can be trained to solve partial differential equations (PDEs) by using the PDE residual as the loss function. This strategy is called "physics-informed neural networks" (PINNs), but it currently cannot produce high-accuracy solutions, typically attaining about $0.1\%$ relative error. We present an adversarial approach that overcomes this limitation, which we call competitive PINNs (CPINNs). CPINNs train a discriminator that is rewarded for predicting mistakes the PINN makes. The discriminator and PINN participate in a zero-sum game with the exact PDE solution as an optimal strategy. This approach avoids squaring the large condition numbers of PDE discretizations, which is the likely reason for failures of previous attempts to decrease PINN errors even on benign problems. Numerical experiments on a Poisson problem show that CPINNs achieve errors four orders of magnitude smaller than the best-performing PINN. We observe relative errors on the order of single-precision accuracy, consistently decreasing with each epoch. To the authors' knowledge, this is the first time this level of accuracy and convergence behavior has been achieved. Additional experiments on the nonlinear Schrödinger, Burgers', and Allen-Cahn equation show that the benefits of CPINNs are not limited to linear problems.
RealNet: Combining Optimized Object Detection with Information Fusion Depth Estimation Co-Design Method on IoT
Venue: arXiv cs.CV 2022
Year: 2022

Depth Estimation and Object Detection Recognition play an important role in autonomous driving technology under the guidance of deep learning artificial intelligence. We propose a hybrid structure called RealNet: a co-design method combining the model-streamlined recognition algorithm, the depth estimation algorithm with information fusion, and deploying them on the Jetson-Nano for unmanned vehicles with monocular vision sensors. We use ROS for experiment. The method proposed in this paper is suitable for mobile platforms with high real-time request. Innovation of our method is using information fusion to compensate the problem of insufficient frame rate of output image, and improve the robustness of target detection and depth estimation under monocular vision.Object Detection is based on YOLO-v5. We have simplified the network structure of its DarkNet53 and realized a prediction speed up to 0.01s. Depth Estimation is based on the VNL Depth Estimation, which considers multiple geometric constraints in 3D global space. It calculates the loss function by calculating the deviation of the virtual normal vector VN and the label, which can obtain deeper depth information. We use PnP fusion algorithm to solve the problem of insufficient frame rate of depth map output. It solves the motion estimation depth from three-dimensional target to two-dimensional point based on corner feature matching, which is faster than VNL calculation. We interpolate VNL output and PnP output to achieve information fusion. Experiments show that this can effectively eliminate the jitter of depth information and improve robustness. At the control end, this method combines the results of target detection and depth estimation to calculate the target position, and uses a pure tracking control algorithm to track it.
Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training
Venue: arXiv cs.CL 2022
Year: 2022

Recent studies on the lottery ticket hypothesis (LTH) show that pre-trained language models (PLMs) like BERT contain matching subnetworks that have similar transfer learning performance as the original PLM. These subnetworks are found using magnitude-based pruning. In this paper, we find that the BERT subnetworks have even more potential than these studies have shown. Firstly, we discover that the success of magnitude pruning can be attributed to the preserved pre-training performance, which correlates with the downstream transferability. Inspired by this, we propose to directly optimize the subnetwork structure towards the pre-training objectives, which can better preserve the pre-training performance. Specifically, we train binary masks over model weights on the pre-training tasks, with the aim of preserving the universal transferability of the subnetwork, which is agnostic to any specific downstream tasks. We then fine-tune the subnetworks on the GLUE benchmark and the SQuAD dataset. The results show that, compared with magnitude pruning, mask training can effectively find BERT subnetworks with improved overall performance on downstream tasks. Moreover, our method is also more efficient in searching subnetworks and more advantageous when fine-tuning within a certain range of data scarcity. Our code is available at https://github.com/llyx97/TAMT.
The Generalized Degrees-of-Freedom of the Asymmetric Interference Channel with Delayed CSIT
Venue: arXiv cs.IT 2022
Year: 2022

In this paper, we investigate the generalized degrees-of-freedom (GDoF) of the asymmetric interference channel with delayed channel state information at the transmitter (CSIT), where each transmitter has two antennas, each receiver has one antenna, and the strength for each interfering link can vary. The optimal sum-GDoF is characterized by matched converse and achievability proof. Through our results, we also reveal that in our antenna setting, the symmetric GDoF lower bound in [Mohanty et. al, TIT 2019] can be elevated, and the symmetric GDoF upper bound in [Mohanty et. al, TIT 2019] is tight in fact.
On Finite-Time Mutual Information
Venue: arXiv cs.IT 2022
Year: 2022

Shannon-Hartley theorem can accurately calculate the channel capacity when the signal observation time is infinite. However, the calculation of finite-time mutual information, which remains unknown, is essential for guiding the design of practical communication systems. In this paper, we investigate the mutual information between two correlated Gaussian processes within a finite-time observation window. We first derive the finite-time mutual information by providing a limit expression. Then we numerically compute the mutual information within a single finite-time window. We reveal that the number of bits transmitted per second within the finite-time window can exceed the mutual information averaged over the entire time axis, which is called the exceed-average phenomenon. Furthermore, we derive a finite-time mutual information formula under a typical signal autocorrelation case by utilizing the Mercer expansion of trace class operators, and reveal the connection between the finite-time mutual information problem and the operator theory. Finally, we analytically prove the existence of the exceed-average phenomenon in this typical case, and demonstrate its compatibility with the Shannon capacity.
Dictionary Attacks on Speaker Verification
Venue: arXiv cs.SD 2022
Year: 2022

In this paper, we propose dictionary attacks against speaker verification - a novel attack vector that aims to match a large fraction of speaker population by chance. We introduce a generic formulation of the attack that can be used with various speech representations and threat models. The attacker uses adversarial optimization to maximize raw similarity of speaker embeddings between a seed speech sample and a proxy population. The resulting master voice successfully matches a non-trivial fraction of people in an unknown population. Adversarial waveforms obtained with our approach can match on average 69% of females and 38% of males enrolled in the target system at a strict decision threshold calibrated to yield false alarm rate of 1%. By using the attack with a black-box voice cloning system, we obtain master voices that are effective in the most challenging conditions and transferable between speaker encoders. We also show that, combined with multiple attempts, this attack opens even more to serious issues on the security of these systems.
Multimedia Services Placement Algorithm for Cloud-Fog Hierarchical Environments
Venue: arXiv cs.PF 2022
Year: 2022

With the rapid development of mobile communication, multimedia services have experienced explosive growth in the last few years. The high quantity of mobile users, both consuming and producing these services to and from the Cloud Computing (CC), can outpace the available bandwidth capacity. Fog Computing (FG) presents itself as a solution to improve on this and other issues. With a reduction in network latency, real-time applications benefit from improved response time and greater overall user experience. Taking this into account, the main goal of this work is threefold. Firstly, it is proposed a method to build an environment based on Cloud-Fog Computing (CFC). Secondly, it is designed two models based on Autoregressive Integrated Moving Average (ARIMA) and Long Short-Term Memory (LSTM). The goal is to predict demand and reserve the nodes' storage capacity to improve the positioning of multimedia services. Later, an algorithm for the multimedia service placement problem which is aware of data traffic prediction is proposed. The goal is to select the minimum number of nodes, considering their hardware capacities for providing multimedia services in such a way that the latency for servicing all the demands is minimized. An evaluation with actual data showed that the proposed algorithm selects the nodes closer to the user to meet their demands. This improves the services delivered to end-users and enhances the deployed network to mitigate provider costs. Moreover, reduce the demand to Cloud allowing turning off servers in the data center not to waste energy
DRT: A Lightweight Single Image Deraining Recursive Transformer
Venue: arXiv cs.CV 2022
Year: 2022

Over parameterization is a common technique in deep learning to help models learn and generalize sufficiently to the given task; nonetheless, this often leads to enormous network structures and consumes considerable computing resources during training. Recent powerful transformer-based deep learning models on vision tasks usually have heavy parameters and bear training difficulty. However, many dense-prediction low-level computer vision tasks, such as rain streak removing, often need to be executed on devices with limited computing power and memory in practice. Hence, we introduce a recursive local window-based self-attention structure with residual connections and propose deraining a recursive transformer (DRT), which enjoys the superiority of the transformer but requires a small amount of computing resources. In particular, through recursive architecture, our proposed model uses only 1.3% of the number of parameters of the current best performing model in deraining while exceeding the state-of-the-art methods on the Rain100L benchmark by at least 0.33 dB. Ablation studies also investigate the impact of recursions on derain outcomes. Moreover, since the model contains no deliberate design for deraining, it can also be applied to other image restoration tasks. Our experiment shows that it can achieve competitive results on desnowing. The source code and pretrained model can be found at https://github.com/YC-Liang/DRT.
Offline Retrieval Evaluation Without Evaluation Metrics
Venue: arXiv cs.IR 2022
Year: 2022

Offline evaluation of information retrieval and recommendation has traditionally focused on distilling the quality of a ranking into a scalar metric such as average precision or normalized discounted cumulative gain. We can use this metric to compare the performance of multiple systems for the same request. Although evaluation metrics provide a convenient summary of system performance, they also collapse subtle differences across users into a single number and can carry assumptions about user behavior and utility not supported across retrieval scenarios. We propose recall-paired preference (RPP), a metric-free evaluation method based on directly computing a preference between ranked lists. RPP simulates multiple user subpopulations per query and compares systems across these pseudo-populations. Our results across multiple search and recommendation tasks demonstrate that RPP substantially improves discriminative power while correlating well with existing metrics and being equally robust to incomplete data.
ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference
Venue: arXiv cs.CL 2022
Year: 2022

State-of-the-art neural models typically encode document-query pairs using cross-attention for re-ranking. To this end, models generally utilize an encoder-only (like BERT) paradigm or an encoder-decoder (like T5) approach. These paradigms, however, are not without flaws, i.e., running the model on all query-document pairs at inference-time incurs a significant computational cost. This paper proposes a new training and inference paradigm for re-ranking. We propose to finetune a pretrained encoder-decoder model using in the form of document to query generation. Subsequently, we show that this encoder-decoder architecture can be decomposed into a decoder-only language model during inference. This results in significant inference time speedups since the decoder-only architecture only needs to learn to interpret static encoder embeddings during inference. Our experiments show that this new paradigm achieves results that are comparable to the more expensive cross-attention ranking approaches while being up to 6.8X faster. We believe this work paves the way for more efficient neural rankers that leverage large pretrained models.
Error Performance Analysis of Multi-user Detection in Uplink-NOMA with Adaptive $\mathcal{M}$-QAM
Venue: arXiv cs.IT 2022
Year: 2022

This work provides a generalized performance analysis for the multi-user uplink-NOMA system with adaptive square quadrature amplitude modulation (QAM) over Rayleigh fading channels. Motivated by the massive IoT connections and unavailability of orthogonal resources for each node, we consider a multi-access scheme where multi-users with single-antenna transmit data to a multiple-antenna base station through the same resource block. By taking advantage of combining diversity paths with the proposed joint maximum-likelihood detector (JMLD), a closed form expression for the upper bound of bit error rate (BER) is obtained. Despite the number of users or the order of modulation, the analytical results endorsed via computer simulations reveal the ability of the MRC-JMLD detector to discard the error floor completely. Moreover, the simulation results show that the MRC-JMLD surpasses its counterparts significantly and ensures a full diversity order.
Automatic Datapath Optimization using E-Graphs
Venue: arXiv cs.AR 2022
Year: 2022

Manual optimization of Register Transfer Level (RTL) datapath is commonplace in industry but holds back development as it can be very time consuming. We utilize the fact that a complex transformation of one RTL into another equivalent RTL can be broken down into a sequence of smaller, localized transformations. By representing RTL as a graph and deploying modern graph rewriting techniques we can automate the circuit design space exploration, allowing us to discover functionally equivalent but optimized architectures. We demonstrate that modern rewriting frameworks can adequately capture a wide variety of complex optimizations performed by human designers on bit-vector manipulating code, including significant error-prone subtleties regarding the validity of transformations under complex interactions of bitwidths. The proposed automated optimization approach is able to reproduce the results of typical industrial manual optimization, resulting in a reduction in circuit area by up to 71%. Not only does our tool discover optimized RTL, but also correctly identifies that the optimal architecture to implement a given arithmetic expression can depend on the width of the operands, thus producing a library of optimized designs rather than the single design point typically generated by manual optimization. In addition, we demonstrate that prior academic work on maximally exploiting carry-save representation and on multiple constant multiplication are both generalized and extended, falling out as special cases of this paper.