OpenRaiser/Intern-Atlas huggingface dataset:
Leveraging the Third Dimension in Contrastive Learning
Venue: arXiv cs.CV 2023
Year: 2023

Self-Supervised Learning (SSL) methods operate on unlabeled data to learn robust representations useful for downstream tasks. Most SSL methods rely on augmentations obtained by transforming the 2D image pixel map. These augmentations ignore the fact that biological vision takes place in an immersive three-dimensional, temporally contiguous environment, and that low-level biological vision relies heavily on depth cues. Using a signal provided by a pretrained state-of-the-art monocular RGB-to-depth model (the \emph{Depth Prediction Transformer}, Ranftl et al., 2021), we explore two distinct approaches to incorporating depth signals into the SSL framework. First, we evaluate contrastive learning using an RGB+depth input representation. Second, we use the depth signal to generate novel views from slightly different camera positions, thereby producing a 3D augmentation for contrastive learning. We evaluate these two approaches on three different SSL methods -- BYOL, SimSiam, and SwAV -- using ImageNette (10 class subset of ImageNet), ImageNet-100 and ImageNet-1k datasets. We find that both approaches to incorporating depth signals improve the robustness and generalization of the baseline SSL methods, though the first approach (with depth-channel concatenation) is superior. For instance, BYOL with the additional depth channel leads to an increase in downstream classification accuracy from 85.3\% to 88.0\% on ImageNette and 84.1\% to 87.0\% on ImageNet-C.
Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
Venue: arXiv cs.CL 2023
Year: 2023

We present the call for papers for the BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus. This shared task is intended for participants with an interest in small scale language modeling, human language acquisition, low-resource NLP, and cognitive modeling. In partnership with CoNLL and CMCL, we provide a platform for approaches to pretraining with a limited-size corpus sourced from data inspired by the input to children. The task has three tracks, two of which restrict the training data to pre-released datasets of 10M and 100M words and are dedicated to explorations of approaches such as architectural variations, self-supervised objectives, or curriculum learning. The final track only restricts the amount of text used, allowing innovation in the choice of the data, its domain, and even its modality (i.e., data from sources other than text is welcome). We will release a shared evaluation pipeline which scores models on a variety of benchmarks and tasks, including targeted syntactic evaluations and natural language understanding.
TrojanSAINT: Gate-Level Netlist Sampling-Based Inductive Learning for Hardware Trojan Detection
Venue: arXiv cs.CR 2023
Year: 2023

We propose TrojanSAINT, a graph neural network (GNN)-based hardware Trojan (HT) detection scheme working at the gate level. Unlike prior GNN-based art, TrojanSAINT enables both pre-/post-silicon HT detection. TrojanSAINT leverages a sampling-based GNN framework to detect and also localize HTs. For practical validation, TrojanSAINT achieves on average (oa) 78% true positive rate (TPR) and 85% true negative rate (TNR), respectively, on various TrustHub HT benchmarks. For best-case validation, TrojanSAINT even achieves 98% TPR and 96% TNR oa. TrojanSAINT outperforms related prior works and baseline classifiers. We release our source codes and result artifacts.
SOBER: Highly Parallel Bayesian Optimization and Bayesian Quadrature over Discrete and Mixed Spaces
Venue: arXiv cs.LG 2023
Year: 2023

Batch Bayesian optimisation and Bayesian quadrature have been shown to be sample-efficient methods of performing optimisation and quadrature where expensive-to-evaluate objective functions can be queried in parallel. However, current methods do not scale to large batch sizes -- a frequent desideratum in practice (e.g. drug discovery or simulation-based inference). We present a novel algorithm, SOBER, which permits scalable and diversified batch global optimisation and quadrature with arbitrary acquisition functions and kernels over discrete and mixed spaces. The key to our approach is to reformulate batch selection for global optimisation as a quadrature problem, which relaxes acquisition function maximisation (non-convex) to kernel recombination (convex). Bridging global optimisation and quadrature can efficiently solve both tasks by balancing the merits of exploitative Bayesian optimisation and explorative Bayesian quadrature. We show that SOBER outperforms 11 competitive baselines on 12 synthetic and diverse real-world tasks.
A Sequential Global Programming Approach for Two-scale Optimization of Homogenized Multiphysics Problems with Application to Biot Porous Media
Venue: arXiv cs.CE 2023
Year: 2023

We present a new approach and an algorithm for optimizing the material configuration and behaviour of a fluid saturated porous medium in a two-scale setting. The state problem is governed by the Biot model describing the fluid-structure interaction in homogenized poroelastic structures. However, the approach is widely applicable to multiphysics problems involving several macroscopic fields where homogenization provides the relationship between the microconfigurations and the macroscopic mathematical model. The optimization variables describe the local microstructure design by virtue of the pore shape which determines the effective medium properties - the material coefficients - computed by the homogenization method. The main idea of the numerical optimization strategy consists in a) employing a precomputed database of the material coefficients associated to the geometric parameters and b) applying the sequential global programming (SGP) method for solving the problem of macroscopically optimized distribution of material coefficients. Although there are similarities with the free material optimization (FMO) approach, only effective material coefficients are considered admissible, for which a well-defined set of corresponding configurable microstructures exist. Due to the flexibility of the SGP approach, different types of microstructures with fully independent parametrizations can easily be handled. The efficiency of the concept is demonstrated by a series of numerical experiments. We show that the SGP method can handle simultaneously multiple types of microstructures with nontrivial parametrizations using a considerably low and stable number of state problems to be solved.
The Produoidal Algebra of Process Decomposition
Venue: arXiv cs.LO 2023
Year: 2023

We introduce the normal produoidal category of monoidal contexts over an arbitrary monoidal category. In the same sense that a monoidal morphism represents a process, a monoidal context represents an incomplete process: a piece of a decomposition, possibly containing missing parts. We characterize monoidal contexts in terms of universal properties. In particular, symmetric monoidal contexts coincide with monoidal lenses, endowing them with a novel universal property. We apply this algebraic structure to the analysis of multi-party interaction protocols in arbitrary theories of processes.
A Learned Cache Eviction Framework with Minimal Overhead
Venue: arXiv cs.OS 2023
Year: 2023

Recent work shows the effectiveness of Machine Learning (ML) to reduce cache miss ratios by making better eviction decisions than heuristics. However, state-of-the-art ML caches require many predictions to make an eviction decision, making them impractical for high-throughput caching systems. This paper introduces Machine learning At the Tail (MAT), a framework to build efficient ML-based caching systems by integrating an ML module with a traditional cache system based on a heuristic algorithm. MAT treats the heuristic algorithm as a filter to receive high-quality samples to train an ML model and likely candidate objects for evictions. We evaluate MAT on 8 production workloads, spanning storage, in-memory caching, and CDNs. The simulation experiments show MAT reduces the number of costly ML predictions-per-eviction from 63 to 2, while achieving comparable miss ratios to the state-of-the-art ML cache system. We compare a MAT prototype system with an LRU-based caching system in the same setting and show that they achieve similar request rates.
Is TinyML Sustainable? Assessing the Environmental Impacts of Machine Learning on Microcontrollers
Venue: arXiv cs.LG 2023
Year: 2023

The sustained growth of carbon emissions and global waste elicits significant sustainability concerns for our environment's future. The growing Internet of Things (IoT) has the potential to exacerbate this issue. However, an emerging area known as Tiny Machine Learning (TinyML) has the opportunity to help address these environmental challenges through sustainable computing practices. TinyML, the deployment of machine learning (ML) algorithms onto low-cost, low-power microcontroller systems, enables on-device sensor analytics that unlocks numerous always-on ML applications. This article discusses both the potential of these TinyML applications to address critical sustainability challenges, as well as the environmental footprint of this emerging technology. Through a complete life cycle analysis (LCA), we find that TinyML systems present opportunities to offset their carbon emissions by enabling applications that reduce the emissions of other sectors. Nevertheless, when globally scaled, the carbon footprint of TinyML systems is not negligible, necessitating that designers factor in environmental impact when formulating new devices. Finally, we outline research directions to enable further sustainable contributions of TinyML.
Uplink Scheduling in Federated Learning: an Importance-Aware Approach via Graph Representation Learning
Venue: arXiv cs.NI 2023
Year: 2023

Federated Learning (FL) has emerged as a promising framework for distributed training of AI-based services, applications, and network procedures in 6G. One of the major challenges affecting the performance and efficiency of 6G wireless FL systems is the massive scheduling of user devices over resource-constrained channels. In this work, we argue that the uplink scheduling of FL client devices is a problem with a rich relational structure. To address this challenge, we propose a novel, energy-efficient, and importance-aware metric for client scheduling in FL applications by leveraging Unsupervised Graph Representation Learning (UGRL). Our proposed approach introduces a relational inductive bias in the scheduling process and does not require the collection of training feedback information from client devices, unlike state-of-the-art importance-aware mechanisms. We evaluate our proposed solution against baseline scheduling algorithms based on recently proposed metrics in the literature. Results show that, when considering scenarios of nodes exhibiting spatial relations, our approach can achieve an average gain of up to 10% in model accuracy and up to 17 times in energy efficiency compared to state-of-the-art importance-aware policies.
Multi-dimensional concept discovery (MCD): A unifying framework with completeness guarantees
Venue: arXiv cs.LG 2023
Year: 2023

The completeness axiom renders the explanation of a post-hoc XAI method only locally faithful to the model, i.e. for a single decision. For the trustworthy application of XAI, in particular for high-stake decisions, a more global model understanding is required. Recently, concept-based methods have been proposed, which are however not guaranteed to be bound to the actual model reasoning. To circumvent this problem, we propose Multi-dimensional Concept Discovery (MCD) as an extension of previous approaches that fulfills a completeness relation on the level of concepts. Our method starts from general linear subspaces as concepts and does neither require reinforcing concept interpretability nor re-training of model parts. We propose sparse subspace clustering to discover improved concepts and fully leverage the potential of multi-dimensional subspaces. MCD offers two complementary analysis tools for concepts in input space: (1) concept activation maps, that show where a concept is expressed within a sample, allowing for concept characterization through prototypical samples, and (2) concept relevance heatmaps, that decompose the model decision into concept contributions. Both tools together enable a detailed understanding of the model reasoning, which is guaranteed to relate to the model via a completeness relation. This paves the way towards more trustworthy concept-based XAI. We empirically demonstrate the superiority of MCD against more constrained concept definitions.
Training Full Spike Neural Networks via Auxiliary Accumulation Pathway
Venue: arXiv cs.NE 2023
Year: 2023

Due to the binary spike signals making converting the traditional high-power multiply-accumulation (MAC) into a low-power accumulation (AC) available, the brain-inspired Spiking Neural Networks (SNNs) are gaining more and more attention. However, the binary spike propagation of the Full-Spike Neural Networks (FSNN) with limited time steps is prone to significant information loss. To improve performance, several state-of-the-art SNN models trained from scratch inevitably bring many non-spike operations. The non-spike operations cause additional computational consumption and may not be deployed on some neuromorphic hardware where only spike operation is allowed. To train a large-scale FSNN with high performance, this paper proposes a novel Dual-Stream Training (DST) method which adds a detachable Auxiliary Accumulation Pathway (AAP) to the full spiking residual networks. The accumulation in AAP could compensate for the information loss during the forward and backward of full spike propagation, and facilitate the training of the FSNN. In the test phase, the AAP could be removed and only the FSNN remained. This not only keeps the lower energy consumption but also makes our model easy to deploy. Moreover, for some cases where the non-spike operations are available, the APP could also be retained in test inference and improve feature discrimination by introducing a little non-spike consumption. Extensive experiments on ImageNet, DVS Gesture, and CIFAR10-DVS datasets demonstrate the effectiveness of DST.
Mapi-Pro: An Energy Efficient Memory Mapping Technique for Intermittent Computing
Venue: arXiv cs.AR 2023
Year: 2023

Battery-less technology evolved to replace battery usage in space, deep mines, and other environments to reduce cost and pollution. Non-volatile memory (NVM) based processors were explored for saving the system state during a power failure. Such devices have a small SRAM and large non-volatile memory. To make the system energy efficient, we need to use SRAM efficiently. So we must select some portions of the application and map them to either SRAM or FRAM. This paper proposes an ILP-based memory mapping technique for Intermittently powered IoT devices. Our proposed technique gives an optimal mapping choice that reduces the system's Energy-Delay Product (EDP). We validated our system using a TI-based MSP430FR6989 and MSP430F5529 development boards. Our proposed memory configuration consumes 38.10% less EDP than the baseline configuration and 9.30% less EDP than the existing work under stable power. Our proposed configuration achieves 15.97% less EDP than the baseline configuration and 21.99% less EDP than the existing work under unstable power. This work supports intermittent computing and works efficiently during frequent power failures.
A New Symmetric Homomorphic Functional Encryption over a Hidden Ring for Polynomial Public Key Encapsulations
Venue: arXiv cs.CR 2023
Year: 2023

This paper proposes a new homomorphic functional encryption using modular multiplications over a hidden ring. Unlike traditional homomorphic encryption where users can only passively perform ciphertext addition or multiplication, the homomorphic functional encryption retains homomorphic addition and scalar multiplication properties, but also allows for the user's inputs through polynomial variables. The proposed homomorphic encryption can be applied to any polynomials over a finite field, with their coefficients considered as their privacy. We denote the polynomials before homomorphic encryption as plain polynomials and after homomorphic encryption as cipher polynomials. A cipher polynomial can be evaluated with variables from the finite field, GF(p), by calculating the monomials of variables modulo a prime p. These properties allow functional homomorphic encryption to be used for public key encryption of certain asymmetric cryptosystems to hide the structure of its central map construction. We propose a new variant of MPKC with homomorphic encryption of its public key. We propose to use a single plaintext vector and a noise vector of multiple variables to be associated with the central map, in place of the secret plaintext vector to be encrypted in MPKC. We call this variant of encrypted MPKC, a Homomorphic Polynomial Public Key algorithm or HPPK algorithm. The HPPK algorithm holds the property of indistinguishability under the chosen-plaintext attacks or IND-CPA. The overall classical complexity to crack the HPPK algorithm is exponential in the size of the prime field GF(p). We briefly report on benchmarking performance results using the SUPERCOP toolkit. Benchmarking results demonstrate that HPPK offers rather fast performance, which is comparable and in some cases outperforms the NIST PQC finalists for key generation, encryption, and decryption.
Improved knowledge distillation by utilizing backward pass knowledge in neural networks
Venue: arXiv cs.LG 2023
Year: 2023

Knowledge distillation (KD) is one of the prominent techniques for model compression. In this method, the knowledge of a large network (teacher) is distilled into a model (student) with usually significantly fewer parameters. KD tries to better-match the output of the student model to that of the teacher model based on the knowledge extracts from the forward pass of the teacher network. Although conventional KD is effective for matching the two networks over the given data points, there is no guarantee that these models would match in other areas for which we do not have enough training samples. In this work, we address that problem by generating new auxiliary training samples based on extracting knowledge from the backward pass of the teacher in the areas where the student diverges greatly from the teacher. We compute the difference between the teacher and the student and generate new data samples that maximize the divergence. This is done by perturbing data samples in the direction of the gradient of the difference between the student and the teacher. Augmenting the training set by adding this auxiliary improves the performance of KD significantly and leads to a closer match between the student and the teacher. Using this approach, when data samples come from a discrete domain, such as applications of natural language processing (NLP) and language understanding, is not trivial. However, we show how this technique can be used successfully in such applications. We evaluated the performance of our method on various tasks in computer vision and NLP domains and got promising results.
Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases
Venue: arXiv cs.CL 2023
Year: 2023

Improving the deployment efficiency of transformer-based language models has been challenging given their high computation and memory cost. While INT8 quantization has recently been shown to be effective in reducing both the memory cost and latency while preserving model accuracy, it remains unclear whether we can leverage INT4 (which doubles peak hardware throughput) to achieve further latency improvement. In this study, we explore the feasibility of employing INT4 weight and activation (W4A4) quantization for language models. Our findings indicate that W4A4 quantization introduces no to negligible accuracy degradation for encoder-only and encoder-decoder models, but causes a significant accuracy drop for decoder-only models. To materialize the performance gain using W4A4, we develop a highly optimized end-to-end W4A4 encoder inference pipeline supporting different quantization strategies. Our INT4 pipeline is $8.5\times$ faster for latency-oriented scenarios and up to $3\times$ for throughput-oriented scenarios compared to the inference of FP16, and improves the SOTA BERT INT8 performance from FasterTransformer by up to $1.7\times$. We provide insights into the failure cases when applying W4A4 to decoder-only models, and further explore the compatibility of INT4 quantization with other compression methods, like pruning and layer reduction.
Epsilon-Identifiability of Causal Quantities
Venue: arXiv cs.AI 2023
Year: 2023

Identifying the effects of causes and causes of effects is vital in virtually every scientific field. Often, however, the needed probabilities may not be fully identifiable from the data sources available. This paper shows how partial identifiability is still possible for several probabilities of causation. We term this epsilon-identifiability and demonstrate its usefulness in cases where the behavior of certain subpopulations can be restricted to within some narrow bounds. In particular, we show how unidentifiable causal effects and counterfactual probabilities can be narrowly bounded when such allowances are made. Often those allowances are easily measured and reasonably assumed. Finally, epsilon-identifiability is applied to the unit selection problem.
Automated Arrangements of Multi-Part Music for Sets of Monophonic Instruments
Venue: arXiv cs.SD 2023
Year: 2023

Arranging music for a different set of instruments that it was originally written for is traditionally a tedious and time-consuming process, performed by experts with intricate knowledge of the specific instruments and involving significant experimentation. In this paper we study the problem of automating music arrangements for music pieces written for monophonic instruments or voices. We designed and implemented an algorithm that can always produce a music arrangement when feasible by transposing the music piece to a different scale, permuting the assigned parts to instruments/voices, and transposing individual parts by one or more octaves. We also published open source software written in Python that processes MusicXML files and allows musicians to experiment with music arrangements. It is our hope that our software can serve as a platform for future extensions that will include music reductions and inclusion of polyphonic instruments.
Non-Hermitian Physics-Inspired Voltage-Controlled Oscillators with Resistive Tuning
Venue: arXiv cs.ET 2023
Year: 2023

This paper presents a non-Hermitian physics-inspired voltage-controlled oscillator (VCO) topology, which is termed parity-time-symmetric topology. The VCO consists of two coupled inductor-capacitor (LC) cores with a balanced gain and loss profile. Due to the interplay between the gain/loss and their coupling, an extra degree of freedom is enabled via resistive tuning, which can enhance the frequency tuning range (FTR) beyond the bounds of conventional capacitive or inductive tuning. A silicon prototype is implemented in a standard 130 nm bulk CMOS process with a core area of 0.15mm^2. Experimental results show that it achieves a 3.1x FTR improvement and 30% phase noise reduction of the baseline VCO with the same amount of capacitive tuning ability.