Miracle-dz/UniFAR-FTUs huggingface dataset:
Boosting in the Presence of Massart Noise

We study the problem of boosting the accuracy of a weak learner in the (distribution-independent) PAC model with Massart noise. In the Massart noise model, the label of each example x is independently misclassiﬁed with probability η ( x ) ≤ η , where η < 1 / 2. The Massart model lies between the random classiﬁcation noise model and the agnostic model. Our main positive result is the ﬁrst computationally eﬃcient boosting algorithm in the presence of Massart noise that achieves misclassiﬁcation error arbitrarily close to η . Prior to our work, no non-trivial booster was known in this setting. Moreover, we show that this error upper bound is best possible for polynomial-time black-box boosters, under standard cryptographic assumptions. Our upper and lower bounds characterize the complexity of boosting in the distribution-independent PAC model with Massart noise. As a simple application of our positive result, we give the ﬁrst eﬃcient Massart learner for unions of high-dimensional rectangles.
A General Purpose Transpiler for Fully Homomorphic Encryption

Fully homomorphic encryption (FHE) is an encryption scheme which enables computation on encrypted data without revealing the underlying data. While there have been many advances in the field of FHE, developing programs using FHE still requires expertise in cryptography. In this white paper, we present a fully homomorphic encryption transpiler that allows developers to convert high-level code (e.g., C++) that works on unencrypted data into high-level code that operates on encrypted data. Thus, our transpiler makes transformations possible on encrypted data. Our transpiler builds on Google's open-source XLS SDK (https://github.com/google/xls) and uses an off-the-shelf FHE library, TFHE (https://tfhe.github.io/tfhe/), to perform low-level FHE operations. The transpiler design is modular, which means the underlying FHE library as well as the high-level input and output languages can vary. This modularity will help accelerate FHE research by providing an easy way to compare arbitrary programs in different FHE schemes side-by-side. We hope this lays the groundwork for eventual easy adoption of FHE by software developers. As a proof-of-concept, we are releasing an experimental transpiler (https://github.com/google/fully-homomorphic-encryption/tree/main/transpiler) as open-source software.
KL Guided Domain Adaptation

Domain adaptation is an important problem and often needed for real-world applications. In this problem, instead of i.i. d. training and testing datapoints, we assume that the source (training) data and the target (testing) data have different distributions. With that setting, the empirical risk minimization training procedure often does not perform well, since it does not account for the change in the distribution. A common approach in the domain adaptation literature is to learn a representation of the input that has the same (marginal) distribution over the source and the target domain. However, these approaches often require additional networks and/or optimizing an adversarial (minimax) objective, which can be very expensive or unstable in practice. To improve upon these marginal alignment techniques, in this paper, we first derive a generalization bound for the target loss based on the training loss and the reverse Kullback-Leibler (KL) divergence between the source and the target representation distributions. Based on this bound, we derive an algorithm that minimizes the KL term to obtain a better generalization to the target domain. We show that with a probabilistic representation network, the KL term can be estimated efficiently via minibatch samples without any additional network or a minimax objective. This leads to a theoretically sound alignment method which is also very efficient and stable in practice. Experimental results also suggest that our method outperforms other representation-alignment approaches.
Photoinduced Lattice Instability in SnSe

Yijing Huang, 2, 3 Shan Yang, Samuel Teitelbaum, 3 Gilberto De la Peña, 3 Takahiro Sato, Matthieu Chollet, Diling Zhu, Jennifer L. Niedziela, 7 Dipanshu Bansal, Andrew F. May, Aaron M. Lindenberg, 3, 8 Olivier Delaire, 9, 10 David A. Reis, 2, 3, 11 and Mariano Trigo 3 Stanford Institute for Materials and Energy Sciences, SLAC National Accelerator Laboratory, Menlo Park, California 94025, USA Department of Applied Physics, Stanford University, Stanford, California 94305, USA PULSE Institute of Ultrafast Energy Science, SLAC National Accelerator Laboratory, Menlo Park, California 94025, USA Department of Mechanical Engineering and Materials Science, Duke University, Durham, North Carolina 27708, USA Linac Coherent Light Source, SLAC National Accelerator Laboratory, Menlo Park, California 94025, USA Department of Mechanical Engineering and Materials Science, Duke University, Durham, North Carolina 27708, USA Materials Science and Technology Division, Oak Ridge National Laboratory, Oak Ridge, Tennessee 37831, USA Department of Materials Science and Engineering, Stanford University, Stanford, CA 94305, USA Department of Physics, Duke University, Durham, North Carolina 27708, USA Department of Chemistry, Duke University, Durham, North Carolina 27708, USA Department of Photon Science, Stanford University, Stanford, California 94305, USA
Direction is what you need: Improving Word Embedding Compression in Large Language Models

The adoption of Transformer-based models in natural language processing (NLP) has led to great success using a massive number of parameters. However, due to deployment constraints in edge devices, there has been a rising interest in the compression of these models to improve their inference time and memory footprint. This paper presents a novel loss objective to compress token embeddings in the Transformer-based models by leveraging an AutoEncoder architecture. More specifically, we emphasize the importance of the direction of compressed embeddings with respect to original uncompressed embeddings. The proposed method is task-agnostic and does not require further language modeling pre-training. Our method significantly outperforms the commonly used SVD-based matrix-factorization approach in terms of initial language model Perplexity. Moreover, we evaluate our proposed approach over SQuAD v1.1 dataset and several downstream tasks from the GLUE benchmark, where we also outperform the baseline in most scenarios. Our code is public.
Theory of unidirectional magnetoresistance and nonlinear Hall effect

We study the unidirectional magnetoresistance (UMR) and the nonlinear Hall effect (NLHE) in the ferromagnetic Rashba model. For this purpose we derive expressions to describe the response of the electric current quadratic in the applied electric field. We compare two different formalisms, namely the standard Keldysh nonequilibrium formalism and the Moyal–Keldysh formalism, to derive the nonlinear conductivities of UMR and NLHE. We find that both formalisms lead to identical numerical results when applied to the ferromagnetic Rashba model. The UMR and the NLHE nonlinear conductivities tend to be comparable in magnitude according to our calculations. Additionally, their dependencies on the Rashba parameter and on the quasiparticle broadening are similar. The nonlinear zero-frequency response considered here is several orders of magnitude higher than the one at optical frequencies that describes the photocurrent generation in the ferromagnetic Rashba model. Additionally, we compare our Keldysh nonequilibrium expression in the independent-particle approximation to literature expressions of the UMR that have been obtained within the constant relaxation time approximation of the Boltzmann formalism. We find that both formalisms converge to the same analytical formula in the limit of infinite relaxation time. However, remarkably, we find that the Boltzmann result does not correspond to the intraband term of the Keldysh expression. Instead, the Boltzmann result corresponds to the sum of the intraband term and an interband term that can be brought into the form of an effective intraband term due to the f-sum rule.
Simon Says: Evaluating and Mitigating Bias in Pruned Neural Networks with Knowledge Distillation

In recent years the ubiquitous deployment of AI has posed great concerns in regards to algorithmic bias, discrimination, and fairness. Compared to traditional forms of bias or discrimination caused by humans, algorithmic bias generated by AI is more abstract and unintuitive therefore more difficult to explain and mitigate. A clear gap exists in the current literature on evaluating and mitigating bias in pruned neural networks. In this work, we strive to tackle the challenging issues of evaluating, mitigating, and explaining induced bias in pruned neural networks. Our paper makes three contributions. First, we propose two simple yet effective metrics, Combined Error Variance (CEV) and Symmetric Distance Error (SDE), to quantitatively evaluate the induced bias prevention quality of pruned models. Second, we demonstrate that knowledge distillation can mitigate induced bias in pruned neural networks, even with unbalanced datasets. Third, we reveal that model similarity has strong correlations with pruning induced bias, which provides a powerful method to explain why bias occurs in pruned neural networks. Our code is available at https://github.com/codestar12/pruning-distilation-bias
BEiT: BERT Pre-Training of Image Transformers

We introduce a self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation from Image Transformers. Following BERT developed in the natural language processing area, we propose a masked image modeling task to pretrain vision Transformers. Specifically, each image has two views in our pre-training, i.e, image patches (such as 16x16 pixels), and visual tokens (i.e., discrete tokens). We first"tokenize"the original image into visual tokens. Then we randomly mask some image patches and fed them into the backbone Transformer. The pre-training objective is to recover the original visual tokens based on the corrupted image patches. After pre-training BEiT, we directly fine-tune the model parameters on downstream tasks by appending task layers upon the pretrained encoder. Experimental results on image classification and semantic segmentation show that our model achieves competitive results with previous pre-training methods. For example, base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K, significantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains 86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%). The code and pretrained models are available at https://aka.ms/beit.
Adaptive normalization for IPW estimation

Abstract Inverse probability weighting (IPW) is a general tool in survey sampling and causal inference, used in both Horvitz–Thompson estimators, which normalize by the sample size, and Hájek/self-normalized estimators, which normalize by the sum of the inverse probability weights. In this work, we study a family of IPW estimators, first proposed by Trotter and Tukey in the context of Monte Carlo problems, that are normalized by an affine combination of the sample size and a sum of inverse weights. We show how selecting an estimator from this family in a data-dependent way to minimize asymptotic variance leads to an iterative procedure that converges to an estimator with connections to regression control methods. We refer to such estimators as adaptively normalized estimators. For mean estimation in survey sampling, the adaptively normalized estimator has asymptotic variance that is never worse than the Horvitz–Thompson and Hájek estimators. Going further, we show that adaptive normalization can be used to propose improvements of the augmented IPW (AIPW) estimator, average treatment effect (ATE) estimators, and policy learning objectives. Appealingly, these proposals preserve both the asymptotic efficiency of AIPW and the regret bounds for policy learning with IPW objectives, and deliver consistent finite sample improvements in simulations for all three of mean estimation, ATE estimation, and policy learning.
S2 Engine: A Novel Systolic Architecture for Sparse Convolutional Neural Networks

Convolutional neural networks (CNNs) have achieved great success in performing cognitive tasks. However, execution of CNNs requires a large amount of computing resources and generates heavy memory traffic, which imposes a severe challenge on computing system design. Through optimizing parallel executions and data reuse in convolution, systolic architecture demonstrates great advantages in accelerating CNN computations. However, regular internal data transmission path in traditional systolic architecture prevents the systolic architecture from completely leveraging the benefits introduced by neural network sparsity. Deployment of fine-grained sparsity on the existing systolic architectures is greatly hindered by the incurred computational overheads. In this work, we propose <inline-formula><tex-math notation="LaTeX">${\mathsf {S}}^{2}$</tex-math><alternatives><mml:math><mml:msup><mml:mrow><mml:mi mathvariant="sans-serif">S</mml:mi></mml:mrow><mml:mn>2</mml:mn></mml:msup></mml:math><inline-graphic xlink:href="yang-ieq1-3087946.gif"/></alternatives></inline-formula>Engine – a novel systolic architecture that can fully exploit the sparsity in CNNs with maximized data reuse. <inline-formula><tex-math notation="LaTeX">${\mathsf {S}}^{2}$</tex-math><alternatives><mml:math><mml:msup><mml:mrow><mml:mi mathvariant="sans-serif">S</mml:mi></mml:mrow><mml:mn>2</mml:mn></mml:msup></mml:math><inline-graphic xlink:href="yang-ieq2-3087946.gif"/></alternatives></inline-formula>Engine transmits compressed data internally and allows each processing element to dynamically select an aligned data from the compressed dataflow in convolution. Compared to the naïve systolic array, <inline-formula><tex-math notation="LaTeX">${\mathsf {S}}^{2}$</tex-math><alternatives><mml:math><mml:msup><mml:mrow><mml:mi mathvariant="sans-serif">S</mml:mi></mml:mrow><mml:mn>2</mml:mn></mml:msup></mml:math><inline-graphic xlink:href="yang-ieq3-3087946.gif"/></alternatives></inline-formula>Engine achieves about <inline-formula><tex-math notation="LaTeX">$3.2\times$</tex-math><alternatives><mml:math><mml:mrow><mml:mn>3</mml:mn><mml:mo>. </mml:mo><mml:mn>2</mml:mn><mml:mo>×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="yang-ieq4-3087946.gif"/></alternatives></inline-formula> and about <inline-formula><tex-math notation="LaTeX">$3.0\times$</tex-math><alternatives><mml:math><mml:mrow><mml:mn>3</mml:mn><mml:mo>. </mml:mo><mml:mn>0</mml:mn><mml:mo>×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="yang-ieq5-3087946.gif"/></alternatives></inline-formula> improvements on speed and energy efficiency, respectively.
Emergence of the 
ρ
 resonance from the HAL QCD potential in lattice QCD

We investigate the $I=1$ $\pi \pi$ interaction using the HAL QCD method in lattice QCD. We employ the (2+1)-flavor gauge configurations on $32^3 \times 64$ lattice at the lattice spacing $a \approx 0.0907$ fm and $m_{\pi} \approx 411$ MeV, in which the $\rho$ meson appears as a resonance state. We find that all-to-all propagators necessary in this calculation can be obtained with reasonable precision by a combination of three techniques, the one-end trick, the sequential propagator, and the covariant approximation averaging (CAA). The non-local $I=1$ $\pi \pi$ potential is determined at the next-to-next-to-leading order (N$^2$LO) of the derivative expansion for the first time, and the resonance parameters of the $\rho$ meson are extracted. The obtained $\rho$ meson mass is found to be consistent with the value in the literature, while the value of the coupling $g_{\rho \pi \pi}$ turns out to be somewhat larger. The latter observation is most likely attributed to the lack of low-energy information in our lattice setup with the center-of-mass frame. Such a limitation may appear in other P-wave resonant systems and we discuss possible improvement in future. With this caution in mind, we positively conclude that we can reasonably extract the N$^2$LO potential and resonance parameters even in the system requiring the all-to-all propagators in the HAL QCD method, which opens up new possibilities for the study of resonances in lattice QCD.
Automated triaging of head MRI examinations using convolutional neural networks

The growing demand for head magnetic resonance imaging (MRI) examinations, along with a global shortage of radiologists, has led to an increase in the time taken to report head MRI scans around the world. For many neurological conditions, this delay can result in increased morbidity and mortality. An automated triaging tool could reduce reporting times for abnormal examinations by identifying abnormalities at the time of imaging and prioritizing the reporting of these scans. In this work, we present a convolutional neural network for detecting clinically-relevant abnormalities in $\text{T}_2$-weighted head MRI scans. Using a validated neuroradiology report classifier, we generated a labelled dataset of 43,754 scans from two large UK hospitals for model training, and demonstrate accurate classification (area under the receiver operating curve (AUC) = 0.943) on a test set of 800 scans labelled by a team of neuroradiologists. Importantly, when trained on scans from only a single hospital the model generalized to scans from the other hospital ($\Delta$AUC $\leq$ 0.02). A simulation study demonstrated that our model would reduce the mean reporting time for abnormal examinations from 28 days to 14 days and from 9 days to 5 days at the two hospitals, demonstrating feasibility for use in a clinical triage environment.
Remarks on Regularized Stokeslets in Slender Body Theory

We remark on the use of regularized Stokeslets in the slender body theory (SBT) approximation of Stokes flow about a thin fiber of radius ϵ>0. Denoting the regularization parameter by δ, we consider regularized SBT based on the most common regularized Stokeslet plus a regularized doublet correction. Given sufficiently smooth force data along the filament, we derive L∞ bounds for the difference between regularized SBT and its classical counterpart in terms of δ, ϵ, and the force data. We show that the regularized and classical expressions for the velocity of the filament itself differ by a term proportional to log(δ/ϵ); in particular, δ=ϵ is necessary to avoid an O(1) discrepancy between the theories. However, the flow at the surface of the fiber differs by an expression proportional to log(1+δ2/ϵ2), and any choice of δ∝ϵ will result in an O(1) discrepancy as ϵ→0. Consequently, the flow around a slender fiber due to regularized SBT does not converge to the solution of the well-posed slender body PDE which classical SBT is known to approximate. Numerics verify this O(1) discrepancy but also indicate that the difference may have little impact in practice.
Code integrity attestation for PLCs using black box neural network predictions

Cyber-physical systems (CPSs) are widespread in critical domains, and significant damage can be caused if an attacker is able to modify the code of their programmable logic controllers (PLCs). Unfortunately, traditional techniques for attesting code integrity (i.e. verifying that it has not been modified) rely on firmware access or roots-of-trust, neither of which proprietary or legacy PLCs are likely to provide. In this paper, we propose a practical code integrity checking solution based on privacy-preserving black box models that instead attest the input/output behaviour of PLC programs. Using faithful offline copies of the PLC programs, we identify their most important inputs through an information flow analysis, execute them on multiple combinations to collect data, then train neural networks able to predict PLC outputs (i.e. actuator commands) from their inputs. By exploiting the black box nature of the model, our solution maintains the privacy of the original PLC code and does not assume that attackers are unaware of its presence. The trust instead comes from the fact that it is extremely hard to attack the PLC code and neural networks at the same time and with consistent outcomes. We evaluated our approach on a modern six-stage water treatment plant testbed, finding that it could predict actuator states from PLC inputs with near-100% accuracy, and thus could detect all 120 effective code mutations that we subjected the PLCs to. Finally, we found that it is not practically possible to simultaneously modify the PLC code and apply discreet adversarial noise to our attesters in a way that leads to consistent (mis-)predictions.
CAN-LOC: Spoofing Detection and Physical Intrusion Localization on an In-Vehicle CAN Bus Based on Deep Features of Voltage Signals

The Controller Area Network (CAN), which is used for communication between in-vehicle devices, has been shown to be vulnerable to spoofing attacks. Voltage-based spoofing detection (VBS-D) mechanisms are considered state-of-the-art solutions, complementing cryptography-based authentication whose security is limited due to the CAN protocol’s limited message size. Unfortunately, VBS-D mechanisms are vulnerable to poisoning performed by a malicious device connected to the CAN bus, specifically designed to poison the deployed VBS-D mechanism as it adapts to environmental changes that take place when the vehicle is moving. In this paper, we harden VBS-D mechanisms using a deep learning-based mechanism which runs immediately, when the vehicle starts; this mechanism utilizes physical side-channels to detect and locate physical intrusions, even when the malicious devices connected to the CAN bus are silent. We demonstrate the mechanism’s effectiveness (100% intrusion detection accuracy and error rates of close to 0%) in various physical intrusion scenarios and varying temperatures on a CAN bus prototype. In addition, we present a deep learning-based VBS-D mechanism that securely adapts to environmental changes. This mechanism’s robustness (99.8% device identification accuracy) is demonstrated on a real moving vehicle.
Self-Supervised Learning with Kernel Dependence Maximization

We approach self-supervised learning of image representations from a statistical dependence perspective, proposing Self-Supervised Learning with the Hilbert-Schmidt Independence Criterion (SSL-HSIC). SSL-HSIC maximizes dependence between representations of transformations of an image and the image identity, while minimizing the kernelized variance of those representations. This framework yields a new understanding of InfoNCE, a variational lower bound on the mutual information (MI) between different transformations. While the MI itself is known to have pathologies which can result in learning meaningless representations, its bound is much better behaved: we show that it implicitly approximates SSL-HSIC (with a slightly different regularizer). Our approach also gives us insight into BYOL, a negative-free SSL method, since SSL-HSIC similarly learns local neighborhoods of samples. SSL-HSIC allows us to directly optimize statistical dependence in time linear in the batch size, without restrictive data assumptions or indirect mutual information estimators. Trained with or without a target network, SSL-HSIC matches the current state-of-the-art for standard linear evaluation on ImageNet, semi-supervised learning and transfer to other classification and vision tasks such as semantic segmentation, depth estimation and object recognition. Code is available at https://github.com/deepmind/ssl_hsic .