Miracle-dz/UniFAR-FTUs huggingface dataset:
Using Self-Supervised Feature Extractors with Attention for Automatic COVID-19 Detection from Speech

The ComParE 2021 COVID-19 Speech Sub-challenge provides a test-bed for the evaluation of automatic detectors of COVID-19 from speech. Such models can be of value by providing test triaging capabilities to health authorities, working alongside traditional testing methods. Herein, we leverage the usage of pre-trained, problem agnostic, speech representations and evaluate their use for this task. We compare the obtained results against a CNN architecture trained from scratch and traditional frequency-domain representations. We also evaluate the usage of Self-Attention Pooling as an utterance-level information aggregation method. Experimental results demonstrate that models trained on features extracted from self-supervised models perform similarly or outperform fully-supervised models and models based on handcrafted features. Our best model improves the Unweighted Average Recall (UAR) from 69.0\% to 72.3\% on a development set comprised of only full-band examples and achieves 64.4\% on the test set. Furthermore, we study where the network is attending, attempting to draw some conclusions regarding its explainability. In this relatively small dataset, we find the network attends especially to vowels and aspirates.
Robust Coreset for Continuous-and-Bounded Learning (with Outliers)

In this big data era, we often confront large-scale data in many machine learning tasks. A common approach for dealing with large-scale data is to build a small summary, e.g., coreset, that can eﬃciently represent the original input. However, real-world datasets usually contain outliers and most existing coreset construction methods are not resilient against outliers (in particular, the outliers can be located arbitrarily in the space by an adversarial attacker). In this paper, we propose a novel robust coreset method for the continuous-and-bounded learning problem (with outliers) which includes a broad range of popular optimization objectives in machine learning, like logistic regression and k -means clustering. Moreover, our robust coreset can be eﬃciently maintained in fully-dynamic environment. To the best of our knowledge, this is the ﬁrst robust and fully-dynamic coreset construction method for these optimization problems. We also conduct the experiments to evaluate the eﬀectiveness of our robust coreset in practice.
Understanding Adversarial Examples Through Deep Neural Network's Response Surface and Uncertainty Regions

Deep neural network (DNN) is a popular model implemented in many systems to handle complex tasks such as image classification, object recognition, natural language processing etc. Consequently DNN structural vulnerabilities become part of the security vulnerabilities in those systems. In this paper we study the root cause of DNN adversarial examples. We examine the DNN response surface to understand its classification boundary. Our study reveals the structural problem of DNN classification boundary that leads to the adversarial examples. Existing attack algorithms can generate from a handful to a few hundred adversarial examples given one clean image. We show there are infinitely many adversarial images given one clean sample, all within a small neighborhood of the clean sample. We then define DNN uncertainty regions and show transferability of adversarial examples is not universal. We also argue that generalization error, the large sample theoretical guarantee established for DNN, cannot adequately capture the phenomenon of adversarial examples. We need new theory to measure DNN robustness.
Latent Execution for Neural Program Synthesis Beyond Domain-Specific Languages

Program synthesis from input-output (IO) examples has been a long-standing challenge. While recent works demonstrated limited success on domain-specific languages (DSL), it remains highly challenging to apply them to real-world programming languages, such as C. Due to complicated syntax and token variation, there are three major challenges: (1) unlike many DSLs, programs in languages like C need to compile first and are not executed via interpreters; (2) the program search space grows exponentially when the syntax and semantics of the programming language become more complex; and (3) collecting a large-scale dataset of real-world programs is non-trivial. As a first step to address these challenges, we propose LaSynth and show its efficacy in a restricted-C domain. More specifically, LaSynth learns the latent representation to approximate the execution of partially generated programs, even if they are incomplete in syntax (addressing (1)). The learned execution significantly improves the performance of next token prediction over existing approaches, facilitating search (addressing (2)). Finally, once trained with randomly generated ground-truth programs and their IO pairs, LaSynth can synthesize more concise programs that resemble human-written code. Furthermore, retraining our model with these synthesized programs yields better performance with fewer samples for both Karel and C program synthesis, indicating the promise of leveraging the learned program synthesizer to improve the dataset quality for input-output program synthesis (addressing (3)). When evaluating on whether the program execution outputs match the IO pairs, LaSynth achieves 55.2% accuracy on generating simple C code with tens of tokens including loops and branches, outperforming existing approaches without executors by around 20%.
SISAL Revisited

Simplex identification via split augmented Lagrangian (SISAL) is a popularly-used algorithm in 4 blind unmixing of hyperspectral images. Developed by José M. Bioucas-Dias in 2009, the algorithm 5 is fundamentally relevant to tackling simplex-structured matrix factorization, and by extension, non6 negative matrix factorization, which have many applications under their umbrellas. In this article, we 7 revisit SISAL and provide new meanings to this quintessential algorithm. The formulation of SISAL 8 was motivated from a geometric perspective, with no noise. We show that SISAL can be explained 9 as an approximation scheme from a probabilistic simplex component analysis framework, which is 10 statistical and is principally more powerful in accommodating the presence of noise. The algorithm 11 for SISAL was designed based on a successive convex approximation method, with a focus on practical 12 utility. It was not known, by analyses, whether the SISAL algorithm has any kind of guarantee 13 of convergence to a stationary point. By establishing associations between the SISAL algorithm 14 and a line-search-based proximal gradient method, we confirm that SISAL can indeed guarantee 15 convergence to a stationary point. Our re-explanation of SISAL also reveals new formulations and 16 algorithms. The performance of these new possibilities is demonstrated by numerical experiments. 17
Predictive Modeling in the Presence of Nuisance-Induced Spurious Correlations

Deep predictive models often make use of spurious correlations between the label and the covariates that differ between training and test distributions. In many classiﬁcation tasks, spurious correlations are induced by a changing relationship between the label and some nuisance variables correlated with the covariates. For example, in classifying animals in natural images, the background, which is the nuisance, can predict the type of animal, but this nuisance label relationship does not always hold. This nuisance-label relationship does not always hold. We formalize a family of distributions that only differ in the nuisance-label relationship and and introduce a distribution where this relationship is broken called the nuisance-randomized distribution. We introduce a set of predictive models built from the nuisance-randomized distribution with representations, that when conditioned on, do not correlate the label and the nuisance. For models in this set, we lower bound the performance for any member of the family with the mutual information between the representation and the label under the nuisance-randomized distribution. To build predictive models that maximize the performance lower bound, we develop Nuisance-Randomized Distillation (N U RD). We evaluate NURD on a synthetic example, colored-MNIST, and classifying chest X-rays. When using non-lung patches as the nuisance in classifying chest X-rays, N U RD produces models that predict pneumonia under strong spurious correlations.
Salypath: A Deep-Based Architecture For Visual Attention Prediction

Human vision is naturally more attracted by some regions within their field of view than others. This intrinsic selectivity mechanism, so-called visual attention, is influenced by both high- and low-level factors; such as the global environment (illumination, background texture, etc.), stimulus characteristics (color, intensity, orientation, etc.), and some prior visual information. Visual attention is useful for many computer vision applications such as image compression, recognition, and captioning. In this paper, we propose an end-to-end deep-based method, so-called SALYPATH (SALiencY and scanPATH), that efficiently predicts the scanpath of an image through features of a saliency model. The idea is predict the scanpath by exploiting the capacity of a deep-based model to predict the saliency. The proposed method was evaluated through 2 well-known datasets. The results obtained showed the relevance of the proposed framework comparing to state-of-the-art models.
Reducing the Variance of Gaussian Process Hyperparameter Optimization with Preconditioning

Gaussian processes remain popular as a ﬂexible and expressive model class, but the computational cost of kernel hyperparameter optimization stands as a major limiting factor to their scaling and broader adoption. Recent work has made great strides combining stochastic estimation with iterative numerical techniques, essentially boiling down GP inference to the cost of (many) matrix-vector multiplies. Preconditioning – a highly effective step for any iterative method involving matrix-vector multiplication – can be used to accelerate convergence and thus reduce bias in hyperparameter optimization. Here, we prove that preconditioning has an additional beneﬁt that has been previously unexplored. It not only reduces the bias of the log -marginal likelihood estimator and its derivatives, but it also simultaneously can reduce variance at essentially negligible cost. We leverage this result to derive sample-efﬁcient algorithms for GP hyperparameter optimization requiring as few as O (log (cid:0) ε − 1 (cid:1) ) instead of O ( ε − 2 ) samples to achieve error ε . Our theoretical results enable provably efﬁcient and scalable optimization of kernel hyperparameters, which we validate empirically on a set of large-scale benchmark problems. There, variance reduction via preconditioning results in an order of magnitude speedup in hyperparameter optimization of exact GPs.
Lipschitz functions on submanifolds in Heisenberg groups

We study the behavior of Lipschitz functions on intrinsic C submanifolds of Heisenberg groups: our main result is their almost everywhere tangential Pansu differentiability. We also provide two applications: a Lusintype approximation of Lipschitz functions on H-rectifiable sets, and a coarea formula on H-rectifiable sets that completes the program started in [16].
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows

We present CSWin Transformer, an efficient and effective Transformer-based backbone for general-purpose vision tasks. A challenging issue in Transformer design is that global self-attention is very expensive to compute whereas local self-attention often limits the field of interactions of each token. To address this issue, we develop the Cross-Shaped Window self-attention mechanism for computing self-attention in the horizontal and vertical stripes in parallel that form a cross-shaped window, with each stripe obtained by splitting the input feature into stripes of equal width. We provide a mathematical analysis of the effect of the stripe width and vary the stripe width for different layers of the Transformer network which achieves strong modeling capability while limiting the computation cost. We also introduce Locally-enhanced Positional Encoding (LePE), which handles the local positional information better than existing encoding schemes. LePE naturally supports arbitrary input resolutions, and is thus especially effective and friendly for downstream tasks. Incorporated with these designs and a hierarchical structure, CSWin Transformer demonstrates competitive performance on common vision tasks. Specifically, it achieves 85.4% Top-1 accuracy on ImageNet-1K without any extra training data or label, 53.9 box AP and 46.4 mask AP on the COCO detection task, and 52.2 mIOU on the ADE20K semantic segmentation task, surpassing previous state-of-the-art Swin Transformer backbone by +1.2, +2.0, +1.4, and +2.0 respectively under the similar FLOPs setting. By further pretraining on the larger dataset ImageNet-21K, we achieve 87.5% Top-1 accuracy on ImageNet-1K and high segmentation performance on ADE20K with 55.7 mIoU. 11Code and pretrain model is available at https://github.com/microsoft/CSWin-Transformer
Variational Diffusion Models

Diffusion-based generative models have demonstrated a capacity for perceptually impressive synthesis, but can they also be great likelihood-based models? We answer this in the affirmative, and introduce a family of diffusion-based generative models that obtain state-of-the-art likelihoods on standard image density estimation benchmarks. Unlike other diffusion-based models, our method allows for efficient optimization of the noise schedule jointly with the rest of the model. We show that the variational lower bound (VLB) simplifies to a remarkably short expression in terms of the signal-to-noise ratio of the diffused data, thereby improving our theoretical understanding of this model class. Using this insight, we prove an equivalence between several models proposed in the literature. In addition, we show that the continuous-time VLB is invariant to the noise schedule, except for the signal-to-noise ratio at its endpoints. This enables us to learn a noise schedule that minimizes the variance of the resulting VLB estimator, leading to faster optimization. Combining these advances with architectural improvements, we obtain state-of-the-art likelihoods on image density estimation benchmarks, outperforming autoregressive models that have dominated these benchmarks for many years, with often significantly faster optimization. In addition, we show how to use the model as part of a bits-back compression scheme, and demonstrate lossless compression rates close to the theoretical optimum. Code is available at https://github.com/google-research/vdm .
Secure Quantized Training for Deep Learning

We implement training of neural networks in secure multi-party computation (MPC) using quantization commonly used in said setting. We are the first to present an MNIST classifier purely trained in MPC that comes within 0.2 percent of the accuracy of the same convolutional neural network trained via plaintext computation. More concretely, we have trained a network with two convolutional and two dense layers to 99.2% accuracy in 3.5 hours (under one hour for 99% accuracy). We have also implemented AlexNet for CIFAR-10, which converges in a few hours. We develop novel protocols for exponentiation and inverse square root. Finally, we present experiments in a range of MPC security models for up to ten parties, both with honest and dishonest majority as well as semi-honest and malicious security.
Global Filter Networks for Image Classification

Recent advances in self-attention and pure multi-layer perceptrons (MLP) models for vision have shown great potential in achieving promising performance with fewer inductive biases. These models are generally based on learning interaction among spatial locations from raw data. The complexity of self-attention and MLP grows quadratically as the image size increases, which makes these models hard to scale up when high-resolution features are required. In this paper, we present the Global Filter Network (GFNet), a conceptually simple yet computationally efficient architecture, that learns long-term spatial dependencies in the frequency domain with log-linear complexity. Our architecture replaces the self-attention layer in vision transformers with three key operations: a 2D discrete Fourier transform, an element-wise multiplication between frequency-domain features and learnable global filters, and a 2D inverse Fourier transform. We exhibit favorable accuracy/complexity trade-offs of our models on both ImageNet and downstream tasks. Our results demonstrate that GFNet can be a very competitive alternative to transformer-style models and CNNs in efficiency, generalization ability and robustness. Code is available at https://github.com/raoyongming/GFNet
Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble

Recent advance in deep offline reinforcement learning (RL) has made it possible to train strong robotic agents from offline datasets. However, depending on the quality of the trained agents and the application being considered, it is often desirable to fine-tune such agents via further online interactions. In this paper, we observe that state-action distribution shift may lead to severe bootstrap error during fine-tuning, which destroys the good initial policy obtained via offline RL. To address this issue, we first propose a balanced replay scheme that prioritizes samples encountered online while also encouraging the use of near-on-policy samples from the offline dataset. Furthermore, we leverage multiple Q-functions trained pessimistically offline, thereby preventing overoptimism concerning unfamiliar actions at novel states during the initial training phase. We show that the proposed method improves sample-efficiency and final performance of the fine-tuned robotic agents on various locomotion and manipulation tasks. Our code is available at: https://github.com/shlee94/Off2OnRL.
Using AntiPatterns to avoid MLOps Mistakes

We describe lessons learned from developing and deploying machine learning models at scale across the enterprise in a range of financial analytics applications. These lessons are presented in the form of antipatterns. Just as design patterns codify best software engineering practices, antipatterns provide a vocabulary to describe defective practices and methodologies. Here we catalog and document numerous antipatterns in financial ML operations (MLOps). Some antipatterns are due to technical errors, while others are due to not having sufficient knowledge of the surrounding context in which ML results are used. By providing a common vocabulary to discuss these situations, our intent is that antipatterns will support better documentation of issues, rapid communication between stakeholders, and faster resolution of problems. In addition to cataloging antipatterns, we describe solutions, best practices, and future directions toward MLOps maturity.
Capturing Event Argument Interaction via A Bi-Directional Entity-Level Recurrent Decoder

Capturing interactions among event arguments is an essential step towards robust event argument extraction (EAE). However, existing efforts in this direction suffer from two limitations: 1) The argument role type information of contextual entities is mainly utilized as training signals, ignoring the potential merits of directly adopting it as semantically rich input features; 2) The argument-level sequential semantics, which implies the overall distribution pattern of argument roles over an event mention, is not well characterized. To tackle the above two bottlenecks, we formalize EAE as a Seq2Seq-like learning problem for the first time, where a sentence with a specific event trigger is mapped to a sequence of event argument roles. A neural architecture with a novel Bi-directional Entity-level Recurrent Decoder (BERD) is proposed to generate argument roles by incorporating contextual entities’ argument role predictions, like a word-by-word text generation process, thereby distinguishing implicit argument distribution patterns within an event more accurately.
Secretive Coded Caching from PDAs

The coded caching problem with secrecy constraint i.e., the users should not be able to gain any information about the content of the files that they did not demand, is known as the secretive coded caching problem. This was proposed by Ravindrakumar et al. in the paper titled "Private Coded Caching" that appeared in IEEE Transactions on Information Forensics and Security, 2018 and is characterised by subpacketization levels growing exponentially with the number of users. In the context of coded caching without secrecy, coded caching schemes at subexponential subpacketization levels are feasible by representing the caching system in the form of a Placement Delivery Array (PDA) and designing placement and delivery policies from it. Motivated by this, we propose a secretive coded caching scheme with low subpacketization using PDA, for users with dedicated caches in the centralized setting. When our scheme is applied to a special class of PDA known as MN PDA, the scheme proposed by Ravindrakumar et al. is recovered.
Few-Shot Learning With a Strong Teacher

Few-shot learning (FSL) aims to generate a classifier using limited labeled examples. Many existing works take the meta-learning approach, constructing a few-shot learner (a meta-model) that can learn from few-shot examples to generate a classifier. Typically, the few-shot learner is constructed or meta-trained by sampling multiple few-shot tasks in turn and optimizing the few-shot learner's performance in generating classifiers for those tasks. The performance is measured by how well the resulting classifiers classify the test (i.e., query) examples of those tasks. In this paper, we point out two potential weaknesses of this approach. First, the sampled query examples may not provide sufficient supervision for meta-training the few-shot learner. Second, the effectiveness of meta-learning diminishes sharply with the increasing number of shots (i.e., the number of training examples per class). To resolve these issues, we propose a novel meta-training objective for the few-shot learner, which is to encourage the few-shot learner to generate classifiers that perform like strong classifiers. Concretely, we associate each sampled few-shot task with a strong classifier, which is trained with ample labeled examples. The strong classifiers can be seen as the target classifiers that we hope the few-shot learner to generate given few-shot examples, and we use the strong classifiers to supervise the few-shot learner. We present an efficient way to construct the strong classifier, making our proposed objective an easily plug-and-play term to existing meta-learning based FSL methods. We validate our approach, (Learning with A Strong Teacher for few-SHOT learning), in combinations with many representative meta-learning methods. On several benchmark datasets including miniImageNet and tieredImageNet, our approach leads to a notable improvement across a variety of tasks. More importantly, with our approach, meta-learning based FSL methods can consistently outperform non-meta-learning based methods at different numbers of shots, even in many-shot settings, greatly strengthening their applicability.