Miracle-dz/UniFAR-FTUs huggingface dataset:
You Don't Need Robust Machine Learning to Manage Adversarial Attack Risks

The robustness of modern machine learning (ML) models has become an increasing concern within the community. The ability to subvert a model into making errant predictions using seemingly inconsequential changes to input is startling, as is our lack of success in building models robust to this concern. Existing research shows progress, but current mitigations come with a high cost and simultaneously reduce the model's accuracy. However, such trade-offs may not be necessary when other design choices could subvert the risk. In this survey we review the current literature on attacks and their real-world occurrences, or limited evidence thereof, to critically evaluate the real-world risks of adversarial machine learning (AML) for the average entity. This is done with an eye toward how one would then mitigate these attacks in practice, the risks for production deployment, and how those risks could be managed. In doing so we elucidate that many AML threats do not warrant the cost and trade-offs of robustness due to a low likelihood of attack or availability of superior non-ML mitigations. Our analysis also recommends cases where an actor should be concerned about AML to the degree where robust ML models are necessary for a complete deployment.
Above-threshold ionization at laser intensity greater than $10^{20}$ W/cm$^{2}$

We present the first experimental observation of above-threshold ionization (ATI) electrons produced by ionization of the neon K-shell in a laser field where intensity exceeds 10$^{20}$ W/cm$^{2}$. An array of plastic scintillating calorimeter detectors was used to measure the high-energy electrons at four angles in the laser forward direction. Coarse energy resolution was obtained using aluminum filters of several thicknesses to block lower-energy electrons. A threshold intensity around $2 \times 10^{20}$ W/cm$^{2}$ is observed for production of energetic ATI electrons in the laser forward direction, with maximum electron energy exceeding 10 MeV. L-shell electrons with energies<1.4 MeV are scattered further forward along the laser direction than expected. We present comparisons of the measured total electron energies to the predictions of a Monte Carlo models employing the ADK-PPT ionization model and the Augst barrier suppression ionization model.
Modularizing while Training: a New Paradigm for Modularizing DNN Models

Deep neural network (DNN) models have become increasingly crucial components in intelligent software systems. However, training a DNN model is typically expensive in terms of both time and money. To address this issue, researchers have recently focused on reusing existing DNN models - borrowing the idea of code reuse in software engineering. However, reusing an entire model could cause extra overhead or inherits the weakness from the undesired functionalities. Hence, existing work proposes to decompose an already trained model into modules, i.e., modularizing-after-training, and enable module reuse. Since trained models are not built for modularization, modularizing-after-training incurs huge overhead and model accuracy loss. In this paper, we propose a novel approach that incorporates modularization into the model training process, i.e., modularizing-while-training (MwT). We train a model to be structurally modular through two loss functions that optimize intra-module cohesion and inter-module coupling. We have implemented the proposed approach for modularizing Convolutional Neural Network (CNN) models in this work. The evaluation results on representative models demonstrate that MwT outperforms the state-of-the-art approach. Specifically, the accuracy loss caused by MwT is only 1.13 percentage points, which is 1.76 percentage points less than that of the baseline. The kernel retention rate of the modules generated by MwT is only 14.58%, with a reduction of 74.31% over the state-of-the-art approach. Furthermore, the total time cost required for training and modularizing is only 108 minutes, half of the baseline.
Evaluating hardware differences for crowdsourcing and traditional recruiting methods

The most frequently used method to collect research data online is crowdsouring and its use continues to grow rapidly. This report investigates for the first time whether researchers also have to expect significantly different hardware performance when deploying to Amazon Mechanical Turk (MTurk). This is assessed by collecting basic hardware parameters (Operating System, GPU, and used browser) from Amazon Mechanical Turk (MTurk) and a traditional recruitment method (i.e., snowballing). The significant hardware differences between crowdsourcing participants (MTurk) and snowball recruiting are reported including relevant descriptive statistics for assessing hardware performance of 3D web applications. The report suggests that hardware differences need to be considered to obtain valid results if the designed experiment application requires graphical intense computations and relies on a coherent user experience of MTurk and more established recruitment strategies (i.e. snowballing).
Retrospective: EIE: Efficient Inference Engine on Sparse and Compressed Neural Network

EIE proposed to accelerate pruned and compressed neural networks, exploiting weight sparsity, activation sparsity, and 4-bit weight-sharing in neural network accelerators. Since published in ISCA'16, it opened a new design space to accelerate pruned and sparse neural networks and spawned many algorithm-hardware co-designs for model compression and acceleration, both in academia and commercial AI chips. In retrospect, we review the background of this project, summarize the pros and cons, and discuss new opportunities where pruning, sparsity, and low precision can accelerate emerging deep learning workloads.
Distillation Strategies for Discriminative Speech Recognition Rescoring

Second-pass rescoring is employed in most state-of-the-art speech recognition systems. Recently, BERT based models have gained popularity for re-ranking the n-best hypothesis by exploiting the knowledge from masked language model pre-training. Further, fine-tuning with discriminative loss such as minimum word error rate (MWER) has shown to perform better than likelihood-based loss. Streaming applications with low latency requirements impose significant constraints on the size of the models, thereby limiting the word error rate (WER) performance gains. In this paper, we propose effective strategies for distilling from large models discriminatively trained with the MWER objective. We experiment on Librispeech and production scale internal dataset for voice-assistant. Our results demonstrate relative improvements of upto 7% WER over student models trained with MWER. We also show that the proposed distillation can reduce the WER gap between the student and the teacher by 62% upto 100%.
The Evolution theory of Learning: From Natural Selection to Reinforcement Learning

Evolution is a fundamental process that shapes the biological world we inhabit, and reinforcement learning is a powerful tool used in artificial intelligence to develop intelligent agents that learn from their environment. In recent years, researchers have explored the connections between these two seemingly distinct fields, and have found compelling evidence that they are more closely related than previously thought. This paper examines these connections and their implications, highlighting the potential for reinforcement learning principles to enhance our understanding of evolution and the role of feedback in evolutionary systems.
Causation, Prediction, and Search, 2nd Edition

What assumptions and methods allow us to turn observations into causal knowledge, and how can even incomplete causal knowledge be used in planning and prediction to influence and control our environment? In this book Peter Spirtes, Clark Glymour, and Richard Scheines address these questions using the formalism of Bayes networks, with results that have been applied in diverse areas of research in the social, behavioral, and physical sciences. The authors show that although experimental and observational study designs may not always permit the same inferences, they are subject to uniform principles. They axiomatize the connection between causal structure and probabilistic independence, explore several varieties of causal indistinguishability, formulate a theory of manipulation, and develop asymptotically reliable procedures for searching over equivalence classes of causal models, including models of categorical data and structural equation models with and without latent variables. The authors show that the relationship between causality and probability can also help to clarify such diverse topics in statistics as the comparative power of experimentation versus observation, Simpson's paradox, errors in regression models, retrospective versus prospective sampling, and variable selection. The second edition contains a new introduction and an extensive survey of advances and applications that have appeared since the first edition was published in 1993.
Scoring rules, and the role of chance: Analysis of the 2008 World Fly Fishing Championships

Abstract We analyze the 2008 World Fly Fishing Championships data to examine two issues: (I) changes to scoring rules are proposed so that catching bigger fish is no longer a disadvantage; and (II) the relative role of chance versus competitor skill in determining the competition outcomes is investigated. For (I), a new quadratic polynomial formula for the number of points awarded for a fish of a certain length meant that the competitor who caught the biggest fish in one of the rivers was no longer disadvantaged, during the 20-min period it took to land the fish, compared to those anglers who caught many small fish. For (II), it was found that an angler A, who is actually better than angler B, has an approximate probability of 8.5% of having, overall, a worse score than angler B. By increasing the number of fishing sessions from five to seven, the probability of misclassification drops to about 7%. Other topics, such as the advantages of the proposed formula and the various fishing strategies, are also discussed.
Comment on “Rubric-based holistic review represents a change from traditional graduate admissions approaches in physics”

A recent paper used complicated machine learning methods to suggest that a new rubric-based graduate admissions approach was significantly different from the previous approach in an unspecified way. Simple inspection of the distributions of metrics shown in the paper shows that the rubric approach succeeded in the often-stated specific goal of increasing admissions of applicants with low undergraduate grade point averages and GRE scores. Nevertheless, the argument that this change is a promising way to improve graduate outcomes is based on a misreading of the prior literature. A method used in some of the analysis for dropping data points before running the machine algorithm is likely to bias those results.
Using Machine Learning Methods for Automation of Size Grid Building and Management

Fashion apparel companies require planning for the next season, a year in advance for supply chain management. This study focuses on size selection decision making for Levi Strauss. Currently, the region and planning group level size grids are built and managed manually. The company suffers from the workload it creates for sizing, merchant and planning teams. This research is aiming to answer two research questions:"Which sizes should be available to the planners under each size grid name for the next season(s)? "and"Which sizes should be adopted for each planning group for the next season(s)? ". We approach to the problem with a classification model, which is one of the popular models used in machine learning. With this research, a more automated process was created by using machine learning techniques. A decrease in workload of the teams in the company is expected after it is put into practice. Unlike many studies in the state of art for fashion and apparel industry, this study focuses on sizes where the stock keeping unit represents a product with a certain size.
On the finitary content of Dykstra's cyclic projections algorithm

We study the asymptotic behaviour of the well-known Dykstra's algorithm through the lens of proof-theoretical techniques. We provide an elementary proof for the convergence of Dykstra's algorithm in which the standard argument is stripped to its central features and where the original compactness principles are circumvented, additionally providing highly uniform primitive recursive rates of metastability in a full general setting. Moreover, under an additional assumption, we are even able to obtain effective general rates of convergence. We argue that such additional condition is actually necessary for the existence of general uniform rates of convergence.
Unsupervised Anomaly Detection via Nonlinear Manifold Learning

Anomalies are samples that significantly deviate from the rest of the data and their detection plays a major role in building machine learning models that can be reliably used in applications such as data-driven design and novelty detection. The majority of existing anomaly detection methods either are exclusively developed for (semi) supervised settings, or provide poor performance in unsupervised applications where there is no training data with labeled anomalous samples. To bridge this research gap, we introduce a robust, efficient, and interpretable methodology based on nonlinear manifold learning to detect anomalies in unsupervised settings. The essence of our approach is to learn a low-dimensional and interpretable latent representation (aka manifold) for all the data points such that normal samples are automatically clustered together and hence can be easily and robustly identified. We learn this low-dimensional manifold by designing a learning algorithm that leverages either a latent map Gaussian process (LMGP) or a deep autoencoder (AE). Our LMGP-based approach, in particular, provides a probabilistic perspective on the learning task and is ideal for high-dimensional applications with scarce data. We demonstrate the superior performance of our approach over existing technologies via multiple analytic examples and real-world datasets.
Uncertainty Quantification via Spatial-Temporal Tweedie Model for Zero-inflated and Long-tail Travel Demand Prediction

Understanding Origin-Destination (O-D) travel demand is crucial for transportation management. However, traditional spatial-temporal deep learning models grapple with addressing the sparse and long-tail characteristics in high-resolution O-D matrices and quantifying prediction uncertainty. This dilemma arises from the numerous zeros and over-dispersed demand patterns within these matrices, which challenge the Gaussian assumption inherent to deterministic deep learning models. To address these challenges, we propose a novel approach: the Spatial-Temporal Tweedie Graph Neural Network (STTD). The STTD introduces the Tweedie distribution as a compelling alternative to the traditional 'zero-inflated' model and leverages spatial and temporal embeddings to parameterize travel demand distributions. Our evaluations using real-world datasets highlight STTD's superiority in providing accurate predictions and precise confidence intervals, particularly in high-resolution scenarios. GitHub code is available online(https://github.com/STTDAnonymous/STTD).
Gauge invariance of radiative jet functions in the position-space formulation of SCET

In subleading powers of soft-collinear effective theory (SCET), the Lagrangian contains couplings between soft quarks and hard-collinear quarks. Matrix elements of the hard-collinear parts of these couplings are radiative jet functions. In the position-space formulation of SCET, the Lagrangians are constructed from gauge-invariant building blocks. Nevertheless, we find violations of gauge invariance in the radiative jet functions that we construct from these Lagrangians. We trace the violations of gauge invariance to Feynman diagrams that would be present in full QCD, but are missing in the radiative jet functions, owing to the requirement that the radiative jet functions contain only hard-collinear lines. We construct gauge-invariant Lagrangians for the interactions between soft quarks and hard-collinear quarks by making use of the soft-quark equation of motion. These gauge-invariant Lagrangians are valid for nonzero, as well as zero, quark masses. We also remark briefly on the gauge invariance of certain Lagrangians that have been constructed in the label-momentum formulation of SCET.
DNA Tile Self-Assembly for 3D-Surfaces: Towards Genus Identification

We introduce a new DNA tile self-assembly model: the Surface Flexible Tile Assembly Model (SFTAM), where 2D tiles are placed on host 3D surfaces made of axis-parallel unit cubes glued together by their faces, called polycubes. The bonds are flexible, so that the assembly can bind on the edges of the polycube. We are interested in the study of SFTAM self-assemblies on 3D surfaces which are not always embeddable in the Euclidean plane, in order to compare their different behaviors and to compute the topological properties of the host surfaces. We focus on a family of polycubes called cuboids. Order-0 cuboids are polycubes that have six rectangular faces, and order-1 cuboids are made from two order-0 cuboids by substracting one from the other. Thus, order-1 cuboids can be of genus 0 or of genus 1 (then they contain a tunnel). We are interested in the genus of these structures, and we present a SFTAM tile assembly system that determines the genus of a given order-1 cuboid. The SFTAM tile assembly system which we design, contains a specific set $Y$ of tile types with the following properties. If the assembly is made on a host order-1 cuboid $C$ of genus 0, no tile of $Y$ appears in any producible assembly, but if $C$ has genus 1, every terminal assembly contains at least one tile of $Y$. Thus, we are able to distinguish the host surfaces according to their genus, by the tiles used in the assembly. This system is specific to order-1 cuboids but the techniques we use should be generalizable to other families of shapes.
Group Orthogonalization Regularization For Vision Models Adaptation and Robustness

As neural networks become deeper, the redundancy within their parameters increases. This phenomenon has led to several methods that attempt to reduce the correlation between convolutional filters. We propose a computationally efficient regularization technique that encourages orthonormality between groups of filters within the same layer. Our experiments show that when incorporated into recent adaptation methods for diffusion models and vision transformers (ViTs), this regularization improves performance on downstream tasks. We further show improved robustness when group orthogonality is enforced during adversarial training. Our code is available at https://github.com/YoavKurtz/GOR.
Design of binocular vision 3D reconstruction system based on motion recovery

An 3D reconstruction system based on binocular stereo vision was improved to obtain higher accuracy and larger scale of 3D reconstruction. A disparity refinement procedure was introduced to the system, so that the matching cost between the original parallax and the adjacent disparity was fitted into a quadratic curve and to re-find a more accurate parallax for the curve. Then, the motion recovery calculation was further applies to estimation of the camera motion matrices and the tracking points and camera motion matrices were taken as the parameters to construct the energy function. Furthermore, the energy function was optimized to reduce errors effectively and reconstruct the motion matrix accurately. Experimental results indicate that the proposed refinement procedure effectively improves the accuracy of reconstruction of point clouds and reduces the error by 16.3 percent in average with avoiding the scale point cloud phenomenon. As the motion recovery calculation steps recover the camera motion matrix accurately, the optimized method reduces the mean re-projection error by 95.5 percent. The point clouds achieved from different angle images are no longer isolated and the reconstructed models are integrated naturally.
Cross-Modal Video to Body-joints Augmentation for Rehabilitation Exercise Quality Assessment

Exercise-based rehabilitation programs have been shown to enhance quality of life and reduce mortality and rehospitalizations. AI-driven virtual rehabilitation programs enable patients to complete exercises independently at home while AI algorithms can analyze exercise data to provide feedback to patients and report their progress to clinicians. This paper introduces a novel approach to assessing the quality of rehabilitation exercises using RGB video. Sequences of skeletal body joints are extracted from consecutive RGB video frames and analyzed by many-to-one sequential neural networks to evaluate exercise quality. Existing datasets for exercise rehabilitation lack adequate samples for training deep sequential neural networks to generalize effectively. A cross-modal data augmentation approach is proposed to resolve this problem. Visual augmentation techniques are applied to video data, and body joints extracted from the resulting augmented videos are used for training sequential neural networks. Extensive experiments conducted on the KInematic assessment of MOvement and clinical scores for remote monitoring of physical REhabilitation (KIMORE) dataset, demonstrate the superiority of the proposed method over previous baseline approaches. The ablation study highlights a significant enhancement in exercise quality assessment following cross-modal augmentation.
Towards Better Orthogonality Regularization with Disentangled Norm in Training Deep CNNs

Orthogonality regularization has been developed to prevent deep CNNs from training instability and feature redundancy. Among existing proposals, kernel orthogonality regularization enforces orthogonality by minimizing the residual between the Gram matrix formed by convolutional filters and the orthogonality matrix. We propose a novel measure for achieving better orthogonality among filters, which disentangles diagonal and correlation information from the residual. The model equipped with the measure under the principle of imposing strict orthogonality between filters surpasses previous regularization methods in near-orthogonality. Moreover, we observe the benefits of improved strict filter orthogonality in relatively shallow models, but as model depth increases, the performance gains in models employing strict kernel orthogonality decrease sharply. Furthermore, based on the observation of the potential conflict between strict kernel orthogonality and growing model capacity, we propose a relaxation theory on kernel orthogonality regularization. The relaxed kernel orthogonality achieves enhanced performance on models with increased capacity, shedding light on the burden of strict kernel orthogonality on deep model performance. We conduct extensive experiments with our kernel orthogonality regularization toolkit on ResNet and WideResNet in CIFAR-10 and CIFAR-100. We observe state-of-the-art gains in model performance from the toolkit, which includes both strict orthogonality and relaxed orthogonality regularization, and obtain more robust models with expressive features. These experiments demonstrate the efficacy of our toolkit and subtly provide insights into the often overlooked challenges posed by strict orthogonality, addressing the burden of strict orthogonality on capacity-rich models.