Miracle-dz/UniFAR-FTUs huggingface dataset:
Byzantine-Robust Distributed Online Learning: Taming Adversarial Participants in An Adversarial Environment

This paper studies distributed online learning under Byzantine attacks. The performance of an online learning algorithm is often characterized by (adversarial) regret, which evaluates the quality of one-step-ahead decision-making when an environment incurs adversarial losses, and a sublinear regret bound is preferred. But we prove that, even with a class of state-of-the-art robust aggregation rules, in an adversarial environment and in the presence of Byzantine participants, distributed online gradient descent can only achieve a linear adversarial regret bound, which is tight. This is the inevitable consequence of Byzantine attacks, even though we can control the constant of the linear adversarial regret to a reasonable level. Interestingly, when the environment is not fully adversarial so that the losses of the honest participants are i.i. d. (independent and identically distributed), we show that sublinear stochastic regret, in contrast to the aforementioned adversarial regret, is possible. We develop Byzantine-robust distributed online momentum algorithms to attain such sublinear stochastic regret bounds for a class of robust aggregation rules. Numerical experiments corroborate our theoretical analysis.
Analyzing Dataset Annotation Quality Management in the Wild

Data quality is crucial for training accurate, unbiased, and trustworthy machine learning models as well as for their correct evaluation. Recent works, however, have shown that even popular datasets used to train and evaluate state-of-the-art models contain a non-negligible amount of erroneous annotations, biases, or artifacts. While practices and guidelines regarding dataset creation projects exist, to our knowledge, large-scale analysis has yet to be performed on how quality management is conducted when creating natural language datasets and whether these recommendations are followed. Therefore, we first survey and summarize recommended quality management practices for dataset creation as described in the literature and provide suggestions for applying them. Then, we compile a corpus of 591 scientific publications introducing text datasets and annotate it for quality-related aspects, such as annotator management, agreement, adjudication, or data validation. Using these annotations, we then analyze how quality management is conducted in practice. A majority of the annotated publications apply good or excellent quality management. However, we deem the effort of 30% of the works as only subpar. Our analysis also shows common errors, especially when using inter-annotator agreement and computing annotation error rates.
Utilization of Pre-trained Language Model for Adapter-based Knowledge Transfer in Software Engineering

Software Engineering (SE) Pre-trained Language Models (PLMs), such as CodeBERT, are pre-trained on large code corpora, and their learned knowledge has shown success in transferring into downstream tasks (e.g., code clone detection) through the fine-tuning of PLMs. In Natural Language Processing (NLP), an alternative in transferring the knowledge of PLMs is explored through the use of adapter, a compact and parameter efficient module that is inserted into a PLM. Although the use of adapters has shown promising results in many NLP-based downstream tasks, their application and exploration in SE-based downstream tasks are limited. Here, we study the knowledge transfer using adapters on multiple down-stream tasks including cloze test, code clone detection, and code summarization. These adapters are trained on code corpora and are inserted into a PLM that is pre-trained on English corpora or code corpora. We called these PLMs as NL-PLM and C-PLM, respectively. We observed an improvement in results using NL-PLM over a PLM that does not have adapters, and this suggested that adapters can transfer and utilize useful knowledge from NL-PLM to SE tasks. The results are sometimes on par with or exceed the results of C-PLM; while being more efficient in terms of the number of parameters and training time. Interestingly, adapters inserted into a C-PLM generally yield better results than a traditional fine-tuned C-PLM. Our results open new directions to build more compact models for SE tasks.
On the Fly Neural Style Smoothing for Risk-Averse Domain Generalization

Achieving high accuracy on data from domains unseen during training is a fundamental challenge in domain generalization (DG). While state-of-the-art DG classifiers have demonstrated impressive performance across various tasks, they have shown a bias towards domain-dependent information, such as image styles, rather than domain-invariant information, such as image content. This bias renders them unreliable for deployment in risk-sensitive scenarios such as autonomous driving where a misclassification could lead to catastrophic consequences. To enable risk-averse predictions from a DG classifier, we propose a novel inference procedure, Test-Time Neural Style Smoothing (TT-NSS), that uses a"style-smoothed"version of the DG classifier for prediction at test time. Specifically, the style-smoothed classifier classifies a test image as the most probable class predicted by the DG classifier on random re-stylizations of the test image. TT-NSS uses a neural style transfer module to stylize a test image on the fly, requires only black-box access to the DG classifier, and crucially, abstains when predictions of the DG classifier on the stylized test images lack consensus. Additionally, we propose a neural style smoothing (NSS) based training procedure that can be seamlessly integrated with existing DG methods. This procedure enhances prediction consistency, improving the performance of TT-NSS on non-abstained samples. Our empirical results demonstrate the effectiveness of TT-NSS and NSS at producing and improving risk-averse predictions on unseen domains from DG classifiers trained with SOTA training methods on various benchmark datasets and their variations.
Automatic Identification of Alzheimer's Disease using Lexical Features extracted from Language Samples

Objective: this study has a twofold goal. First, it aims to improve the understanding of the impact of Dementia of type Alzheimer's Disease (AD) on different aspects of the lexicon. Second, it aims to demonstrate that such aspects of the lexicon, when used as features of a machine learning classifier, can help achieve state-of-the-art performance in automatically identifying language samples produced by patients with AD. Methods: data is derived from the ADDreSS challenge, which is a part of the DementiaBank corpus. The used dataset consists of transcripts of Cookie Theft picture descriptions, produced by 54 subjects in the training part and 24 subjects in the test part. The number of narrative samples is 108 in the training set and 48 in the test set. First, the impact of AD on 99 selected lexical features is studied using both the training and testing parts of the dataset. Then some machine learning experiments were conducted on the task of classifying transcribed speech samples with text samples that were produced by people with AD from those produced by normal subjects. Several experiments were conducted to compare the different areas of lexical complexity, identify the subset of features that help achieve optimal performance, and study the impact of the size of the input on the classification. To evaluate the generalization of the models built on narrative speech, two generalization tests were conducted using written data from two British authors, Iris Murdoch and Agatha Christie, and the transcription of some speeches by former President Ronald Reagan. Results: using lexical features only, state-of-the-art classification, F1 and accuracies, of over 91% were achieved in categorizing language samples produced by individuals with AD from the ones produced by healthy control subjects. This confirms the substantial impact of AD on lexicon processing.
SINC: Self-Supervised In-Context Learning for Vision-Language Tasks

Large Pre-trained Transformers exhibit an intriguing capacity for in-context learning. Without gradient updates, these models can rapidly construct new predictors from demonstrations presented in the inputs. Recent works promote this ability in the vision-language domain by incorporating visual information into large language models that can already make in-context predictions. However, these methods could inherit issues in the language domain, such as template sensitivity and hallucination. Also, the scale of these language models raises a significant demand for computations, making learning and operating these models resource-intensive. To this end, we raise a question: "How can we enable in-context learning without relying on the intrinsic in-context ability of large language models? ". To answer it, we propose a succinct and general framework, Self-supervised IN-Context learning (SINC), that introduces a meta-model to learn on self-supervised prompts consisting of tailored demonstrations. The learned models can be transferred to downstream tasks for making in-context predictions on-the-fly. Extensive experiments show that SINC outperforms gradient-based methods in various vision-language tasks under few-shot settings. Furthermore, the designs of SINC help us investigate the benefits of in-context learning across different tasks, and the analysis further reveals the essential components for the emergence of in-context learning in the vision-language domain.
Stress-Strain Relation of Steel-Polypropylene-Blended Fiber-Reinforced Concrete under Uniaxial Cyclic Compression

This paper investigates the cyclic stress-strain behavior of steel-polypropylene-blended fiber-reinforced concrete (BFRC) under uniaxial cyclic compression. A total of 48 prism specimens were tested for different fiber volume fractions and aspect ratios. The results show that the introduction of blended fibers has synergetic effects on improving the cyclic behavior of concrete in terms of peak strength, postpeak ductility, hysteretic energy dissipation, and stiffness degradation. Moreover, the increase in the volume fractions of both steel and polypropylene fibers can lead to a remarkable decrease in plastic strain accumulation. Furthermore, the stiffness degradation ratio as well as the stress deterioration ratio of BFRC can be significantly alleviated in comparison with those of plain concrete, notwithstanding that the degradation amount is insensitive to the variations of fiber parameters. Subsequently, based on the test results, a constitutive model is developed to generalize the cyclic stress-strain responses of BFRC, with the contributions of blended fibers taken into account. The developed model is then verified by independent experimental results and other test data reported in the literature. It is observed that the prediction yields a close estimation of the cyclic compressive behavior of BFRC with varying fiber parameters.
Multi-Task Cross-Modality Attention-Fusion for 2D Object Detection

Accurate and robust object detection is critical for autonomous driving. Image-based detectors face difficulties caused by low visibility in adverse weather conditions. Thus, radar-camera fusion is of particular interest but presents challenges in optimally fusing heterogeneous data sources. To approach this issue, we propose two new radar preprocessing techniques to better align radar and camera data. In addition, we introduce a Multi-Task Cross-Modality Attention-Fusion Network (MCAF-Net) for object detection, which includes two new fusion blocks. These allow for exploiting information from the feature maps more comprehensively. The proposed algorithm jointly detects objects and segments free space, which guides the model to focus on the more relevant part of the scene, namely, the occupied space. Our approach outperforms current state-of-the-art radar-camera fusion-based object detectors in the nuScenes dataset and achieves more robust results in adverse weather conditions and nighttime scenarios.
Supervised Dynamic PCA: Linear Dynamic Forecasting with Many Predictors

This paper proposes a novel dynamic forecasting method using a new supervised Principal Component Analysis (PCA) when a large number of predictors are available. The new supervised PCA provides an effective way to bridge the gap between predictors and the target variable of interest by scaling and combining the predictors and their lagged values, resulting in an effective dynamic forecasting. Unlike the traditional diffusion-index approach, which does not learn the relationships between the predictors and the target variable before conducting PCA, we first re-scale each predictor according to their significance in forecasting the targeted variable in a dynamic fashion, and a PCA is then applied to a re-scaled and additive panel, which establishes a connection between the predictability of the PCA factors and the target variable. Furthermore, we also propose to use penalized methods such as the LASSO approach to select the significant factors that have superior predictive power over the others. Theoretically, we show that our estimators are consistent and outperform the traditional methods in prediction under some mild conditions. We conduct extensive simulations to verify that the proposed method produces satisfactory forecasting results and outperforms most of the existing methods using the traditional PCA. A real example of predicting U.S. macroeconomic variables using a large number of predictors showcases that our method fares better than most of the existing ones in applications. The proposed method thus provides a comprehensive and effective approach for dynamic forecasting in high-dimensional data analysis.
Diffusion Models Beat GANs on Image Classification

While many unsupervised learning models focus on one family of tasks, either generative or discriminative, we explore the possibility of a unified representation learner: a model which uses a single pre-training stage to address both families of tasks simultaneously. We identify diffusion models as a prime candidate. Diffusion models have risen to prominence as a state-of-the-art method for image generation, denoising, inpainting, super-resolution, manipulation, etc. Such models involve training a U-Net to iteratively predict and remove noise, and the resulting model can synthesize high fidelity, diverse, novel images. The U-Net architecture, as a convolution-based architecture, generates a diverse set of feature representations in the form of intermediate feature maps. We present our findings that these embeddings are useful beyond the noise prediction task, as they contain discriminative information and can also be leveraged for classification. We explore optimal methods for extracting and using these embeddings for classification tasks, demonstrating promising results on the ImageNet classification task. We find that with careful feature selection and pooling, diffusion models outperform comparable generative-discriminative methods such as BigBiGAN for classification tasks. We investigate diffusion models in the transfer learning regime, examining their performance on several fine-grained visual classification datasets. We compare these embeddings to those generated by competing architectures and pre-trainings for classification tasks.
Gradient-free training of neural ODEs for system identification and control using ensemble Kalman inversion

Ensemble Kalman inversion (EKI) is a sequential Monte Carlo method used to solve inverse problems within a Bayesian framework. Unlike backpropagation, EKI is a gradient-free optimization method that only necessitates the evaluation of artificial neural networks in forward passes. In this study, we examine the effectiveness of EKI in training neural ordinary differential equations (neural ODEs) for system identification and control tasks. To apply EKI to optimal control problems, we formulate inverse problems that incorporate a Tikhonov-type regularization term. Our numerical results demonstrate that EKI is an efficient method for training neural ODEs in system identification and optimal control problems, with runtime and quality of solutions that are competitive with commonly used gradient-based optimizers.
Trapping $\mathbf{Ba}^+$ with Seven-fold Enhanced Efficiency Utilizing an Autoionizing Resonance

Trapped ions have emerged as a front runner in quantum information processing due to their identical nature, all-to-all connectivity, and high fidelity quantum operations. As current trapped ion technologies are scaled, it will be important to improve the efficiency of loading ions, which is currently the slowest process in operating a trapped ion quantum computer. Here, we compare two isotope-selective photoionization schemes for loading $^{138}\mathrm{Ba}^+$ ions. We show that a two-step photoionization scheme ending in an autoionizing transition increases the ion loading rate nearly an order of magnitude compared to an established technique which does not excite an autoionizing state. The only additional technology required to implement the autoionizing transition is a commercial diode laser. Our technique can be extended to all isotopes of barium, and autoionizing resonances exist in every species currently used for trapped ion quantum processing, making this a promising technique to drastically increase the loading rates for all trapped ion computers.
Modulated Neural ODEs

Neural ordinary differential equations (NODEs) have been proven useful for learning non-linear dynamics of arbitrary trajectories. However, current NODE methods capture variations across trajectories only via the initial state value or by auto-regressive encoder updates. In this work, we introduce Modulated Neural ODEs (MoNODEs), a novel framework that sets apart dynamics states from underlying static factors of variation and improves the existing NODE methods. In particular, we introduce $\textit{time-invariant modulator variables}$ that are learned from the data. We incorporate our proposed framework into four existing NODE variants. We test MoNODE on oscillating systems, videos and human walking trajectories, where each trajectory has trajectory-specific modulation. Our framework consistently improves the existing model ability to generalize to new dynamic parameterizations and to perform far-horizon forecasting. In addition, we verify that the proposed modulator variables are informative of the true unknown factors of variation as measured by $R^2$ scores.
THE RADIATIVE

The criticality problem in nuclear engineering asks for the principal eigen-pair of a Boltzmann operator describing neutron transport in a reactor core. Being able to reliably design, and control such reactors requires assessing these quantities within quantiﬁable accuracy tolerances. In this paper we propose a paradigm that deviates from the common practice of approximately solving the corresponding spectral problem with a ﬁxed, presumably suﬃciently ﬁne discretization. Instead, the present approach is based on ﬁrst contriving iterative schemes, formulated in function space, that are shown to converge at a quantitative rate without assuming any a priori excess regularity properties, and that exploit only properties of the optical parameters in the underlying radiative transfer model. We develop the analytical and numerical tools for approximately realizing each iteration step withing judiciously chosen accuracy tolerances, veriﬁed by a posteriori estimates, so as to still warrant quantiﬁable convergence to the exact eigen-pair. This is carried out in full ﬁrst for a Newton scheme. Since this is only locally convergent we analyze in addition the convergence of a power iteration in function space to produce suﬃciently accurate initial guesses. Here we have to deal with intrinsic diﬃculties posed by compact but unsymmetric operators preventing standard arguments used in the ﬁnite dimensional case. Our main point is that we can avoid any condition on an initial guess to be already in a small neighborhood of the exact solution. We close with a discussion of remaining intrinsic obstructions to a certiﬁable numerical implementation, mainly related to not knowing the gap between the principal eigenvalue and the next smaller one in modulus.
A cyclostratigraphic framework of the Upper Carboniferous Westoe and Cleaver formations in the southern North Sea Basin as a methodology for stratigraphic reservoir characterisation

Abstract Orbital driven climate control on sedimentation produces regional, stratigraphically repetitive characters and so cyclostratigraphic correlation can improve correlation and identify stratigraphic trends in borehole sections. This concept is commonly used to correlate marine and lacustrine strata. However, in the alluvial domain, its use is more challenging because internal, local dynamics controlling sedimentation may interfere with the expression of cyclic climate forcing. Intervals of low net-to-gross may be important for successful application in this domain as they tend to better document regional changes. This study applies climate-based stratigraphic correlation concepts to improve well correlations, characterise vertical sand distribution, and identify potential reservoir targets in a generally low net-to-gross interval. Coarsening upward sedimentary repetitions (cyclothems) are identified and correlated with high certainty in nineteen well sections in the upper Carboniferous Westoe and Cleaver formations of the Silverpit Basin. Local sedimentary dynamics provide variability in the character of the cyclothems and several types of cyclothem are classified. Correlation of sections using cyclothems recognised on wireline logs is done twice: once manually and once semi-automatically. The semi-automated correlation is based on calculation of deviation curves which depict stratigraphic changes that are less dependent on absolute wireline values and follow vertical trends more clearly. The correlations provide composite stratigraphies that are analysed using vertical proportions curves. Both approaches yield similar results in terms of stratigraphic trends. However, for detailed correlation of wells, the manual correlation is better at accounting for any local variability within the system. The same two zones of higher net-to-gross ratios are found using both correlation methods. These are linked to palaeoclimatic changes driven by long eccentricity and the proposed climate stratigraphic model has predictive value for identifying sandstone occurrence. The climate-based stratigraphic correlation improves the assessment reservoir distribution and properties on small (10–20 m thickness) and large (100–200 m thickness) stratigraphical scales.
Mean Zonal Flows Induced by a Vertically Propagating Rossby Wave Packet

The second order mean motion induced around an internal Rossby wave packet, having an infinite zonal length and propagating vertically in an inviscid Boussinesq fluid at rest in a channel, is discussed, and the validity of photon analogy to such a wave packet is examined. It is shown that the second order mean zonal momentum averaged in the meridional direction is just equal to the wave momentum E/C (where E and C are the wave energy and the phase velocity in the zonal direction respectively). This guarantees the validity of photon analogy to the wave packet, and also implies that the treatment done rather intuitively in the previous paper by the author (Uryu, 1974) is essentially correct. It is shown that, to the first order in * (where * is a small parameter characterizing the slowness of variation of wave amplitude), the vertical component of Lagrangian mean velocity is zero, as a result of the cancellation between the Stokes drift and the Eulerian mean velocity. The case of propagation in a shear flow is also treated, and it is shown that the absorption of wave at the critical level occurs as a result that the wave momentum is stored up in the mean zonal flow there. It is also shown that the results obtained in case of Rossby wave packet can be obtained without any essential change also in case of internal gravity wave packet under the same situation.