Miracle-dz/UniFAR-FTUs huggingface dataset:
A Comparative Study of Pre-trained Encoders for Low-Resource Named Entity Recognition

Pre-trained language models (PLM) are effective components of few-shot named entity recognition (NER) approaches when augmented with continued pre-training on task-specific out-of-domain data or fine-tuning on in-domain data. However, their performance in low-resource scenarios, where such data is not available, remains an open question. We introduce an encoder evaluation framework, and use it to systematically compare the performance of state-of-the-art pre-trained representations on the task of low-resource NER. We analyze a wide range of encoders pre-trained with different strategies, model architectures, intermediate-task fine-tuning, and contrastive learning. Our experimental results across ten benchmark NER datasets in English and German show that encoder performance varies significantly, suggesting that the choice of encoder for a specific low-resource scenario needs to be carefully evaluated.
Auditory-Based Data Augmentation for end-to-end Automatic Speech Recognition

End-to-end models have achieved significant improvement on automatic speech recognition. One common method to improve performance of these models is expanding the data-space through data augmentation. Meanwhile, human auditory inspired front-ends have also demonstrated improvement for automatic speech recognisers. In this work, a well-verified auditory-based model, which can simulate various hearing abilities, is investigated for the purpose of data augmentation for end-to-end speech recognition. By introducing the auditory model into the data augmentation process, end-to-end systems are encouraged to ignore variation from the signal that cannot be heard and thereby focus on robust features for speech recognition. Two mechanisms in the auditory model, spectral smearing and loudness recruitment, are studied on the LibriSpeech dataset with a transformer-based end-to-end model. The results show that the proposed augmentation methods can bring statistically significant improvement on the performance of the state-of-the-art SpecAugment.
Team ÚFAL at CMCL 2022 Shared Task: Figuring out the correct recipe for predicting Eye-Tracking features using Pretrained Language Models

Eye-Tracking data is a very useful source of information to study cognition and especially language comprehension in humans. In this paper, we describe our systems for the CMCL 2022 shared task on predicting eye-tracking information. We describe our experiments withpretrained models like BERT and XLM and the different ways in which we used those representations to predict four eye-tracking features. Along with analysing the effect of using two different kinds of pretrained multilingual language models and different ways of pooling the token-level representations, we also explore how contextual information affects the performance of the systems. Finally, we also explore if factors like augmenting linguistic information affect the predictions. Our submissions achieved an average MAE of 5.72 and ranked 5th in the shared task. The average MAE showed further reduction to 5.25 in post task evaluation.
gTLO: A Generalized and Non-linear Multi-Objective Deep Reinforcement Learning Approach

In real-world decision optimization, often multiple competing objectives must be taken into account. Following classical reinforcement learning, these objectives have to be combined into a single reward function. In contrast, multi-objective reinforcement learning (MORL) methods learn from vectors of per-objective rewards instead. In the case of multi-policy MORL, sets of decision policies for various preferences regarding the conflicting objectives are optimized. This is especially important when target preferences are not known during training or when preferences change dynamically during application. While it is, in general, straightforward to extend a single-objective reinforcement learning method for MORL based on linear scalarization, solutions that are reachable by these methods are limited to convex regions of the Pareto front. Non-linear MORL methods like Thresholded Lexicographic Ordering (TLO) are designed to overcome this limitation. Generalized MORL methods utilize function approximation to generalize across objective preferences and thereby implicitly learn multiple policies in a data-efficient manner, even for complex decision problems with high-dimensional or continuous state spaces. In this work, we propose \textit{generalized Thresholded Lexicographic Ordering} (gTLO), a novel method that aims to combine non-linear MORL with the advantages of generalized MORL. We introduce a deep reinforcement learning realization of the algorithm and present promising results on a standard benchmark for non-linear MORL and a real-world application from the domain of manufacturing process control.
Assessment of Massively Multilingual Sentiment Classifiers

Models are increasing in size and complexity in the hunt for SOTA. But what if those 2%increase in performance does not make a difference in a production use case? Maybe benefits from a smaller, faster model outweigh those slight performance gains. Also, equally good performance across languages in multilingual tasks is more important than SOTA results on a single one. We present the biggest, unified, multilingual collection of sentiment analysis datasets. We use these to assess 11 models and 80 high-quality sentiment datasets (out of 342 raw datasets collected) in 27 languages and included results on the internally annotated datasets. We deeply evaluate multiple setups, including fine-tuning transformer-based models for measuring performance. We compare results in numerous dimensions addressing the imbalance in both languages coverage and dataset sizes. Finally, we present some best practices for working with such a massive collection of datasets and models for a multi-lingual perspective.
Lyapunov-Guided Representation of Recurrent Neural Network Performance

Recurrent Neural Networks (RNN) are ubiquitous computing systems for sequences and multivariate time series data. While several robust architectures of RNN are known, it is unclear how to relate RNN initialization, architecture, and other hyperparameters with accuracy for a given task. In this work, we propose to treat RNN as dynamical systems and to correlate hyperparameters with accuracy through Lyapunov spectral analysis, a methodology specifically designed for nonlinear dynamical systems. To address the fact that RNN features go beyond the existing Lyapunov spectral analysis, we propose to infer relevant features from the Lyapunov spectrum with an Autoencoder and an embedding of its latent representation (AeLLE). Our studies of various RNN architectures show that AeLLE successfully correlates RNN Lyapunov spectrum with accuracy. Furthermore, the latent representation learned by AeLLE is generalizable to novel inputs from the same task and is formed early in the process of RNN training. The latter property allows for the prediction of the accuracy to which RNN would converge when training is complete. We conclude that representation of RNN through Lyapunov spectrum along with AeLLE provides a novel method for organization and interpretation of variants of RNN architectures.
Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts

Despite the advances in digital healthcare systems offering curated structured knowledge, much of the critical information still lies in large volumes of unlabeled and unstructured clinical texts. These texts, which often contain protected health information (PHI), are exposed to information extraction tools for downstream applications, risking patient identification. Existing works in de-identification rely on using large-scale annotated corpora in English, which often are not suitable in real-world multilingual settings. Pre-trained language models (LM) have shown great potential for cross-lingual transfer in low-resource settings. In this work, we empirically show the few-shot cross-lingual transfer property of LMs for named entity recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke domain. We annotate a gold evaluation dataset to assess few-shot setting performance where we only use a few hundred labeled examples for training. Our model improves the zero-shot F1-score from 73.7% to 91.2% on the gold evaluation set when adapting Multilingual BERT (mBERT) (CITATION) from the MEDDOCAN (CITATION) corpus with our few-shot cross-lingual target corpus. When generalized to an out-of-sample test set, the best model achieves a human-evaluation F1-score of 97.2%.
Fast evaluation of B-spline functions and rendering of multiple B-spline curves using linear-time algorithm for computing the Bernstein-Bézier coefficients of B-spline functions

A new diﬀerential-recurrence relation for the B-spline functions of the same degree is proved. From this relation, a recursive method of computing the coeﬃcients of B-spline functions of degree m in the Bernstein-B´ezier form is derived. Its complexity is proportional to the number of coeﬃcients in the case of coincident boundary knots. This means that, asymptotically, the algorithm is optimal. In other cases, the complexity is increased by at most O ( m 3 ). When the Bernstein-B´ezier coeﬃcients of B-spline basis functions are known, it is possible to compute any B-spline function in linear time with respect to its degree by performing the geometric algorithm proposed recently by the authors. Using a similar approach, one can also convert a d -dimensional B-spline curve of degree m over one knot span to a B´ezier curve in O ( m 2 ) time and then evaluate it in O ( md ) time. Since one only needs to convert each knot span once, this algorithm scales well when evaluating the B-spline curve at multiple points, e.g., in order to render it. When evaluating many B-spline curves at multiple points, such approach has lower computational complexity than using the de Boor-Cox algorithm. The problem of ﬁnding the coeﬃcients of the B-spline functions in the power basis can be solved similarly.
Predicting aggregation of sequence-deﬁned macromolecules with Recurrent Neural Networks

Self-assembly of dilute sequence-deﬁned macromolecules is a complex phenomenon in which the local arrangement of chemical moieties leads to the formation of a long-range structure. The dependence of this structure on the sequence necessarily implies that a mapping between the two exists, yet it has been diﬃcult to model so far. Predicting the aggregation behavior of these macromolecules is challenging due to the lack of eﬀective order parameters, a vast design space, inherent variability, and high computational costs associated with currently available simulation tech-niques. Here, we accurately predict the morphology of aggregates self-assembled from sequence-deﬁned macromolecules using supervised machine learning. We ﬁnd that regression models with implicit representation learning perform signiﬁcantly better than those based on engineered features such as k -mer counting, and a Recurrent-Neural-Network-based regressor performs the best. Further, we demonstrate the high-throughput screening of monomer sequences using the regression model to identify candidates for self-assembly into selected morphologies. Our strategy is shown to successfully identify multiple suitable sequences in every test we performed, so we hope the insights gained here can be extended to other increasingly complex design scenarios in the future, such as the design of sequences under polydispersity and at varying environmental conditions.
Confusing Image Quality Assessment: Toward Better Augmented Reality Experience

With the development of multimedia technology, Augmented Reality (AR) has become a promising next-generation mobile platform. The primary value of AR is to promote the fusion of digital contents and real-world environments, however, studies on how this fusion will influence the Quality of Experience (QoE) of these two components are lacking. To achieve better QoE of AR, whose two layers are influenced by each other, it is important to evaluate its perceptual quality first. In this paper, we consider AR technology as the superimposition of virtual scenes and real scenes, and introduce visual confusion as its basic theory. A more general problem is first proposed, which is evaluating the perceptual quality of superimposed images, i.e., confusing image quality assessment. A ConFusing Image Quality Assessment (CFIQA) database is established, which includes 600 reference images and 300 distorted images generated by mixing reference images in pairs. Then a subjective quality perception experiment is conducted towards attaining a better understanding of how humans perceive the confusing images. Based on the CFIQA database, several benchmark models and a specifically designed CFIQA model are proposed for solving this problem. Experimental results show that the proposed CFIQA model achieves state-of-the-art performance compared to other benchmark models. Moreover, an extended ARIQA study is further conducted based on the CFIQA study. We establish an ARIQA database to better simulate the real AR application scenarios, which contains 20 AR reference images, 20 background (BG) reference images, and 560 distorted images generated from AR and BG references, as well as the correspondingly collected subjective quality ratings. Three types of full-reference (FR) IQA benchmark variants are designed to study whether we should consider the visual confusion when designing corresponding IQA algorithms. An ARIQA metric is finally proposed for better evaluating the perceptual quality of AR images. Experimental results demonstrate the good generalization ability of the CFIQA model and the state-of-the-art performance of the ARIQA model. The databases, benchmark models, and proposed metrics are available at: https://github.com/DuanHuiyu/ARIQA.
Evaluating the Adversarial Robustness for Fourier Neural Operators

In recent years, Machine-Learning (ML)-driven approaches have been widely used in scientific discovery domains. Among them, the Fourier Neural Operator (FNO) (Li et al., 2020) was the first to simulate turbulent flow with zero-shot super-resolution and superior accuracy, which significantly improves the speed when compared to traditional partial differential equation (PDE) solvers. To inspect the trustworthiness, we provide the first study on the adversarial robustness of scientific discovery models by generating adversarial examples for FNO, based on norm-bounded data input perturbations. Evaluated on the mean squared error between the FNO model’s output and the PDE solver’s output, our results show that the model’s robustness degrades rapidly with increasing perturbation levels, particularly in non-simplistic cases like the 2D Darcy and the Navier cases. Our research provides a sensitivity analysis tool and evaluation principles for assessing the adversarial robustness of ML-based scientific discovery models.
On-chip distribution of quantum information using traveling phonons

Distributing quantum entanglement on a chip is a crucial step toward realizing scalable quantum processors. Using traveling phonons—quantized guided mechanical wave packets—as a medium to transmit quantum states is now gaining substantial attention due to their small size and low propagation speed compared to other carriers, such as electrons or photons. Moreover, phonons are highly promising candidates to connect heterogeneous quantum systems on a chip, such as microwave and optical photons for long-distance transmission of quantum states via optical fibers. Here, we experimentally demonstrate the feasibility of distributing quantum information using phonons by realizing quantum entanglement between two traveling phonons and creating a time-bin–encoded traveling phononic qubit. The mechanical quantum state is generated in an optomechanical cavity and then launched into a phononic waveguide in which it propagates for around 200 micrometers. We further show how the phononic, together with a photonic qubit, can be used to violate a Bell-type inequality.
Cello: Efficient Computer Systems Optimization with Predictive Early Termination and Censored Regression

Sample-efficient machine learning (SEML) has been widely applied to find optimal latency and power tradeoffs for configurable computer systems. Instead of randomly sampling from the configuration space, SEML reduces the search cost by dramatically reducing the number of configurations that must be sampled to optimize system goals (e.g., low latency or energy). Nevertheless, SEML only reduces one component of cost -- the total number of samples collected -- but does not decrease the cost of collecting each sample. Critically, not all samples are equal; some take much longer to collect because they correspond to slow system configurations. This paper present Cello, a computer systems optimization framework that reduces sample collection costs -- especially those that come from the slowest configurations. The key insight is to predict ahead of time whether samples will have poor system behavior (e.g., long latency or high energy) and terminate these samples early before their measured system behavior surpasses the termination threshold, which we call it predictive early termination. To predict the future system behavior accurately before it manifests as high runtime or energy, Cello uses censored regression to produces accurate predictions for running samples. We evaluate Cello by optimizing latency and energy for Apache Spark workloads. We give Cello a fixed amount of time to search a combined space of hardware and software configuration parameters. Our evaluation shows that compared to the state-of-the-art SEML approach in computer systems optimization, Cello improves latency by 1.19X for minimizing latency under a power constraint, and improves energy by 1.18X for minimizing energy under a latency constraint.
Submodular Maximization Subject to Matroid Intersection on the Fly

Despite a surge of interest in submodular maximization in the data stream model, there remain significant gaps in our knowledge about what can be achieved in this setting, especially when dealing with multiple constraints. In this work, we nearly close several basic gaps in submodular maximization subject to $k$ matroid constraints in the data stream model. We present a new hardness result showing that super polynomial memory in $k$ is needed to obtain an $o(k / \log k)$-approximation. This implies near optimality of prior algorithms. For the same setting, we show that one can nevertheless obtain a constant-factor approximation by maintaining a set of elements whose size is independent of the stream size. Finally, for bipartite matching constraints, a well-known special case of matroid intersection, we present a new technique to obtain hardness bounds that are significantly stronger than those obtained with prior approaches. Prior results left it open whether a $2$-approximation may exist in this setting, and only a complexity-theoretic hardness of $1.91$ was known. We prove an unconditional hardness of $2.69$.
Deflection and gravitational lensing of null and timelike signals in the Kiselev black hole spacetime in the weak field limit

In this work we study the deflection and gravitational lensing of null and timelike signals in the Kiselev spacetime in the weak field limit, to investigate the effects of the equation of state parameter ω and the matter amount parameter α. In doing this, we extend a perturbative method previously developed for asymptotically flat spacetimes whose metric functions have integer-power asymptotic expansions to the case that may or may not be asymptotically flat but with non-integer power expansions. It is found that in the asymptotically flat case (−1/3 < ω < 0) the deflection angles are expressable as quasi-power series of the dimensionless quantities M/b, b/r s,d and α/M 1+3ω where M, b, r s,d are respectively the lens mass, impact parameter and source/detector radius. A similar series exists for the non-asymptotically flat case of (−1 < ω < −1/3), but with the closest radius r 0 replacing b. In the asymptotically flat (or non-flat) case, the increase of α or decrease of ω will increase (or increase) the deflection angle. Since the obtained deflection angles naturally take into account the finite distance effect of the source and the detector, we can establish an exact gravitational lensing equation, from which the apparent angles of the images and their magnifications are solved. It is found that generally for the asymptotically flat case, increasing α or decreasing ω will increase the apparent angles of the images. While for the non-asymptotically flat case, increasing α or ω will both lead to smaller apparent angles.
Neural networks embrace learned diversity

Diversity conveys advantages in nature, yet homogeneous neurons typically comprise the layers of artiﬁcial neural networks. Here we construct neural networks from neurons that learn their own activation functions, quickly diversify, and subsequently outperform their homogeneous counter-parts. Sub-networks instantiate the neurons, which meta-learn especially eﬃcient sets of nonlinear responses. Such learned diversity provides examples of dynamical systems selecting diversity over uniformity and elucidates the role of diversity in natural and artiﬁcial systems.
Iterative Depth-First Search for Fully Observable Non-Deterministic Planning

Fully Observable Non-Deterministic (FOND) planning models uncertainty through actions with non-deterministic effects. Existing FOND planning algorithms are effective and employ a wide range of techniques. However, most of the existing algorithms are not robust for dealing with both non-determinism and task size. In this paper, we develop a novel iterative depth-first search algorithm that solves FOND planning tasks and produces strong cyclic policies. Our algorithm is explicitly designed for FOND planning, addressing more directly the non-deterministic aspect of FOND planning, and it also exploits the benefits of heuristic functions to make the algorithm more effective during the iterative searching process. We compare our proposed algorithm to well-known FOND planners, and show that it has robust performance over several distinct types of FOND domains considering different metrics.
Maxwell Matter Waves

Maxwell matter waves emerge from a perspective, complementary to de Broglie’s, that matter is fundamentally a wave phenomenon whose particle aspects are revealed by quantum mechanics. Their quantum mechanical description is derived through the introduction of a matter vector potential, having frequency ω 0 , to Schrodinger’s equation for a massive particle. Maxwell matter waves are then seen to be coherent excitations of a single-mode of the matter-wave ﬁeld. In the classical regime, their mechanics is captured by a matter analog of Maxwell’s equations for the electromagnetic ﬁeld. As such, Maxwell matter waves enable a spectrum of systems that have useful optical analogs, such as resonant matter-wave interferometric sensors and matter-wave parametric oscillators. These waves are associated with a wavelength that is tied to the drive frequency ω 0 rather than to the massive particle’s energy, as is ordinarily the case with de Broglie matter waves. As a result, simple interferometric measurements lead to diﬀerent outcomes for the two types of waves. While their apparent departure from de Broglie character is surprising, Maxwell matter waves are wholly consistent with quantum mechanics.