Miracle-dz/UniFAR-FTUs huggingface dataset:
Catch the "Tails" of BERT

Recently, contextualized word embeddings outperform static word embeddings on many NLP tasks. However, we still do not know much about the mechanism inside these representations. Do they have any common patterns? If so, where do these patterns come from? We find that almost all the contextualized word vectors of BERT and RoBERTa have a common pattern. For BERT, the $557^{th}$ element is always the smallest. For RoBERTa, the $588^{th}$ element is always the largest, and the $77^{th}$ element is the smallest. We call them "tails" of models. We introduce a new neuron-level method to analyze where these "tails" come from. We find that these "tails" are closely related to the positional information. We also investigate what will happen if we "cutting the tails" (zero-out). Our results show that "tails" are the major cause of anisotropy of vector space. After "cutting the tails", a word's different vectors are more similar to each other. The internal representations have a better ability to distinguish a word's different senses with the word-in-context (WiC) dataset. The performance on the word sense disambiguation task is better for BERT and unchanged for RoBERTa. We can also better induce phrase grammar from the vector space. These suggest that "tails" are less related to the sense and syntax information in vectors. These findings provide insights into the inner workings of contextualized word vectors.
Simplifying Backscatter Deployment: Full-Duplex LoRa Backscatter

Due to the practical challenges in the deployment of existing half-duplex systems, the promise of ubiquitous backscatter connectivity has eluded us. To address this, we design the first long-range full-duplex LoRa backscatter reader. We leverage existing LoRa chipsets as transceivers and use a microcontroller in combination with inexpensive passive elements including a hybrid coupler, inductors, tunable capacitors, and resistors to achieve 78 dB of self-interference cancellation and build a low-cost, long-range, and small-form-factor Full-Duplex LoRa Backscatter reader. We evaluate our system in various deployments and show that we can successfully communicate with a backscatter tag at distances of up to 300 ft in line of sight, and through obstacles, such as walls and cubicles, in a 4,000 ft$^2$ office area. We reconfigure our reader to conform to the size and power requirements of a smartphone, and demonstrate communication with a contact-lens-form-factor prototype device. Finally, we attach our reader to a drone and demonstrate backscatter sensing for precision agriculture with an instantaneous coverage of 7,850 ft$^2$.
Rethinking the Value of Transformer Components

Transformer becomes the state-of-the-art translation model, while it is not well studied how each intermediate component contributes to the model performance, which poses significant challenges for designing optimal architectures. In this work, we bridge this gap by evaluating the impact of individual component (sub-layer) in trained Transformer models from different perspectives. Experimental results across language pairs, training strategies, and model capacities show that certain components are consistently more important than the others. We also report a number of interesting findings that might help humans better analyze, understand and improve Transformer models. Based on these observations, we further propose a new training strategy that can improves translation performance by distinguishing the unimportant components in training.
Triangular research and innovation collaborations in the European area

In the current study, we examine the multiplex network of patents and European Framework Programmes (FPs) aiming to uncover temporal variations in the formation patterns of triangles (a fully connected network between any three nodes). More specifically, the multiplex network consists of two layers whose nodes are the NUTS2 regions. On the first layer we depict the regions of the inventors that collaborated for the creation of a patent, and on the second those of the scientists in European Framework Programme (FP) funded projects. A link between two nodes exists when scientists or inventors from different regions collaborate. We split the network temporally into 28 shorter sub-networks with a span of 6 years each, and calculate the number of triangles formed at the end of the 6-year period. Next, we shuffle the data creating again 28 six-year randomized networks, in order to identify whether there is a hidden mechanism that favors a non-random behavior. Real and shuffled data are compared using a z-score, a measure of the differences of standard deviations between them. In addition, we repeat the same analysis using the clustering coefficient, which is the number of triangles over the number of triples (possible triangles). The results show that triangular FP collaborations tend to be favored over random ones, while in patents the case is strongly the opposite. Furthermore, results using triangles tend to be more comprehensive as opposed to those of the clustering coefficient. Finally, we identify which NUTS2 regions frequently exhibit a high clustering coefficient in either of the layers, and we present a map with these values for all regions. The results of this research can help policy making organizations understand the spatial dimension of subsidized research and patented innovation collaboration networks.
Upper Extremity Load Reduction for Lower Limb Exoskeleton Trajectory Generation Using Ankle Torque Minimization

Recently, the lower limb exoskeletons which provide mobility for paraplegic patients to support their daily life have drawn much attention. However, the pilots are required to apply excessive force through a pair of crutches to maintain balance during walking. This paper proposes a novel gait trajectory generation algorithm for exoskeleton locomotion on flat ground and stair which aims to minimize the force applied by the pilot without increasing the degree of freedom (DoF) of the system. First, the system is modelled as a five-link mechanism dynamically for torque computing. Then, an optimization approach is used to generate the trajectory minimizing the ankle torque which is correlated to the supporting force. Finally, experiment is conducted to compare the different gait generation algorithms through measurement of ground reaction force (GRF) applied on the crutches.
ACED: Accelerated Computational Electrochemical systems Discovery

Large-scale electrification is vital to addressing the climate crisis, but many engineering challenges remain to fully electrifying both the chemical industry and transportation. In both of these areas, new electrochemical materials and systems will be critical, but developing these systems currently relies heavily on computationally expensive first-principles simulations as well as human-time-intensive experimental trial and error. We propose to develop an automated workflow that accelerates these computational steps by introducing both automated error handling in generating the first-principles training data as well as physics-informed machine learning surrogates to further reduce computational cost. It will also have the capacity to include automated experiments "in the loop" in order to dramatically accelerate the overall materials discovery pipeline.
Online Sparse Reinforcement Learning

We investigate the hardness of online reinforcement learning in fixed horizon, sparse linear Markov decision process (MDP), with a special focus on the high-dimensional regime where the ambient dimension is larger than the number of episodes. Our contribution is two-fold. First, we provide a lower bound showing that linear regret is generally unavoidable in this case, even if there exists a policy that collects well-conditioned data. The lower bound construction uses an MDP with a fixed number of states while the number of actions scales with the ambient dimension. Note that when the horizon is fixed to one, the case of linear stochastic bandits, the linear regret can be avoided. Second, we show that if the learner has oracle access to a policy that collects well-conditioned data then a variant of Lasso fitted Q-iteration enjoys a nearly dimension-free regret of $\tilde{O}( s^{2/3} N^{2/3})$ where $N$ is the number of episodes and $s$ is the sparsity level. This shows that in the large-action setting, the difficulty of learning can be attributed to the difficulty of finding a good exploratory policy.
A compact high frequency voltage amplifier for superconductor–semiconductor logic interface

The many advantages of cryogenically cooled single-flux quantum (SFQ) circuits imply that the high speed and low voltage output signals must be amplified and interfaced with standard electronics. State-of-the-art low-noise and wide-band amplifiers are required to interface with room temperature electronics. One solution is to place preamplifiers at the cryogenic stage, which requires specific semiconductor design and fabrication. However, a more viable and energy-efficient approach is to integrate the pulsed logic circuit output stages with on-chip superconductor preamplifiers. We designed, fabricated, and tested an on-chip compact voltage multiplier integrated with the output stage of SFQ circuits to increase the voltage amplitude of SFQ pulses. The circuit is designed with the same technology as the logic circuit hence its noise level is lower, and it works at higher frequencies compared to complementary metal-oxide semi-conductor amplifiers and due to quantized nature of it there is no added noise. The fabricated circuit has a compact size of 160 µm × 320 µm and provides about 10 dB gain with measured 600 µV output voltage at frequencies up to ∼25 GHz in simulations. By stacking more levels, over 20 dB gain at circuit level is achievable as shown in simulations. Moreover the gain of the superconductor voltage amplifier is quantized and programmable.
Predictive Analysis of Diabetic Retinopathy with Transfer Learning

With the prevalence of diabetes, Diabetes Mellitus Retinopathy (DR) is becoming a major health problem across the world. The long-term medical complications arising due to DR significantly affect the patient as well as the society, as the disease mostly affects young and productive individuals. Early detection and treatment can help reduce the extent of damage to the patients. The rise of Convolutional Neural Networks for predictive analysis in the medical field paves the way for a robust solution to DR detection. This paper studies the performance of several highly efficient and scalable CNN architectures for Diabetic Retinopathy Classification with the help of Transfer Learning. The research focuses on VGG16, Resnet50 V2, and EfficientNet B0 models. The classification performance is analyzed using several performance measures including True Positive Rate, False Positive Rate, Accuracy, etc. Also, several performance graphs are plotted for visualizing the architecture performance including Confusion Matrix, ROC Curve, etc. The results indicate that Transfer Learning with ImageNet weights using VGG 16 model demonstrates the best classification performance with an accuracy of 95%. It is closely followed by ResNet50 V2 architecture with an accuracy of 93%. This paper shows that predictive analysis of DR from retinal images is achieved with Transfer Learning on Convolutional Neural Networks.
Multilevel Ensemble Kalman-Bucy Filters

In this article we consider the linear filtering problem in continuous-time. We develop and apply multilevel Monte Carlo (MLMC) strategies for ensemble Kalman--Bucy filters (EnKBFs). These filters can be viewed as approximations of conditional McKean--Vlasov-type diffusion processes. They are also interpreted as the continuous-time analogue of the ensemble Kalman filter, which has proven to be successful due to its applicability and computational cost. We prove that our multilevel EnKBF can achieve a mean square error (MSE) of $\mathcal{O}(\epsilon^2), \ \epsilon>0$ with a cost of order $\mathcal{O}(\epsilon^{-2}\log(\epsilon)^2)$. This implies a reduction in cost compared to the (single level) EnKBF which requires a cost of $\mathcal{O}(\epsilon^{-3})$ to achieve an MSE of $\mathcal{O}(\epsilon^2)$. In order to prove this result we provide a Monte Carlo convergence and approximation bounds associated to time-discretized EnKBFs. To the best of our knowledge, these are the first set of Monte-Carlo type results associated with the discretized EnKBF. We test our theory on a linear problem, which we motivate through a relatively high-dimensional example of order $\sim 10^3$.
Ecology of the cold-adapted species Nebria germari (Coleoptera: Carabidae): the role of supraglacial stony debris as refugium during the current interglacial period

In the current scenario of climate change, cold-adapted insects are among the most threatened organisms in high-altitude habitats of the Alps. Upslope shifts and changes in phenology are two of the most investigated responses to climate change, but there is an increasing interest in evaluating the presence of high-altitude landforms acting as refugia. Nebria germari Heer, 1837 (Coleoptera: Carabidae) is a hygrophilic and cold-adapted species that still exhibits large populations on supraglacial debris of the Eastern Alps. This work aims at describing the ecology and phenology of the populations living on supraglacial debris. To this end, we analysed the populations from three Dolomitic glaciers whose surfaces are partially covered by stony debris. We found that supraglacial debris is characterised by more stable colder and wetter conditions than the surrounding debris slopes and by a shorter snow-free period. The populations found on supraglacial debris were spring breeders, differently from those documented in the 1980s on Dolomitic high alpine grasslands, which were reported as autumn breeders. Currently, Nebria germari seems, therefore, to find a suitable habitat on supraglacial debris, where micrometeorological conditions are appropriate for its life-cycle and competition and predation are reduced.
Contrastive Topographic Models: Energy-based density models applied to the understanding of sensory coding and cortical topography

We address the problem of building theoretical models that help elucidate the function of the visual brain at computational/algorithmic and structural/mechanistic levels. We seek to understand how the receptive fields and topographic maps found in visual cortical areas relate to underlying computational desiderata. We view the development of sensory systems from the popular perspective of probability density estimation; this is motivated by the notion that an effective internal representational scheme is likely to reflect the statistical structure of the environment in which an organism lives. We apply biologically based constraints on elements of the model. The thesis begins by surveying the relevant literature from the fields of neurobiology, theoretical neuroscience, and machine learning. After this review we present our main theoretical and algorithmic developments: we propose a class of probabilistic models, which we refer to as "energy-based models", and show equivalences between this framework and various other types of probabilistic model such as Markov random fields and factor graphs; we also develop and discuss approximate algorithms for performing maximum likelihood learning and inference in our energy based models. The rest of the thesis is then concerned with exploring specific instantiations of such models. By performing constrained optimisation of model parameters to maximise the likelihood of appropriate, naturalistic datasets we are able to qualitatively reproduce many of the receptive field and map properties found in vivo, whilst simultaneously learning about statistical regularities in the data.
Frequency–time analysis, low-rank reconstruction and denoising of turbulent flows using SPOD

Abstract Four different applications of spectral proper orthogonal decomposition (SPOD) are demonstrated on large-eddy simulation data of a turbulent jet. These are: low-rank reconstruction, denoising, frequency–time analysis and prewhitening. We demonstrate SPOD-based flow-field reconstruction using direct inversion of the SPOD algorithm (frequency-domain approach) and propose an alternative approach based on projection of the time series data onto the modes (time-domain approach). We further present a SPOD-based denoising strategy that is based on hard thresholding of the SPOD eigenvalues. The proposed strategy achieves significant noise reduction while facilitating drastic data compression. In contrast to standard methods of frequency–time analysis such as wavelet transform, a proposed SPOD-based approach yields a spectrogram that characterises the temporal evolution of spatially coherent flow structures. A convolution-based strategy is proposed to compute the time-continuous expansion coefficients. When applied to the turbulent jet data, SPOD-based frequency–time analysis reveals that the intermittent occurrence of large-scale coherent structures is directly associated with high-energy events. This work suggests that the time-domain approach is preferable for low-rank reconstruction of individual snapshots, and the frequency-domain approach for denoising and frequency–time analysis.
Topology Inference for Multi-agent Cooperation under Unmeasurable Latent Input

Topology inference is a crucial problem for cooperative control in multi-agent systems. Different from most prior works, this paper is dedicated to inferring the directed network topology from the observations that consist of a single, noisy and finite time-series system trajectory, where the cooperation dynamics is stimulated with the initial network state and the unmeasurable latent input. The unmeasurable latent input refers to intrinsic system signal and extrinsic environment interference. Considering the time-invariant/varying nature of the input, we propose two-layer optimization-based and iterative estimation based topology inference algorithms (TO-TIA and IE-TIA), respectively. TO-TIA allows us to capture the separability of global agent state and eliminates the unknown influence of time-invariant input on system dynamics. IE-TIA further exploits the identifiability and estimability of more general time-varying input and provides an asymptotic solution with desired convergence properties, with higher computation cost compared with TO-TIA. Our novel algorithms relax the dependence of observation scale and leverage the empirical risk reformulation to improve the inference accuracy in terms of the topology structure and edge weight. Comprehensive theoretical analysis and simulations for various topologies are provided to illustrate the inference feasibility and the performance of the proposed algorithms.
Improved List-Decodability of Reed-Solomon Codes via Tree Packings

This paper shows that there exist Reed--Solomon (RS) codes, over large finite fields, that are combinatorially list-decodable well beyond the Johnson radius, in fact almost achieving list-decoding capacity. In particular, we show that for any $\epsilon\in (0,1]$ there exist RS codes with rate $\Omega(\frac{\epsilon}{\log(1/\epsilon)+1})$ that are list-decodable from radius of $1-\epsilon$. We generalize this result to obtain a similar result on list-recoverability of RS codes. Along the way we use our techniques to give a new proof of a result of Blackburn on optimal linear perfect hash matrices, and strengthen it to obtain a construction of strongly perfect hash matrices. To derive the results in this paper we show a surprising connection of the above problems to graph theory, and in particular to the tree packing theorem of Nash-Williams and Tutte. En route to our results on RS codes, we prove a generalization of the tree packing theorem to hypergraphs (and we conjecture that an even stronger generalization holds). We hope that this generalization to hypergraphs will be of independent interest.
Topological obstructed atomic limit by annihilating Dirac fermions

We show that annihilating a pair of Dirac fermions implies a topological transition from the critical semi-metallic phase to an Obstructed Atomic Limit (OAL) insulator phase instead of a trivial insulator. This is shown to happen because of branch-cuts in the phase of the wave functions, leading to non trivial Zak phase along certain directions. To this end, we study their Z$_2$ invariant and also study the phase transition using Entanglement Entropy. We use low energy Hamiltonians and numerical result from model systems to show this effect. These transitions are observed in realistic materials including strained graphene and buckled honeycomb group-V (Sb/As).
Unmasking Communication Partners: A Low-Cost AI Solution for Digitally Removing Head-Mounted Displays in VR-Based Telepresence

Face-to-face conversation in Virtual Reality (VR) is a challenge when participants wear head-mounted displays (HMD). A significant portion of a participant’s face is hidden and facial expressions are difficult to perceive. Past research has shown that high-fidelity face reconstruction with personal avatars in VR is possible under laboratory conditions with high-cost hardware. In this paper, we propose one of the first low-cost systems for this task which uses only open source, free software and affordable hardware. Our approach is to track the user’s face underneath the HMD utilizing a Convolutional Neural Network (CNN) and generate corresponding expressions with Generative Adversarial Networks (GAN) for producing RGBD images of the person’s face. We use commodity hardware with low-cost extensions such as 3Dprinted mounts and miniature cameras. Our approach learns end-to-end without manual intervention, runs in real time, and can be trained and executed on an ordinary gaming computer. We report evaluation results showing that our low-cost system does not achieve the same fidelity of research prototypes using high-end hardware and closed source software, but it is capable of creating individual facial avatars with personspecific characteristics in movements and expressions.
Prediction problems inspired by animal learning

We present three problems modeled after animal learning experiments designed to test online state construction or representation learning algorithms. Our test problems require the learning system to construct compact summaries of their past interaction with the world in order to predict the future, updating online and incrementally on each time step without an explicit training-testing split. The majority of recent work in Deep Reinforcement Learning focuses on either fully observable tasks, or games where stacking a handful of recent frames is sufficient for good performance. Current benchmarks used for evaluating memory and recurrent learning make use of 3D visual environments (e.g., DeepMind Lab) which require billions of training samples, complex agent architectures, and cloud-scale compute. These domains are thus not well suited for rapid prototyping, hyper-parameter study, or extensive replication study. In this paper, we contribute a set of test problems and benchmark results to fill this gap. Our test problems are designed to be the simplest instantiation and test of learning capabilities which animals readily exhibit, including (1) trace conditioning (remembering a cue in order to predict another far in the future), (2) patterning (a particular combination of cues predict another), (3) and combinations of both with additional non-relevant distracting signals. We provide baselines for each problem including heuristics from the early days of neural network learning and simple ideas inspired by computational models of animal learning. Our results highlight the difficulty of our test problems for online recurrent learning systems and how the agent’s performance often exhibits substantial sensitivity to the choice of key problem and agent parameters.