Miracle-dz/UniFAR-FTUs huggingface dataset:
TartanVO: A Generalizable Learning-based VO

We present the first learning-based visual odometry (VO) model, which generalizes to multiple datasets and real-world scenarios and outperforms geometry-based methods in challenging scenes. We achieve this by leveraging the SLAM dataset TartanAir, which provides a large amount of diverse synthetic data in challenging environments. Furthermore, to make our VO model generalize across datasets, we propose an up-to-scale loss function and incorporate the camera intrinsic parameters into the model. Experiments show that a single model, TartanVO, trained only on synthetic data, without any finetuning, can be generalized to real-world datasets such as KITTI and EuRoC, demonstrating significant advantages over the geometry-based methods on challenging trajectories. Our code is available at this https URL.
Nonparametric Identification of Production Function, Total Factor Productivity, and Markup from Revenue Data

Commonly used methods of production function estimation assume that a firm’s output quantity can be observed as data, but typical datasets contain only revenue, not output quantity. We examine the nonparametric identification of production function from revenue data when a firm faces a general nonparametric demand function under imperfect competition. Under standard assumptions, we provide the constructive nonparametric identification of various firm-level objects: gross production function, total factor productivity, price markups over marginal costs, output prices, output quantities, a demand system, and a representative consumer’s utility function.
AI Marker-based Large-scale AI Literature Mining

The knowledge contained in academic literature is interesting to mine. Inspired by the idea of molecular markers tracing in the field of biochemistry, three named entities, namely, methods, datasets and metrics are used as AI markers for AI literature. These entities can be used to trace the research process described in the bodies of papers, which opens up new perspectives for seeking and mining more valuable academic information. Firstly, the entity extraction model is used in this study to extract AI markers from large-scale AI literature. Secondly, original papers are traced for AI markers. Statistical and propagation analysis are performed based on tracing results. Finally, the co-occurrences of AI markers are used to achieve clustering. The evolution within method clusters and the influencing relationships amongst different research scene clusters are explored. The above-mentioned mining based on AI markers yields many meaningful discoveries. For example, the propagation of effective methods on the datasets is rapidly increasing with the development of time; effective methods proposed by China in recent years have increasing influence on other countries, whilst France is the opposite. Saliency detection, a classic computer vision research scene, is the least likely to be affected by other research scenes.
Pose Randomization for Weakly Paired Image Style Translation

Utilizing the trained model under different conditions without data annotation is attractive for robot applications. Towards this goal, one class of methods is to translate the image style from the training environment to the current one. Conventional studies on image style translation mainly focus on two settings: paired data on images from two domains with exactly aligned content, and unpaired data, with independent content. In this paper, we would like to propose a new setting, where the content in the two images is aligned with error in poses. We consider that this setting is more practical since robots with various sensors are able to align the data up to some error level, even with different styles. To solve this problem, we propose PRoGAN to learn a style translator by intentionally transforming the original domain images with a noisy pose, then matching the distribution of translated transformed images and the distribution of the target domain images. The adversarial training enforces the network to learn the style translation, avoiding being entangled with other variations. In addition, we propose two pose estimation based self-supervised tasks to further improve the performance. Finally, PRoGAN is validated on both simulated and real-world collected data to show the effectiveness. Results on down-stream tasks, classification, road segmentation, object detection, and feature matching show its potential for real applications. https://github.com/wrld/PRoGAN .
Concurrent Multisensory Integration and Segregation with Complementary Congruent and Opposite Neurons

Our brain perceives the world by exploiting multiple sensory modalities to extract information about various aspects of external stimuli. If these sensory cues are from the same stimulus of interest, they should be integrated to improve perception; otherwise, they should be segregated to distinguish different stimuli. In reality, however, the brain faces the challenge of recognizing stimuli without knowing in advance whether sensory cues come from the same or different stimuli. To address this challenge and to recognize stimuli rapidly, we argue that the brain should carry out multisensory integration and segregation concurrently with complementary neuron groups. Studying an example of inferring heading-direction via visual and vestibular cues, we develop a concurrent multisensory processing neural model which consists of two reciprocally connected modules, the dorsal medial superior temporal area (MSTd) and the ventral intraparietal area (VIP), and that at each module, there exists two distinguishing groups of neurons, congruent and opposite neurons. Specifically, congruent neurons implement cue integration, while opposite neurons compute the cue disparity, both optimally as described by Bayesian inference. The two groups of neurons provide complementary information which enables the neural system to assess the validity of cue integration and, if necessary, to recover the lost information associated with individual cues without re-gathering new inputs. Through this process, the brain achieves rapid stimulus perception if the cues come from the same stimulus of interest, and differentiates and recognizes stimuli based on individual cues with little time delay if the cues come from different stimuli of interest. Our study unveils the indispensable role of opposite neurons in multisensory processing and sheds light on our understanding of how the brain achieves multisensory processing efficiently and rapidly. Significance Statement Our brain perceives the world by exploiting multiple sensory cues. These cues need to be integrated to improve perception if they come from the same stimulus and otherwise be segregated. To address the challenge of recognizing whether sensory cues come from the same or different stimuli that are unknown in advance, we propose that the brain should carry out multisensory integration and segregation concurrently with two different neuron groups. Specifically, congruent neurons implement cue integration, while opposite neurons compute the cue disparity, and the interplay between them achieves rapid stimulus recognition without information loss. We apply our model to the example of inferring heading-direction based on visual and vestibular cues and reproduce the experimental data successfully.
FireCommander: An Interactive, Probabilistic Multi-agent Environment for Joint Perception-Action Tasks

The purpose of this tutorial is to help individuals use the \underline{FireCommander} game environment for research applications. The FireCommander is an interactive, probabilistic joint perception-action reconnaissance environment in which a composite team of agents (e.g., robots) cooperate to fight dynamic, propagating firespots (e.g., targets). In FireCommander game, a team of agents must be tasked to optimally deal with a wildfire situation in an environment with propagating fire areas and some facilities such as houses, hospitals, power stations, etc. The team of agents can accomplish their mission by first sensing (e.g., estimating fire states), communicating the sensed fire-information among each other and then taking action to put the firespots out based on the sensed information (e.g., dropping water on estimated fire locations). The FireCommander environment can be useful for research topics spanning a wide range of applications from Reinforcement Learning (RL) and Learning from Demonstration (LfD), to Coordination, Psychology, Human-Robot Interaction (HRI) and Teaming. There are four important facets of the FireCommander environment that overall, create a non-trivial game: (1) Complex Objectives: Multi-objective Stochastic Environment, (2)Probabilistic Environment: Agents' actions result in probabilistic performance, (3) Hidden Targets: Partially Observable Environment and, (4) Uni-task Robots: Perception-only and Action-only agents. The FireCommander environment is first-of-its-kind in terms of including Perception-only and Action-only agents for coordination. It is a general multi-purpose game that can be useful in a variety of combinatorial optimization problems and stochastic games, such as applications of Reinforcement Learning (RL), Learning from Demonstration (LfD) and Inverse RL (iRL).
An Embarrassingly Simple Approach to Training Ternary Weight Networks

Deep neural networks (DNNs) have achieved great successes in various domains of artificial intelligence, but they require large amounts of memory and computational power. This severely restricts their implementation on resource-limited hardware. One approach to solving this problem is to train DNNs with ternary weights \{-1, 0, +1\}, thus avoiding multiplications and dramatically reducing the memory and computation requirements. However, the existing approaches to training ternary weight networks either have a large performance gap to the full precision counterparts or have a complex training process, which makes ternary weight networks not widely used. In this paper, we propose an embarrassingly simple approach (ESA) to training ternary weight networks. Specifically, ESA first parameterizes the weights $W$ in a DNN with $\tanh(\Theta)$ where $\Theta$ are the parameters, so that the weight values are limited in the range between -1 and +1, and then a weight discretization regularization (WDR) is used to force the weights to be ternary. Consequently, ESA has an extremely high code reuse rate when converting a full precision weight DNN to the ternary version. More importantly, ESA is able to control the sparsity (i.e., the percentage of 0s) of the ternary weights through a controller $\alpha$ in WDR. We theoretically and empirically show that the sparsity of the trained ternary weights is positively related to $\alpha$. To the best of our knowledge, ESA is the first sparsity-controlling approach to training ternary weight networks. Extensive experiments on several benchmark datasets demonstrate that ESA beats the state-of-the-art approaches significantly and matches the performances of the full precision weight networks.
AutoAudit: Mining Accounting and Time-Evolving Graphs

How can we spot money laundering in large-scale graph-like accounting datasets? How to identify the most suspicious period in a time-evolving accounting graph? What kind of accounts and events should practitioners prioritize under time constraints? To tackle these crucial challenges in accounting and auditing tasks, we propose a flexible system called AutoAudit, which can be valuable for auditors and risk management professionals. To sum up, there are four major advantages of the proposed system: (a) "Smurfing" Detection, spots nearly 100% of injected money laundering transactions automatically in real-world datasets. (b) Attention Routing, attends to the most suspicious part of time-evolving graphs and provides an intuitive interpretation. (c) Insight Discovery, identifies similar month-pair patterns proved by "success stories" and patterns following Power Laws in log-logistic scales. (d) Scalability and Generality, ensures AutoAudit scales linearly and can be easily extended to other real-world graph datasets. Experiments on various real-world datasets illustrate the effectiveness of our method. To facilitate reproducibility and accessibility, we make the code, figure, and results public at https://github.com/mengchillee/AutoAudit.
Neuromorphic control for optic-flow-based landing of MAVs using the Loihi processor

Neuromorphic processors like Loihi offer a promising alternative to conventional computing modules for endowing constrained systems like micro air vehicles (MAVs) with robust, efficient and autonomous skills such as take-off and landing, obstacle avoidance, and pursuit. However, a major challenge for using such processors on robotic platforms is the reality gap between simulation and the real world. In this study, we present for the very first time a fully embedded application of the Loihi neuromorphic chip prototype in a flying robot. A spiking neural network (SNN) was evolved to compute the thrust command based on the divergence of the ventral optic flow field to perform autonomous landing. Evolution was performed in a Python-based simulator using the PySNN library. The resulting network architecture consists of only 35 neurons distributed among 3 layers. Quantitative analysis between simulation and Loihi reveals a root-mean-square error of the thrust setpoint as low as 0.005 g, along with a 99.8% matching of the spike sequences in the hidden layer, and 99.7% in the output layer. The proposed approach successfully bridges the reality gap, offering important insights for future neuromorphic applications in robotics. Supplementary material is available at https://mavlab.tudelft. nl/loihi/.
Temporary changes in large-scale memory neural networks after fear learning and extinction in healthy adults

The analysis of the functional connectivity of brain sections associated with fear-conditioned training is one of the methods of study of memory neural networks. Before, the majority of studies focused on the functional connectivity of the brain regions that are known for emotional processing such as amygdala and areas of the ventromedial prefrontal cortex in the resting state right after the formation of a fear-conditioned reflex. In the present study authors applied the methods of the theory of graphs to search for changes in the functional connectivity at the level of the brain in the resting state in a week dynamics after the extinction of a conditioned reflex with partial reinforcement. The most significant changes were observed in the functional connectivity of the left parahippocampal area. In particular, the rostral part of the left parahippocampal gyrus became the center of a new subnetwork connected with the rostral part of the left hippocampus in all the sessions and after the extinction of a conditioned reflex and the lateral part of the left amygdala right after the extinction and a day after the extinction of a conditioned reflex. A week after the extinction of a conditioned reflex, the rostral part of the left parahippocampal gyrus also had more connections with the areas of the middle frontal gyrus in comparison with the parameters of the baseline resting state to the stimulus. Besides, these changes remained within one week from the moment of the extinction of a conditioned reflex, which could be explained by the chosen paradigm of conditioned reflex with partial reinforcement that led to a slower extinction than a paradigm with full reinforcement.
Wireless Power Transfer With Distributed Antennas: System Design, Prototype, and Experiments

In this article, we design and experiment a far-field wireless power transfer (WPT) architecture based on a distributed antenna system (DAS), so-called WPT DAS, that dynamically selects transmit antenna and frequency to increase the output dc power. Uniquely, spatial and frequency diversities are jointly exploited in the proposed WPT DAS with low complexity, low cost, and flexible deployment to combat the wireless fading channel. A numerical experiment is designed to show the benefits using antenna and frequency selections in spatially and frequency selective fading channels for single-user and multiuser cases. Accordingly, the proposed WPT DAS for single-user and two-user cases is prototyped. At the transmitter, we adopt antenna selection to exploit the spatial diversity and adopt frequency selection to exploit the frequency diversity. A low-complexity over-the-air limited feedback using an IEEE 802.15. 4 RF interface is designed for antenna and frequency selections and reporting from the receiver to the transmitter. The proposed WPT DAS prototype is demonstrated in a real indoor environment. The measurements show that WPT DAS can boost the output dc power by up to 30 dB in a single-user case and boost the sum of the output dc power by up to 21.8 dB in the two-user case and broaden the service coverage area in a low cost, low complexity, and flexible manner.
SMRTer Chatbots: Improving Non-Task-Oriented Dialog with Simulated Multi-Reference Training

Non-task-oriented dialog models suffer from poor quality and non-diverse responses. To overcome limited conversational data, we apply Simulated Multiple Reference Training (SMRT; Khayrallah et al., 2020), and use a paraphraser to simulate multiple responses per training prompt. We find SMRT improves over a strong Transformer baseline as measured by human and automatic quality scores and lexical diversity. We also find SMRT is comparable to pretraining in human evaluation quality, and outperforms pretraining on automatic quality and lexical diversity, without requiring related-domain dialog data.
Personalized Multimodal Feedback Generation in Education

The automatic feedback of school assignments is an important application of AI in education. In this work, we focus on the task of personalized multimodal feedback generation, which aims to generate personalized feedback for teachers to evaluate students’ assignments involving multimodal inputs such as images, audios, and texts. This task involves the representation and fusion of multimodal information and natural language generation, which presents the challenges from three aspects: (1) how to encode and integrate multimodal inputs; (2) how to generate feedback specific to each modality; and (3) how to fulfill personalized feedback generation. In this paper, we propose a novel Personalized Multimodal Feedback Generation Network (PMFGN) armed with a modality gate mechanism and a personalized bias mechanism to address these challenges. Extensive experiments on real-world K-12 education data show that our model significantly outperforms baselines by generating more accurate and diverse feedback. In addition, detailed ablation experiments are conducted to deepen our understanding of the proposed framework.
Bound Controller for a Quadruped Robot using Pre-Fitting Deep Reinforcement Learning

The bound gait is an important gait in quadruped robot locomotion. It can be used to cross obstacles and often serves as transition mode between trot and gallop. However, because of the complexity of the models, the bound gait built by the conventional control method is often unnatural and slow to compute. In the present work, we introduce a method to achieve the bound gait based on model-free pre-fit deep reinforcement learning (PF-DRL). We first constructed a net with the same structure as an actor net in the PPO2 and pre-fit it using the data collected from a robot using conventional model-based controller. Next, the trained weights are transferred into the PPO2 and be optimized further. Moreover, target on the symmetrical and periodic characteristic during bounding, we designed a reward function based on contact points. We also used feature engineering to improve the input features of the DRL model and improve performance on flat ground. Finally, we trained the bound controller in simulation and successfully deployed it on the Jueying Mini robot. It performs better than the conventional method with higher computational efficiency and more stable center-of-mass height in our experiments.
Two-layer clustering-based sparsifying transform learning for low-dose CT reconstruction

Achieving high-quality reconstructions from low-dose computed tomography (LDCT) measurements is of much importance in clinical settings. Model-based image reconstruction methods have been proven to be effective in removing artifacts in LDCT. In this work, we propose an approach to learn a rich two-layer clustering-based sparsifying transform model (MCST2), where image patches and their subsequent feature maps (filter residuals) are clustered into groups with different learned sparsifying filters per group. We investigate a penalized weighted least squares (PWLS) approach for LDCT reconstruction incorporating learned MCST2 priors. Experimental results show the superior performance of the proposed PWLS-MCST2 approach compared to other related recent schemes.
Insights into the inner structures of the fully charmed tetraquark state 
X(6900)

The recently discovered fully-charmed tetraquark candidate $X(6900)$ is analyzed within the frameworks of effective-range expansion, compositeness relation and width saturation, and a three-coupled channel dynamical study. By taking into account constraints from heavy-quark spin symmetry, the coupled-channel amplitude including the $J/\psi J/\psi,~ \chi_{c0}\chi_{c0}$ and $\chi_{c1}\chi_{c1}$ is constructed to fit the experimental di-$J/\psi$ event distributions around the energy region near 6.9 GeV. The three different theoretical approaches lead to similar conclusions that the two-meson components do not play dominant roles in the $X(6900)$. Our determinations of the resonance poles from the refined couple-channel study are found to be consistent with the experimental analyses. We give predictions to the line shapes of the $\chi_{c0}\chi_{c0}$ and $\chi_{c1}\chi_{c1}$ channels, which could provide useful guides for the future experimental measurements.
Emergent Communication Pretraining for Few-Shot Machine Translation

While state-of-the-art models that rely upon massively multilingual pretrained encoders achieve sample efficiency in downstream applications, they still require abundant amounts of unlabelled text. Nevertheless, most of the world’s languages lack such resources. Hence, we investigate a more radical form of unsupervised knowledge transfer in the absence of linguistic data. In particular, for the first time we pretrain neural networks via emergent communication from referential games. Our key assumption is that grounding communication on images—as a crude approximation of real-world environments—inductively biases the model towards learning natural languages. On the one hand, we show that this substantially benefits machine translation in few-shot settings. On the other hand, this also provides an extrinsic evaluation protocol to probe the properties of emergent languages ex vitro. Intuitively, the closer they are to natural languages, the higher the gains from pretraining on them should be. For instance, in this work we measure the influence of communication success and maximum sequence length on downstream performances. Finally, we introduce a customised adapter layer and annealing strategies for the regulariser of maximum-a-posteriori inference during fine-tuning. These turn out to be crucial to facilitate knowledge transfer and prevent catastrophic forgetting. Compared to a recurrent baseline, our method yields gains of 59.0% 147.6% in BLEU score with only 500 NMT training instances and 65.1% 196.7% with 1,000 NMT training instances across four language pairs. These proof-of-concept results reveal the potential of emergent communication pretraining for both natural language processing tasks in resource-poor settings and extrinsic evaluation of artificial languages.