Miracle-dz/UniFAR-FTUs huggingface dataset:
MMTSA: Multimodal Temporal Segment Attention Network for Efficient Human Activity Recognition

Multimodal sensors (e.g., visual, non-visual, and wearable) provide complementary information to develop robust perception systems for recognizing activities. However, most existing algorithms use dense sampling and heterogeneous subnetwork to extract unimodal features and fuse them at the end of their framework, which causes data redundancy, lack of complementary multimodal information and high computational cost. In this paper, we propose a new novel multimodal neural architecture based on RGB and IMU wearable sensors (e.g., accelerometer, gyroscope) for human activity recognition called Multimodal Temporal Segment Attention Network (MMTSA). MMTSA first employs a multimodal data isomorphism mechanism based on Gramian Angular Field (GAF) and then applies a novel multimodal sparse sampling method to reduce redundancy. Moreover, we propose an inter-segment attention module in MMTSA to fuse multimodal features effectively and efficiently. We demonstrate the importance of imu data imaging and attention mechanism in human activity recognition by rigorous evaluation on three public datasets, and achieve superior improvements (11.13% on the MMAct dataset) than the previous state-of-the-art methods. The code is available at: https://github.com/THU-CS-PI/MMTSA .
Hierarchical Decentralized Deep Reinforcement Learning Architecture for a Simulated Four-Legged Agent

Legged locomotion is widespread in nature and has inspired the design of current robots. The controller of these legged robots is often realized as one centralized instance. However, in nature, control of movement happens in a hierarchical and decentralized fashion. Introducing these biological design principles into robotic control systems has motivated this work. We tackle the question whether decentralized and hierarchical control is beneficial for legged robots and present a novel decentral, hierarchical architecture to control a simulated legged agent. Three different tasks varying in complexity are designed to benchmark five architectures (centralized, decentralized, hierarchical and two different combinations of hierarchical decentralized architectures). The results demonstrate that decentralizing the different levels of the hierarchical architectures facilitates learning of the agent, ensures more energy efficient movements as well as robustness towards new unseen environments. Furthermore, this comparison sheds light on the importance of modularity in hierarchical architectures to solve complex goal-directed tasks. We provide an open-source code implementation of our architecture (https://github.com/wzaielamri/hddrl).
ProtoVAE: A Trustworthy Self-Explainable Prototypical Variational Model

The need for interpretable models has fostered the development of self-explainable classifiers. Prior approaches are either based on multi-stage optimization schemes, impacting the predictive performance of the model, or produce explanations that are not transparent, trustworthy or do not capture the diversity of the data. To address these shortcomings, we propose ProtoVAE, a variational autoencoder-based framework that learns class-specific prototypes in an end-to-end manner and enforces trustworthiness and diversity by regularizing the representation space and introducing an orthonormality constraint. Finally, the model is designed to be transparent by directly incorporating the prototypes into the decision process. Extensive comparisons with previous self-explainable approaches demonstrate the superiority of ProtoVAE, highlighting its ability to generate trustworthy and diverse explanations, while not degrading predictive performance.
TweetNERD - End to End Entity Linking Benchmark for Tweets

Named Entity Recognition and Disambiguation (NERD) systems are foundational for information retrieval, question answering, event detection, and other natural language processing (NLP) applications. We introduce TweetNERD, a dataset of 340K+ Tweets across 2010-2021, for benchmarking NERD systems on Tweets. This is the largest and most temporally diverse open sourced dataset benchmark for NERD on Tweets and can be used to facilitate research in this area. We describe evaluation setup with TweetNERD for three NERD tasks: Named Entity Recognition (NER), Entity Linking with True Spans (EL), and End to End Entity Linking (End2End); and provide performance of existing publicly available methods on specific TweetNERD splits. TweetNERD is available at: https://doi.org/10.5281/zenodo.6617192 under Creative Commons Attribution 4.0 International (CC BY 4.0) license. Check out more details at https://github.com/twitter-research/TweetNERD.
Invertible Monotone Operators for Normalizing Flows

Normalizing flows model probability distributions by learning invertible transformations that transfer a simple distribution into complex distributions. Since the architecture of ResNet-based normalizing flows is more flexible than that of coupling-based models, ResNet-based normalizing flows have been widely studied in recent years. Despite their architectural flexibility, it is well-known that the current ResNet-based models suffer from constrained Lipschitz constants. In this paper, we propose the monotone formulation to overcome the issue of the Lipschitz constants using monotone operators and provide an in-depth theoretical analysis. Furthermore, we construct an activation function called Concatenated Pila (CPila) to improve gradient flow. The resulting model, Monotone Flows, exhibits an excellent performance on multiple density estimation benchmarks (MNIST, CIFAR-10, ImageNet32, ImageNet64). Code is available at https://github.com/mlvlab/MonotoneFlows.
tegdet: An extensible Python Library for Anomaly Detection using Time-Evolving Graphs

This paper presents a new Python library for anomaly detection in unsupervised learning approaches. The input for the library is a univariate time series representing observations of a given phenomenon. Then, it can identify anomalous epochs, i.e., time intervals where the observations are above a given percentile of a baseline distribution, defined by a dissimilarity metric. Using time-evolving graphs for the anomaly detection, the library leverages valuable information given by the inter-dependencies among data. Currently, the library implements 28 different dissimilarity metrics, and it has been designed to be easily extended with new ones. Through an API, the library exposes a complete functionality to carry out the anomaly detection. Summarizing, to the best of our knowledge, this library is the only one publicly available, that based on dynamic graphs, can be extended with other state-of-the-art anomaly detection techniques. Our experimentation shows promising results regarding the execution times of the algorithms and the accuracy of the implemented techniques. Additionally, the paper provides guidelines for setting the parameters of the detectors to improve their performance and prediction accuracy.
Disentangling Confidence Score Distribution for Out-of-Domain Intent Detection with Energy-Based Learning

Detecting Out-of-Domain (OOD) or unknown intents from user queries is essential in a taskoriented dialog system. Traditional softmaxbased confidence scores are susceptible to the overconfidence issue. In this paper, we propose a simple but strong energy-based score function to detect OOD where the energy scores of OOD samples are higher than IND samples. Further, given a small set of labeled OOD samples, we introduce an energy-based margin objective for supervised OOD detection to explicitly distinguish OOD samples from INDs. Comprehensive experiments and analysis prove our method helps disentangle confidence score distributions of IND and OOD data.
Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training

Visual question answering (VQA) is a hallmark of vision and language reasoning and a challenging task under the zero-shot setting. We propose Plug-and-Play VQA (PNP-VQA), a modular framework for zero-shot VQA. In contrast to most existing works, which require substantial adaptation of pretrained language models (PLMs) for the vision modality, PNP-VQA requires no additional training of the PLMs. Instead, we propose to use natural language and network interpretation as an intermediate representation that glues pretrained models together. We first generate question-guided informative image captions, and pass the captions to a PLM as context for question answering. Surpassing end-to-end trained baselines, PNP-VQA achieves state-of-the-art results on zero-shot VQAv2 and GQA. With 11B parameters, it outperforms the 80B-parameter Flamingo model by 8.5% on VQAv2. With 738M PLM parameters, PNP-VQA achieves an improvement of 9.1% on GQA over FewVLM with 740M PLM parameters. Code is released at https://github.com/salesforce/LAVIS/tree/main/projects/pnp-vqa
Scale-Agnostic Super-Resolution in MRI using Feature-Based Coordinate Networks

We propose using a coordinate network decoder for the task of super-resolution in MRI. The continuous signal representation of coordinate networks enables this approach to be scale-agnostic, i.e. one can train over a continuous range of scales and subsequently query at arbitrary resolutions. Due to the difficulty of performing super-resolution on inherently noisy data, we analyze network behavior under multiple denoising strategies. Lastly we compare this method to a standard convolutional decoder using both quantitative metrics and a radiologist study implemented in Voxel, our newly developed tool for web-based evaluation of medical images.
Probing and harnessing photonic Fermi arc surface states using light-matter interactions

Fermi arcs, i.e., surface states connecting topologically distinct Weyl points, represent a paradigmatic manifestation of the topological aspects of Weyl physics. We investigate a light-matter interface based on the photonic counterpart of these states and prove that it can lead to phenomena with no analog in other setups. First, we show how to image the Fermi arcs by studying the spontaneous decay of one or many emitters coupled to the system’s border. Second, we demonstrate that, exploiting the negative refraction of these modes, the Fermi arc surface states can act as a robust quantum link, enabling, e.g., the occurrence of perfect quantum state transfer between the considered emitters or the formation of highly entangled states. In addition to their fundamental interest, our findings evidence the potential offered by the photonic Fermi arc light-matter interfaces for the design of more robust quantum technologies.
Product Ranking for Revenue Maximization with Multiple Purchases

Product ranking is the core problem for revenue-maximizing online retailers. To design proper product ranking algorithms, various consumer choice models are proposed to characterize the consumers' behaviors when they are provided with a list of products. However, existing works assume that each consumer purchases at most one product or will keep viewing the product list after purchasing a product, which does not agree with the common practice in real scenarios. In this paper, we assume that each consumer can purchase multiple products at will. To model consumers' willingness to view and purchase, we set a random attention span and purchase budget, which determines the maximal amount of products that he/she views and purchases, respectively. Under this setting, we first design an optimal ranking policy when the online retailer can precisely model consumers' behaviors. Based on the policy, we further develop the Multiple-Purchase-with-Budget UCB (MPB-UCB) algorithms with $\~O(\sqrt{T})$ regret that estimate consumers' behaviors and maximize revenue simultaneously in online settings. Experiments on both synthetic and semi-synthetic datasets prove the effectiveness of the proposed algorithms.
Bose-Einstein condensates and the thin-shell limit in anisotropic bubble traps

Within the many different models, that appeared with the use of cold atoms to create BECs, the bubble trap shaped potential has been of great interest. However, the relationship between the physical parameters and the resulting manifold geometry remains yet to be fully understood for the anisotropic bubble trap physics in the thin-shell limit. In this paper, we work towards this goal by showing how the parameters of the system must be manipulated in order to allow for a non-collapsing thin-shell limit. In such a limit, a dimensional compactification takes place, thus leading to an effective 2D Hamiltonian which relates to up-to-date bubble trap experiments. At last, the resulting Hamiltonian is perturbatively solved for both the ground-state wave function and the excitation frequencies in the leading order of deviations from a spherical bubble trap.
DataTV: Streaming Data Videos for Storytelling

Data videos—motion graphics that incorporate visualizations—have been recognized as an effective way to com-municate ideas, but creating such video requires both time and expertise, precluding them from being created and streamed live. We introduce DataTV, a system for combining multiple media sources in real time. We validate our work through an expert review involving researchers using the DataTV prototype to create a one-minute data video for their current project. Results show that the new method facilitates rapid creation and enables users to focus on the narrative rather than mechanics of video production. in a highly streamlined and optimized workﬂow that allows a single content creator to control the entire process during live streaming.Tovalidate the DataTV platform, we asked two visualization experts to create a short data video using DataTV on a speciﬁc topic to collect real experiences on using the system for storytelling. Results from this expert review (participant-generated clips are included in the companion video for this paper) indicate that these researchers were impressed with the relative ease they could create these videos. Interestingly, our ﬁndings suggest that the integrated functionality in the DataTV platform enabled the researchers to focus on storytelling rather than the mechanics of video production. Their feedback informed further improvements to the platform.
Joint Multilingual Knowledge Graph Completion and Alignment

Knowledge graph (KG) alignment and completion are usually treated as two independent tasks. While recent work has leveraged entity and relation alignments from multiple KGs, such as alignments between multilingual KGs with common entities and relations, a deeper understanding of the ways in which multilingual KG completion (MKGC) can aid the creation of multilingual KG alignments (MKGA) is still limited. Motivated by the observation that structural inconsistencies -- the main challenge for MKGA models -- can be mitigated through KG completion methods, we propose a novel model for jointly completing and aligning knowledge graphs. The proposed model combines two components that jointly accomplish KG completion and alignment. These two components employ relation-aware graph neural networks that we propose to encode multi-hop neighborhood structures into entity and relation representations. Moreover, we also propose (i) a structural inconsistency reduction mechanism to incorporate information from the completion into the alignment component, and (ii) an alignment seed enlargement and triple transferring mechanism to enlarge alignment seeds and transfer triples during KGs alignment. Extensive experiments on a public multilingual benchmark show that our proposed model outperforms existing competitive baselines, obtaining new state-of-the-art results on both MKGC and MKGA tasks. We publicly release the implementation of our model at https://github.com/vinhsuhi/JMAC
Meta-Learning via Classifier(-free) Guidance

State-of-the-art meta-learning techniques do not optimize for zero-shot adaptation to unseen tasks, a setting in which humans excel. On the contrary, meta-learning algorithms learn hyperparameters and weight initializations that explicitly optimize for few-shot learning performance. In this work, we take inspiration from recent advances in generative modeling and language-conditioned image synthesis to pro-pose meta-learning techniques that use natural language guidance to achieve higher zero-shot performance compared to the state-of-the-art. We do so by recasting the meta-learning problem as a multi-modal generative modeling problem: given a task, we consider its adapted neural network weights and its natural language description as equivalent multi-modal task representations. We ﬁrst train an unconditional generative hypernetwork model to produce neural network weights; then we train a second “guidance” model that, given a natural language task description, traverses the hypernetwork latent space to ﬁnd high-performance task-adapted weights in a zero-shot manner. We explore two alternative approaches for latent space guidance: “HyperCLIP”-based classiﬁer guidance and a conditional Hypernetwork Latent Diffusion Model (“HyperLDM”), which we show to beneﬁt
Teacher Forcing Recovers Reward Functions for Text Generation

Reinforcement learning (RL) has been widely used in text generation to alleviate the exposure bias issue or to utilize non-parallel datasets. The reward function plays an important role in making RL training successful. However, previous reward functions are typically task-specific and sparse, restricting the use of RL. In our work, we propose a task-agnostic approach that derives a step-wise reward function directly from a model trained with teacher forcing. We additionally propose a simple modification to stabilize the RL training on non-parallel datasets with our induced reward function. Empirical results show that our method outperforms self-training and reward regression methods on several text generation tasks, confirming the effectiveness of our reward function.
Tencent AI Lab - Shanghai Jiao Tong University Low-Resource Translation System for the WMT22 Translation Task

This paper describes Tencent AI Lab - Shanghai Jiao Tong University (TAL-SJTU) Low-Resource Translation systems for the WMT22 shared task. We participate in the general translation task on English-Livonian.Our system is based on M2M100 with novel techniques that adapt it to the target language pair. (1) Cross-model word embedding alignment: inspired by cross-lingual word embedding alignment, we successfully transfer a pre-trained word embedding to M2M100, enabling it to support Livonian. (2) Gradual adaptation strategy: we exploit Estonian and Latvian as auxiliary languages for many-to-many translation training and then adapt to English-Livonian. (3) Data augmentation: to enlarge the parallel data for English-Livonian, we construct pseudo-parallel data with Estonian and Latvian as pivot languages. (4) Fine-tuning: to make the most of all available data, we fine-tune the model with the validation set and online back-translation, further boosting the performance.In model evaluation: (1) We find that previous work underestimated the translation performance of Livonian due to inconsistent Unicode normalization, which may cause a discrepancy of up to 14.9 BLEU score. (2) In addition to the standard validation set, we also employ round-trip BLEU to evaluate the models, which we find more appropriate for this task. Finally, our unconstrained system achieves BLEU scores of 17.0 and 30.4 for English to/from Livonian.
Assessing the Optimality of LinDist3Flow for Optimal Tap Selection of Step Voltage Regulators in Unbalanced Distribution Networks

The adoption of distributed energy resources such as photovoltaics (PVs) has increased dramatically during the previous decade. The increased penetration of PVs into distribution networks (DNs) can cause voltage fluctuations that have to be mitigated. One of the key utility assets employed to this end are step-voltage regulators (SVRs). It is desirable to include tap selection of SVRs in optimal power flow (OPF) routines, a task that turns out to be challenging because the resultant OPF problem is nonconvex with added complexities stemming from accurate SVR modeling. While several convex relaxations based on semi-definite programming (SDP) have been presented in the literature for optimal tap selection, SDP based schemes do not scale well and are challenging to implement in large-scale planning or operational frameworks. This paper deals with the optimal tap selection (OPTS) problem for wye-connected SVRs using linear approximations of power flow equations. Specifically, the LinDist3Flow model is adopted and the effective SVR ratio is assumed to be continuous– enabling the formulation of a problem called LinDist3Flow-OPTS, which amounts to a linear program. The scalability and optimality gap of LinDist3Flow-OPTS are evaluated with respect to existing SDP-based and nonlinear programming techniques for optimal tap selection in three standard feeders, namely, the IEEE 13-bus, 123-bus, and 8500-node DNs. For all DNs considered, LinDist3Flow-OPTS achieves an optimality gap of approximately 1% or less while significantly lowering the computational burden.
Equivariant enumerative geometry

We formulate an equivariant conservation of number, which proves that a generalized Euler number of a complex equivariant vector bundle can be computed as a sum of local indices of an arbitrary section. This involves an expansion of the Pontryagin-Thom transfer in the equivariant setting. We leverage this result to commence a study of enumerative geometry in the presence of a group action. As an illustration of the power of this machinery, we prove that any smooth complex cubic surface defined by a symmetric polynomial has 27 lines whose orbit types under the $S_4$-action on $\mathbb{C}P^3$ are given by $[S_4/C_2]+[S_4/C_2']+[S_4/D_8]$, where $C_2$ and $C_2'$ denote two non-conjugate cyclic subgroups of order two. As a consequence we demonstrate that a real symmetric cubic surface can only contain 3 or 27 lines.
Characterization of elastic topological states using dynamic mode decomposition

Elastic topological states have been receiving increased intention in numerous scientific and engineering fields due to their defect-immune nature, resulting in applications of vibration control and information processing. Here, we present the data-driven discovery of elastic topological states using dynamic mode decomposition (DMD). The DMD spectrum and DMD modes are retrieved from the propagation of the relevant states along the topological boundary, where their nature is learned by DMD. Applications such as classification and prediction can be achieved by the underlying characteristics from DMD. We demonstrate the classification between topological and traditional metamaterials using DMD modes. Moreover, the model enabled by the DMD modes realizes the prediction of topological state propagation along the given interface. Our approach to characterizing topological states using DMD can pave the way towards data-driven discovery of topological phenomena in material physics and more broadly lattice systems.