OpenRaiser/Intern-Atlas huggingface dataset:
Self-Validated Physics-Embedding Network: A General Framework for Inverse Modelling
Venue: arXiv cs.NE 2023
Year: 2023

Physics-based inverse modeling techniques are typically restricted to particular research fields, whereas popular machine-learning-based ones are too data-dependent to guarantee the physical compatibility of the solution. In this paper, Self-Validated Physics-Embedding Network (SVPEN), a general neural network framework for inverse modeling is proposed. As its name suggests, the embedded physical forward model ensures that any solution that successfully passes its validation is physically reasonable. SVPEN operates in two modes: (a) the inverse function mode offers rapid state estimation as conventional supervised learning, and (b) the optimization mode offers a way to iteratively correct estimations that fail the validation process. Furthermore, the optimization mode provides SVPEN with reconfigurability i.e., replacing components like neural networks, physical models, and error calculations at will to solve a series of distinct inverse problems without pretraining. More than ten case studies in two highly nonlinear and entirely distinct applications: molecular absorption spectroscopy and Turbofan cycle analysis, demonstrate the generality, physical reliability, and reconfigurability of SVPEN. More importantly, SVPEN offers a solid foundation to use existing physical models within the context of AI, so as to striking a balance between data-driven and physics-driven models.
When are Local Queries Useful for Robust Learning?
Venue: arXiv cs.LG 2023
Year: 2023

Distributional assumptions have been shown to be necessary for the robust learnability of concept classes when considering the exact-in-the-ball robust risk and access to random examples by Gourdeau et al. (2019). In this paper, we study learning models where the learner is given more power through the use of local queries, and give the first distribution-free algorithms that perform robust empirical risk minimization (ERM) for this notion of robustness. The first learning model we consider uses local membership queries (LMQ), where the learner can query the label of points near the training sample. We show that, under the uniform distribution, LMQs do not increase the robustness threshold of conjunctions and any superclass, e.g., decision lists and halfspaces. Faced with this negative result, we introduce the local equivalence query ($\mathsf{LEQ}$) oracle, which returns whether the hypothesis and target concept agree in the perturbation region around a point in the training sample, as well as a counterexample if it exists. We show a separation result: on the one hand, if the query radius $λ$ is strictly smaller than the adversary's perturbation budget $ρ$, then distribution-free robust learning is impossible for a wide variety of concept classes; on the other hand, the setting $λ=ρ$ allows us to develop robust ERM algorithms. We then bound the query complexity of these algorithms based on online learning guarantees and further improve these bounds for the special case of conjunctions. We finish by giving robust learning algorithms for halfspaces on $\{0,1\}^n$ and then obtaining robustness guarantees for halfspaces in $\mathbb{R}^n$ against precision-bounded adversaries.
DiPA: Probabilistic Multi-Modal Interactive Prediction for Autonomous Driving
Venue: arXiv cs.RO 2023
Year: 2023

Accurate prediction is important for operating an autonomous vehicle in interactive scenarios. Prediction must be fast, to support multiple requests from a planner exploring a range of possible futures. The generated predictions must accurately represent the probabilities of predicted trajectories, while also capturing different modes of behaviour (such as turning left vs continuing straight at a junction). To this end, we present DiPA, an interactive predictor that addresses these challenging requirements. Previous interactive prediction methods use an encoding of k-mode-samples, which under-represents the full distribution. Other methods optimise closest-mode evaluations, which test whether one of the predictions is similar to the ground-truth, but allow additional unlikely predictions to occur, over-representing unlikely predictions. DiPA addresses these limitations by using a Gaussian-Mixture-Model to encode the full distribution, and optimising predictions using both probabilistic and closest-mode measures. These objectives respectively optimise probabilistic accuracy and the ability to capture distinct behaviours, and there is a challenging trade-off between them. We are able to solve both together using a novel training regime. DiPA achieves new state-of-the-art performance on the INTERACTION and NGSIM datasets, and improves over the baseline (MFP) when both closest-mode and probabilistic evaluations are used. This demonstrates effective prediction for supporting a planner on interactive scenarios.
SilverAlign: MT-Based Silver Data Algorithm For Evaluating Word Alignment
Venue: arXiv cs.CL 2023
Year: 2023

Word alignments are essential for a variety of NLP tasks. Therefore, choosing the best approaches for their creation is crucial. However, the scarce availability of gold evaluation data makes the choice difficult. We propose SilverAlign, a new method to automatically create silver data for the evaluation of word aligners by exploiting machine translation and minimal pairs. We show that performance on our silver data correlates well with gold benchmarks for 9 language pairs, making our approach a valid resource for evaluation of different domains and languages when gold data are not available. This addresses the important scenario of missing gold data alignments for low-resource languages.
Zero-Shot On-the-Fly Event Schema Induction
Venue: arXiv cs.CL 2023
Year: 2023

What are the events involved in a pandemic outbreak? What steps should be taken when planning a wedding? The answers to these questions can be found by collecting many documents on the complex event of interest, extracting relevant information, and analyzing it. We present a new approach in which large language models are utilized to generate source documents that allow predicting, given a high-level event definition, the specific events, arguments, and relations between them to construct a schema that describes the complex event in its entirety. Using our model, complete schemas on any topic can be generated on-the-fly without any manual data collection, i.e., in a zero-shot manner. Moreover, we develop efficient methods to extract pertinent information from texts and demonstrate in a series of experiments that these schemas are considered to be more complete than human-curated ones in the majority of examined scenarios. Finally, we show that this framework is comparable in performance with previous supervised schema induction methods that rely on collecting real texts while being more general and flexible without the need for a predefined ontology.
Centralized Training with Hybrid Execution in Multi-Agent Reinforcement Learning
Venue: arXiv cs.LG 2023
Year: 2023

We introduce hybrid execution in multi-agent reinforcement learning (MARL), a new paradigm in which agents aim to successfully complete cooperative tasks with arbitrary communication levels at execution time by taking advantage of information-sharing among the agents. Under hybrid execution, the communication level can range from a setting in which no communication is allowed between agents (fully decentralized), to a setting featuring full communication (fully centralized), but the agents do not know beforehand which communication level they will encounter at execution time. To formalize our setting, we define a new class of multi-agent partially observable Markov decision processes (POMDPs) that we name hybrid-POMDPs, which explicitly model a communication process between the agents. We contribute MARO, an approach that makes use of an auto-regressive predictive model, trained in a centralized manner, to estimate missing agents' observations at execution time. We evaluate MARO on standard scenarios and extensions of previous benchmarks tailored to emphasize the negative impact of partial observability in MARL. Experimental results show that our method consistently outperforms relevant baselines, allowing agents to act with faulty communication while successfully exploiting shared information.
From Merging Frameworks to Merging Stars: Experiences using HPX, Kokkos and SIMD Types
Venue: arXiv cs.DC 2023
Year: 2023

Octo-Tiger, a large-scale 3D AMR code for the merger of stars, uses a combination of HPX, Kokkos and explicit SIMD types, aiming to achieve performance-portability for a broad range of heterogeneous hardware. However, on A64FX CPUs, we encountered several missing pieces, hindering performance by causing problems with the SIMD vectorization. Therefore, we add std::experimental::simd as an option to use in Octo-Tiger's Kokkos kernels alongside Kokkos SIMD, and further add a new SVE (Scalable Vector Extensions) SIMD backend. Additionally, we amend missing SIMD implementations in the Kokkos kernels within Octo-Tiger's hydro solver. We test our changes by running Octo-Tiger on three different CPUs: An A64FX, an Intel Icelake and an AMD EPYC CPU, evaluating SIMD speedup and node-level performance. We get a good SIMD speedup on the A64FX CPU, as well as noticeable speedups on the other two CPU platforms. However, we also experience a scaling issue on the EPYC CPU.
Are Sample-Efficient NLP Models More Robust?
Venue: arXiv cs.CL 2023
Year: 2023

Recent results in image classification and extractive question answering have observed that pre-trained models trained on less in-distribution data have better out-of-distribution performance. However, it is unclear how broadly these trends hold. We conduct a large empirical study across three tasks, three broadly-applicable modeling interventions (increasing model size, using a different adaptation method, and pre-training on more data), and 14 diverse datasets to investigate the relationship between sample efficiency (amount of data needed to reach a given ID accuracy) and robustness (how models fare on OOD evaluation). We find that higher sample efficiency is only correlated with better average OOD robustness on some modeling interventions and tasks, but not others. On individual datasets, models with lower sample efficiency can even be more robust. These results suggest that general-purpose methods for improving sample efficiency are unlikely to yield universal OOD robustness improvements, since such improvements are highly dataset- and task-dependent. Even in an era of large, multi-purpose pretrained models, task-specific decisions may often be necessary for OOD generalization.
How to Sift Out a Clean Data Subset in the Presence of Data Poisoning?
Venue: arXiv cs.CR 2023
Year: 2023

Given the volume of data needed to train modern machine learning models, external suppliers are increasingly used. However, incorporating external data poses data poisoning risks, wherein attackers manipulate their data to degrade model utility or integrity. Most poisoning defenses presume access to a set of clean data (or base set). While this assumption has been taken for granted, given the fast-growing research on stealthy poisoning attacks, a question arises: can defenders really identify a clean subset within a contaminated dataset to support defenses? This paper starts by examining the impact of poisoned samples on defenses when they are mistakenly mixed into the base set. We analyze five defenses and find that their performance deteriorates dramatically with less than 1% poisoned points in the base set. These findings suggest that sifting out a base set with high precision is key to these defenses' performance. Motivated by these observations, we study how precise existing automated tools and human inspection are at identifying clean data in the presence of data poisoning. Unfortunately, neither effort achieves the precision needed. Worse yet, many of the outcomes are worse than random selection. In addition to uncovering the challenge, we propose a practical countermeasure, Meta-Sift. Our method is based on the insight that existing attacks' poisoned samples shifts from clean data distributions. Hence, training on the clean portion of a dataset and testing on the corrupted portion will result in high prediction loss. Leveraging the insight, we formulate a bilevel optimization to identify clean data and further introduce a suite of techniques to improve efficiency and precision. Our evaluation shows that Meta-Sift can sift a clean base set with 100% precision under a wide range of poisoning attacks. The selected base set is large enough to give rise to successful defenses.
A New Interference-Alignment Scheme for Wireless MapReduce
Venue: arXiv cs.IT 2023
Year: 2023

We consider a full-duplex wireless Distributed Computing (DC) system under the MapReduce framework. New upper and lower bounds on the optimal tradeoff between Normalized Delivery Time (NDT) and computation load are presented. The upper bound strictly improves over the previous reported upper bounds and is based on a novel interference alignment (IA) scheme tailored to the interference cancellation capabilities of MapReduce nodes. The lower bound is proved through information-theoretic converse arguments.
Overlooked Video Classification in Weakly Supervised Video Anomaly Detection
Venue: arXiv cs.CV 2023
Year: 2023

Current weakly supervised video anomaly detection algorithms mostly use multiple instance learning (MIL) or their varieties. Almost all recent approaches focus on how to select the correct snippets for training to improve the performance. They overlook or do not realize the power of video classification in boosting the performance of anomaly detection. In this paper, we study explicitly the power of video classification supervision using a BERT or LSTM. With this BERT or LSTM, CNN features of all snippets of a video can be aggregated into a single feature which can be used for video classification. This simple yet powerful video classification supervision, combined into the MIL framework, brings extraordinary performance improvement on all three major video anomaly detection datasets. Particularly it improves the mean average precision (mAP) on the XD-Violence from SOTA 78.84\% to new 82.10\%. The source code is available at https://github.com/wjtan99/BERT_Anomaly_Video_Classification.
Shortcomings of Question Answering Based Factuality Frameworks for Error Localization
Venue: arXiv cs.CL 2023
Year: 2023

Despite recent progress in abstractive summarization, models often generate summaries with factual errors. Numerous approaches to detect these errors have been proposed, the most popular of which are question answering (QA)-based factuality metrics. These have been shown to work well at predicting summary-level factuality and have potential to localize errors within summaries, but this latter capability has not been systematically evaluated in past research. In this paper, we conduct the first such analysis and find that, contrary to our expectations, QA-based frameworks fail to correctly identify error spans in generated summaries and are outperformed by trivial exact match baselines. Our analysis reveals a major reason for such poor localization: questions generated by the QG module often inherit errors from non-factual summaries which are then propagated further into downstream modules. Moreover, even human-in-the-loop question generation cannot easily offset these problems. Our experiments conclusively show that there exist fundamental issues with localization using the QA framework which cannot be fixed solely by stronger QA and QG models.
Sparse-grids uncertainty quantification of part-scale additive manufacturing processes
Venue: arXiv cs.CE 2023
Year: 2023

The present paper aims at applying uncertainty quantification methodologies to process simulations of powder bed fusion of metal. In particular, for a part-scale thermomechanical model of an Inconel 625 super-alloy beam, we study the uncertainties of three process parameters, namely the activation temperature, the powder convection coefficient and the gas convection coefficient. First, we perform a variance-based global sensitivity analysis to study how each uncertain parameter contributes to the variability of the beam displacements. The results allow us to conclude that the gas convection coefficient has little impact and can therefore be fixed to a constant value for subsequent studies. Then, we conduct an inverse uncertainty quantification analysis, based on a Bayesian approach on synthetic displacements data, to quantify the uncertainties of the two remaining parameters, namely the activation temperature and the powder convection coefficient. Finally, we use the results of the inverse uncertainty quantification analysis to perform a data-informed forward uncertainty quantification analysis of the residual strains. Crucially, we make use of surrogate models based on sparse grids to keep to a minimum the computational burden of every step of the uncertainty quantification analysis. The proposed uncertainty quantification workflow allows us to substantially ease the typical trial-and-error approach used to calibrate power bed fusion part-scale models, and to greatly reduce uncertainties on the numerical prediction of the residual strains. In particular, we demonstrate the possibility of using displacement measurements to obtain a data-informed probability density function of the residual strains, a quantity much more complex to measure than displacements.
Beeping Shortest Paths via Hypergraph Bipartite Decomposition
Venue: arXiv cs.DC 2023
Year: 2023

Constructing a shortest path between two network nodes is a fundamental task in distributed computing. This work develops schemes for the construction of shortest paths in randomized beeping networks between a predetermined source node and an arbitrary set of destination nodes. Our first scheme constructs a (single) shortest path to an arbitrary destination in $O (D \log\log n + \log^3 n)$ rounds with high probability. Our second scheme constructs multiple shortest paths, one per each destination, in $O (D \log^2 n + \log^3 n)$ rounds with high probability. Our schemes are based on a reduction of the above shortest path construction tasks to a decomposition of hypergraphs into bipartite hypergraphs: We develop a beeping procedure that partitions the (polynomially-large) hyperedge set of a hypergraph $H = (V_H, E_H)$ into $k = Θ(\log^2 n)$ disjoint subsets $F_1 \cup \cdots \cup F_k = E_H$ such that the (sub-)hypergraph $(V_H, F_i)$ is bipartite in the sense that there exists a vertex subset $U \subseteq V$ such that $|U \cap e| = 1$ for every $e \in F_i$. This procedure turns out to be instrumental in speeding up shortest path constructions under the beeping model.
Visual Reinforcement Learning with Self-Supervised 3D Representations
Venue: arXiv cs.LG 2023
Year: 2023

A prominent approach to visual Reinforcement Learning (RL) is to learn an internal state representation using self-supervised methods, which has the potential benefit of improved sample-efficiency and generalization through additional learning signal and inductive biases. However, while the real world is inherently 3D, prior efforts have largely been focused on leveraging 2D computer vision techniques as auxiliary self-supervision. In this work, we present a unified framework for self-supervised learning of 3D representations for motor control. Our proposed framework consists of two phases: a pretraining phase where a deep voxel-based 3D autoencoder is pretrained on a large object-centric dataset, and a finetuning phase where the representation is jointly finetuned together with RL on in-domain data. We empirically show that our method enjoys improved sample efficiency in simulated manipulation tasks compared to 2D representation learning methods. Additionally, our learned policies transfer zero-shot to a real robot setup with only approximate geometric correspondence, and successfully solve motor control tasks that involve grasping and lifting from a single, uncalibrated RGB camera. Code and videos are available at https://yanjieze.com/3d4rl/ .
Autoencoder-Aided Visualization of Collections of Morse Complexes
Venue: arXiv cs.HC 2023
Year: 2023

Though analyzing a single scalar field using Morse complexes is well studied, there are few techniques for visualizing a collection of Morse complexes. We focus on analyses that are enabled by looking at a Morse complex as an embedded domain decomposition. Specifically, we target 2D scalar fields, and we encode the Morse complex through binary images of the boundaries of decomposition. Then we use image-based autoencoders to create a feature space for the Morse complexes. We apply additional dimensionality reduction methods to construct a scatterplot as a visual interface of the feature space. This allows us to investigate individual Morse complexes, as they relate to the collection, through interaction with the scatterplot. We demonstrate our approach using a synthetic data set, microscopy images, and time-varying vorticity magnitude fields of flow. Through these, we show that our method can produce insights about structures within the collection of Morse complexes.
MTEB: Massive Text Embedding Benchmark
Venue: arXiv cs.CL 2023
Year: 2023

Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks. MTEB comes with open-source code and a public leaderboard at https://github.com/embeddings-benchmark/mteb.
"John is 50 years old, can his son be 65?" Evaluating NLP Models' Understanding of Feasibility
Venue: arXiv cs.CL 2023
Year: 2023

In current NLP research, large-scale language models and their abilities are widely being discussed. Some recent works have also found notable failures of these models. Often these failure examples involve complex reasoning abilities. This work focuses on a simple commonsense ability, reasoning about when an action (or its effect) is feasible. To this end, we introduce FeasibilityQA, a question-answering dataset involving binary classification (BCQ) and multi-choice multi-correct questions (MCQ) that test understanding of feasibility. We show that even state-of-the-art models such as GPT-3, GPT-2, and T5 struggle to answer the feasibility questions correctly. Specifically, on MCQ and BCQ questions, GPT-3 achieves an accuracy of just (19%, 62%) and (25%, 64%) in zero-shot and few-shot settings, respectively. We also evaluate models by providing relevant knowledge statements required to answer the question. We find that the additional knowledge leads to a 7% gain in performance, but the overall performance still remains low. These results make one wonder how much commonsense knowledge about action feasibility is encoded in state-of-the-art models and how well they can reason about it.