OpenRaiser/Intern-Atlas huggingface dataset:
Improving Uncertainty Quantification of Variance Networks by Tree-Structured Learning
Venue: arXiv cs.LG 2023
Year: 2023

To improve the uncertainty quantification of variance networks, we propose a novel tree-structured local neural network model that partitions the feature space into multiple regions based on uncertainty heterogeneity. A tree is built upon giving the training data, whose leaf nodes represent different regions where region-specific neural networks are trained to predict both the mean and the variance for quantifying uncertainty. The proposed Uncertainty-Splitting Neural Regression Tree (USNRT) employs novel splitting criteria. At each node, a neural network is trained on the full data first, and a statistical test for the residuals is conducted to find the best split, corresponding to the two sub-regions with the most significant uncertainty heterogeneity between them. USNRT is computationally friendly because very few leaf nodes are sufficient and pruning is unnecessary. Furthermore, an ensemble version can be easily constructed to estimate the total uncertainty including the aleatory and epistemic. On extensive UCI datasets, USNRT or its ensemble shows superior performance compared to some recent popular methods for quantifying uncertainty with variances. Through comprehensive visualization and analysis, we uncover how USNRT works and show its merits, revealing that uncertainty heterogeneity does exist in many datasets and can be learned by USNRT.
On Realization of Intelligent Decision-Making in the Real World: A Foundation Decision Model Perspective
Venue: arXiv cs.AI 2023
Year: 2023

The pervasive uncertainty and dynamic nature of real-world environments present significant challenges for the widespread implementation of machine-driven Intelligent Decision-Making (IDM) systems. Consequently, IDM should possess the ability to continuously acquire new skills and effectively generalize across a broad range of applications. The advancement of Artificial General Intelligence (AGI) that transcends task and application boundaries is critical for enhancing IDM. Recent studies have extensively investigated the Transformer neural architecture as a foundational model for various tasks, including computer vision, natural language processing, and reinforcement learning. We propose that a Foundation Decision Model (FDM) can be developed by formulating diverse decision-making tasks as sequence decoding tasks using the Transformer architecture, offering a promising solution for expanding IDM applications in complex real-world situations. In this paper, we discuss the efficiency and generalization improvements offered by a foundation decision model for IDM and explore its potential applications in multi-agent game AI, production scheduling, and robotics tasks. Lastly, we present a case study demonstrating our FDM implementation, DigitalBrain (DB1) with 1.3 billion parameters, achieving human-level performance in 870 tasks, such as text generation, image captioning, video game playing, robotic control, and traveling salesman problems. As a foundation decision model, DB1 represents an initial step toward more autonomous and efficient real-world IDM applications.
Computational Complexity of Minimal Trap Spaces in Boolean Networks
Venue: arXiv cs.DM 2023
Year: 2023

A Boolean network (BN) is a discrete dynamical system defined by a Boolean function that maps to the domain itself. A trap space of a BN is a generalization of a fixed point, which is defined as the sub-hypercubes closed by the function of the BN. A trap space is minimal if it does not contain any smaller trap space. Minimal trap spaces have applications for the analysis of attractors of BNs with various update modes. This paper establishes the computational complexity results of three decision problems related to minimal trap spaces: the decision of the trap space property of a sub-hypercube, the decision of its minimality, and the decision of the membership of a given configuration to a minimal trap space. Under several cases on Boolean function representations, we investigate the computational complexity of each problem. In the general case, we demonstrate that the trap space property is coNP-complete, and the minimality and the membership properties are $Π_2^{\text P}$-complete. The complexities drop by one level in the polynomial hierarchy whenever the local functions of the BN are either unate, or are represented using truth-tables, binary decision diagrams, or double DNFs (Petri net encoding): the trap space property can be decided in a polynomial time, whereas deciding the minimality and the membership are coNP- complete. When the BN is given as its functional graph, all these problems are in P.
Neural Shape Compiler: A Unified Framework for Transforming between Text, Point Cloud, and Program
Venue: arXiv cs.CV 2023
Year: 2023

3D shapes have complementary abstractions from low-level geometry to part-based hierarchies to languages, which convey different levels of information. This paper presents a unified framework to translate between pairs of shape abstractions: $\textit{Text}$ $\Longleftrightarrow$ $\textit{Point Cloud}$ $\Longleftrightarrow$ $\textit{Program}$. We propose $\textbf{Neural Shape Compiler}$ to model the abstraction transformation as a conditional generation process. It converts 3D shapes of three abstract types into unified discrete shape code, transforms each shape code into code of other abstract types through the proposed $\textit{ShapeCode Transformer}$, and decodes them to output the target shape abstraction. Point Cloud code is obtained in a class-agnostic way by the proposed $\textit{Point}$VQVAE. On Text2Shape, ShapeGlot, ABO, Genre, and Program Synthetic datasets, Neural Shape Compiler shows strengths in $\textit{Text}$ $\Longrightarrow$ $\textit{Point Cloud}$, $\textit{Point Cloud}$ $\Longrightarrow$ $\textit{Text}$, $\textit{Point Cloud}$ $\Longrightarrow$ $\textit{Program}$, and Point Cloud Completion tasks. Additionally, Neural Shape Compiler benefits from jointly training on all heterogeneous data and tasks.
Navigation Method Enhancing Music Listening Experience by Stimulating Both Neck Sides with Modulated Music Vibration
Venue: arXiv cs.HC 2023
Year: 2023

We propose a method that stimulates musical vibration (generated from and synchronized with musical signals), modulated by the direction and distance to the target, on both sides of a user's neck with Hapbeat, a necklace-type haptic device.We conducted three experiments to confirm that the proposed method can achieve both haptic navigation and enhance the music-listening experience.Experiment 1 consisted of conducting a questionnaire survey to examine the effect of stimulating musical vibrations.Experiment 2 evaluated the accuracy (deg) of users' ability to adjust their direction toward a target using the proposed method.Experiment 3 examined the ability of four different navigation methods by performing navigation tasks in a virtual environment.The results of the experiments showed that stimulating musical vibration enhanced the music-listening experience, and that the proposed method is able to provide sufficient information to guide the users: accuracy in identifying directions was about 20 deg, participants reached the target in all navigation tasks, and in about 80% of all trials participants reached the target using the shortest route.Furthermore, the proposed method succeeded in conveying distance information, and Hapbeat can be combined with conventional navigation methods without interfering with music listening.
Highlighting Named Entities in Input for Auto-Formulation of Optimization Problems
Venue: arXiv cs.CL 2023
Year: 2023

Operations research deals with modeling and solving real-world problems as mathematical optimization problems. While solving mathematical systems is accomplished by analytical software, formulating a problem as a set of mathematical operations has been typically done manually by domain experts. Recent machine learning methods have shown promise in converting textual problem descriptions to corresponding mathematical formulations. This paper presents an approach that converts linear programming word problems into mathematical formulations. We leverage the named entities in the input and augment the input to highlight these entities. Our approach achieves the highest accuracy among all submissions to the NL4Opt Competition, securing first place in the generation track.
Monoidal Width
Venue: arXiv cs.LO 2023
Year: 2023

We introduce monoidal width as a measure of complexity for morphisms in monoidal categories. Inspired by well-known structural width measures for graphs, like tree width and rank width, monoidal width is based on a notion of syntactic decomposition: a monoidal decomposition of a morphism is an expression in the language of monoidal categories, where operations are monoidal products and compositions, that specifies this morphism. Monoidal width penalises the composition operation along ``big'' objects, while it encourages the use of monoidal products. We show that, by choosing the correct categorical algebra for decomposing graphs, we can capture tree width and rank width. For matrices, monoidal width is related to the rank. These examples suggest monoidal width as a good measure for structural complexity of processes modelled as morphisms in monoidal categories.
Behavioral Cloning via Search in Video PreTraining Latent Space
Venue: arXiv cs.LG 2023
Year: 2023

Our aim is to build autonomous agents that can solve tasks in environments like Minecraft. To do so, we used an imitation learning-based approach. We formulate our control problem as a search problem over a dataset of experts' demonstrations, where the agent copies actions from a similar demonstration trajectory of image-action pairs. We perform a proximity search over the BASALT MineRL-dataset in the latent representation of a Video PreTraining model. The agent copies the actions from the expert trajectory as long as the distance between the state representations of the agent and the selected expert trajectory from the dataset do not diverge. Then the proximity search is repeated. Our approach can effectively recover meaningful demonstration trajectories and show human-like behavior of an agent in the Minecraft environment.
Hybrid Message Passing Algorithm for Downlink FDD Massive MIMO-OFDM Channel Estimation
Venue: arXiv cs.IT 2023
Year: 2023

The design of message passing (MP) algorithms on factor graphs is an effective manner to implement channel estimation (CE) in wireless communication systems, which performance can be further improved by exploiting prior probability models that accurately match the channel characteristics. In this work, we study the CE problem in a downlink massive multiple-input multiple-output (MIMO) orthogonal frequency division multiplexing (OFDM) system. As the prior probability, we propose the Markov chain two-state Gaussian mixture with large variance differences (TSGM-LVD) model to exploit the structured sparsity in the angle-frequency domain of the channel. Existing single and combined MP rules cannot deal with the message computation of the proposed probability model. To overcome this issue, we present a general method to derive the hybrid message passing (HMP) rule, which allows the calculation of messages described by mixed linear and non-linear functions. Accordingly, we design the HMP-TSGM-LVD algorithm under the structured turbo framework (STF). Simulation results demonstrate that the proposed algorithm converges faster and obtains better and more stable performance than its counterparts. In particular, the gain of the proposed approach is maximum (3 dB) in the high signal-to-noise ratio regime, while benchmark approaches experience oscillating behavior due to the improper prior model characterization.
Robust Cross-vendor Mammographic Texture Models Using Augmentation-based Domain Adaptation for Long-term Breast Cancer Risk
Venue: arXiv cs.CV 2023
Year: 2023

Purpose: Risk-stratified breast cancer screening might improve early detection and efficiency without comprising quality. However, modern mammography-based risk models do not ensure adaptation across vendor-domains and rely on cancer precursors, associated with short-term risk, which might limit long-term risk assessment. We report a cross-vendor mammographic texture model for long-term risk. Approach: The texture model was robustly trained using two systematically designed case-control datasets. Textural features, indicative of future breast cancer, were learned by excluding samples with diagnosed/potential malignancies from training. An augmentation-based domain adaption technique, based on flavorization of mammographic views, ensured generalization across vendor-domains. The model was validated in 66,607 consecutively screened Danish women with flavorized Siemens views and 25,706 Dutch women with Hologic-processed views. Performances were evaluated for interval cancers (IC) within two years from screening and long-term cancers (LTC) from two years after screening. The texture model was combined with established risk factors to flag 10% of women with the highest risk. Results: In Danish women, the texture model achieved an area under the receiver operating characteristic (AUC) of 0.71 and 0.65 for ICs and LTCs, respectively. In Dutch women with Hologic-processed views, the AUCs were not different from AUCs in Danish women with flavorized views. The AUC for texture combined with established risk factors increased to 0.68 for LTCs. The 10% of women flagged as high-risk accounted for 25.5% of ICs and 24.8% of LTCs. Conclusions: The texture model robustly estimated long-term breast cancer risk while adapting to an unseen processed vendor-domain and identified a clinically relevant high-risk subgroup.
A Survey on Table-and-Text HybridQA: Concepts, Methods, Challenges and Future Directions
Venue: arXiv cs.CL 2023
Year: 2023

Table-and-text hybrid question answering (HybridQA) is a widely used and challenging NLP task commonly applied in the financial and scientific domain. The early research focuses on migrating other QA task methods to HybridQA, while with further research, more and more HybridQA-specific methods have been present. With the rapid development of HybridQA, the systematic survey is still under-explored to summarize the main techniques and advance further research. So we present this work to summarize the current HybridQA benchmarks and methods, then analyze the challenges and future directions of this task. The contributions of this paper can be summarized in three folds: (1) first survey, to our best knowledge, including benchmarks, methods and challenges for HybridQA; (2) systematic investigation with the reasonable comparison of the existing systems to articulate their advantages and shortcomings; (3) detailed analysis of challenges in four important dimensions to shed light on future directions.
Inferring Displacement Fields from Sparse Measurements Using the Statistical Finite Element Method
Venue: arXiv cs.CE 2023
Year: 2023

A well-established approach for inferring full displacement and stress fields from possibly sparse data is to calibrate the parameter of a given constitutive model using a Bayesian update. After calibration, a (stochastic) forward simulation is conducted with the identified model parameters to resolve physical fields in regions that were not accessible to the measurement device. A shortcoming of model calibration is that the model is deemed to best represent reality, which is only sometimes the case, especially in the context of the aging of structures and materials. While this issue is often addressed with repeated model calibration, a different approach is followed in the recently proposed statistical Finite Element Method (statFEM). Instead of using Bayes' theorem to update model parameters, the displacement is chosen as the stochastic prior and updated to fit the measurement data more closely. For this purpose, the statFEM framework introduces a so-called model-reality mismatch, parametrized by only three hyperparameters. This makes the inference of full-field data computationally efficient in an online stage: If the stochastic prior can be computed offline, solving the underlying partial differential equation (PDE) online is unnecessary. Compared to solving a PDE, identifying only three hyperparameters and conditioning the state on the sensor data requires much fewer computational resources. This paper presents two contributions to the existing statFEM approach: First, we use a non-intrusive polynomial chaos method to compute the prior, enabling the use of complex mechanical models in deterministic formulations. Second, we examine the influence of prior material models (linear elastic and St.Venant Kirchhoff material with uncertain Young's modulus) on the updated solution. We present statFEM results for 1D and 2D examples, while an extension to 3D is straightforward.
Limitations of Information-Theoretic Generalization Bounds for Gradient Descent Methods in Stochastic Convex Optimization
Venue: arXiv cs.LG 2023
Year: 2023

To date, no "information-theoretic" frameworks for reasoning about generalization error have been shown to establish minimax rates for gradient descent in the setting of stochastic convex optimization. In this work, we consider the prospect of establishing such rates via several existing information-theoretic frameworks: input-output mutual information bounds, conditional mutual information bounds and variants, PAC-Bayes bounds, and recent conditional variants thereof. We prove that none of these bounds are able to establish minimax rates. We then consider a common tactic employed in studying gradient methods, whereby the final iterate is corrupted by Gaussian noise, producing a noisy "surrogate" algorithm. We prove that minimax rates cannot be established via the analysis of such surrogates. Our results suggest that new ideas are required to analyze gradient descent using information-theoretic techniques.
FoldingNet Autoencoder model to create a geospatial grouping of CityGML building dataset
Venue: arXiv cs.CY 2023
Year: 2023

Explainable numerical representations or latent information of otherwise complex datasets are more convenient to analyze and study. These representations assist in identifying clusters and outliers, assess similar data points, and explore and interpolate data. Dataset of three-dimensional (3D) building models possesses inherent complexity in various footprint shapes, distinct roof types, walls, height, and volume. Traditionally, grouping similar buildings or 3D shapes requires matching their known properties and shape metrics with each other. However, this requires obtaining a plethora of such properties to calculate similarity. This study, in contrast, utilizes an autoencoder to compute the shape information in a fixed-size vector form that can be compared and grouped with the help of distance metrics. The study uses 'FoldingNet,' a 3D autoencoder, to generate the latent representation of each building from the obtained LoD 2 CityGML dataset. The efficacy of the embeddings obtained from the autoencoder is further analyzed by dataset reconstruction, latent spread visualization, and hierarchical clustering methods. While the clusters give an overall perspective of the type of build forms, they do not include geospatial information in the clustering. A geospatial model is therefore created to iteratively find the geographical groupings of buildings using cosine similarity approaches in embedding vectors. The German federal states of Brandenburg and Berlin are taken as an example to test the methodology. The output provides a detailed overview of the build forms in the form of semantic topological clusters and geographical groupings. This approach is beneficial and scalable for complex analytics, e.g., in large urban simulations, urban morphological studies, energy analysis, or evaluations of building stock.
Assisted Living in the United States: an Open Dataset
Venue: arXiv cs.CY 2023
Year: 2023

An assisted living facility (ALF) is a place where someone can live, have access to social supports such as transportation, and receive assistance with the activities of daily living such as toileting and dressing. Despite the important role of ALFs, they are not required to be certified with Medicare and there is no public national database of these facilities. We present the first public dataset of ALFs in the United States, covering all 50 states and DC with 44,638 facilities and over 1.2 million beds. This dataset can help provide answers to existing public health questions as well as help those in need find a facility. The dataset was validated by replicating the results of a nationwide study of ALFs that uses closed data [4], where the prevalence of ALFs is assessed with respect to county-level socioeconomic variables related to health disparity such as race, disability, and income. To showcase the value of this dataset, we also propose a novel metric to assess access to community-based care. We calculate the average distance an individual in need must travel in order to reach an ALF. The dataset and all relevant code are available at github.com/antonstengel/assisted-living-data.
Robust representations of oil wells' intervals via sparse attention mechanism
Venue: arXiv cs.LG 2023
Year: 2023

Transformer-based neural network architectures achieve state-of-the-art results in different domains, from natural language processing (NLP) to computer vision (CV). The key idea of Transformers, the attention mechanism, has already led to significant breakthroughs in many areas. The attention has found their implementation for time series data as well. However, due to the quadratic complexity of the attention calculation regarding input sequence length, the application of Transformers is limited by high resource demands. Moreover, their modifications for industrial time series need to be robust to missing or noised values, which complicates the expansion of the horizon of their application. To cope with these issues, we introduce the class of efficient Transformers named Regularized Transformers (Reguformers). We implement the regularization technique inspired by the dropout ideas to improve robustness and reduce computational expenses. The focus in our experiments is on oil&gas data, namely, well logs, a prominent example of multivariate time series. The goal is to solve the problems of similarity and representation learning for them. To evaluate our models for such problems, we work with an industry-scale open dataset consisting of well logs of more than 20 wells. The experiments show that all variations of Reguformers outperform the previously developed RNNs, classical Transformer model, and robust modifications of it like Informer and Performer in terms of well-intervals' classification and the quality of the obtained well-intervals' representations. Moreover, the sustainability to missing and incorrect data in our models exceeds that of others by a significant margin. The best result that the Reguformer achieves on well-interval similarity task is the mean PR~AUC score equal to 0.983, which is comparable to the classical Transformer and outperforms the previous models.