OpenRaiser/Intern-Atlas huggingface dataset:
Domain-informed graph neural networks: a quantum chemistry case study
Venue: arXiv cs.LG 2022
Year: 2022

We explore different strategies to integrate prior domain knowledge into the design of a deep neural network (DNN). We focus on graph neural networks (GNN), with a use case of estimating the potential energy of chemical systems (molecules and crystals) represented as graphs. We integrate two elements of domain knowledge into the design of the GNN to constrain and regularise its learning, towards higher accuracy and generalisation. First, knowledge on the existence of different types of relations (chemical bonds) between atoms is used to modulate the interaction of nodes in the GNN. Second, knowledge of the relevance of some physical quantities is used to constrain the learnt features towards a higher physical relevance using a simple multi-task paradigm. We demonstrate the general applicability of our knowledge integrations by applying them to two architectures that rely on different mechanisms to propagate information between nodes and to update node states.
Learning to Construct 3D Building Wireframes from 3D Line Clouds
Venue: arXiv cs.CV 2022
Year: 2022

Line clouds, though under-investigated in the previous work, potentially encode more compact structural information of buildings than point clouds extracted from multi-view images. In this work, we propose the first network to process line clouds for building wireframe abstraction. The network takes a line cloud as input , i.e., a nonstructural and unordered set of 3D line segments extracted from multi-view images, and outputs a 3D wireframe of the underlying building, which consists of a sparse set of 3D junctions connected by line segments. We observe that a line patch, i.e., a group of neighboring line segments, encodes sufficient contour information to predict the existence and even the 3D position of a potential junction, as well as the likelihood of connectivity between two query junctions. We therefore introduce a two-layer Line-Patch Transformer to extract junctions and connectivities from sampled line patches to form a 3D building wireframe model. We also introduce a synthetic dataset of multi-view images with ground-truth 3D wireframe. We extensively justify that our reconstructed 3D wireframe models significantly improve upon multiple baseline building reconstruction methods. The code and data can be found at https://github.com/Luo1Cheng/LC2WF.
FusePose: IMU-Vision Sensor Fusion in Kinematic Space for Parametric Human Pose Estimation
Venue: arXiv cs.CV 2022
Year: 2022

There exist challenging problems in 3D human pose estimation mission, such as poor performance caused by occlusion and self-occlusion. Recently, IMU-vision sensor fusion is regarded as valuable for solving these problems. However, previous researches on the fusion of IMU and vision data, which is heterogeneous, fail to adequately utilize either IMU raw data or reliable high-level vision features. To facilitate a more efficient sensor fusion, in this work we propose a framework called \emph{FusePose} under a parametric human kinematic model. Specifically, we aggregate different information of IMU or vision data and introduce three distinctive sensor fusion approaches: NaiveFuse, KineFuse and AdaDeepFuse. NaiveFuse servers as a basic approach that only fuses simplified IMU data and estimated 3D pose in euclidean space. While in kinematic space, KineFuse is able to integrate the calibrated and aligned IMU raw data with converted 3D pose parameters. AdaDeepFuse further develops this kinematical fusion process to an adaptive and end-to-end trainable manner. Comprehensive experiments with ablation studies demonstrate the rationality and superiority of the proposed framework. The performance of 3D human pose estimation is improved compared to the baseline result. On Total Capture dataset, KineFuse surpasses previous state-of-the-art which uses IMU only for testing by 8.6\%. AdaDeepFuse surpasses state-of-the-art which uses IMU for both training and testing by 8.5\%. Moreover, we validate the generalization capability of our framework through experiments on Human3.6M dataset.
Joint Scheduling and Coding for Reliable, Latency-bounded Transmission over Parallel Wireless Links
Venue: arXiv cs.NI 2022
Year: 2022

Several novel industrial applications involve human control of vehicles, cranes, or mobile robots through various high-throughput feedback systems, such as Virtual Reality (VR) and tactile/haptic signals. The near real-time interaction between the system and the operator requires strict latency constraints in packet exchange, which is difficult to guarantee over wireless communication links. In this work, we advocate that packet-level coding and packet scheduling over multiple parallel (unreliable) links have the potential to provide reliable, latency-bounded communication for applications with periodic data generation patterns. However, this goal can be reached only through a careful joint design of such mechanisms, whose interactions can be subtle and difficult to predict. In this paper we first discuss these aspects in general terms, and then present a Markov Decision Process (MDP) model that can be used to find a scheme that optimally exploits the multichannel wireless access in order to maximize the fraction of data blocks delivered within deadline. Our illustrative example is then used to show the optimal coding/scheduling strategies under different combinations of wireless links, also showing that the common solution of backing up a high bitrate unreliable mmWave link with a low bitrate more stable sub-6 GHz link can actually be ineffective in the considered scenario
Coherent photonic crossbar as a universal linear operator
Venue: arXiv cs.ET 2022
Year: 2022

Linear optics aim at realizing any real- and/or complex-valued matrix operator via optical elements, addressing a broad field of applications in the areas of quantum photonics, microwave photonics and optical neural networks. The transfer of linear operators into photonic experimental layouts typically relies on Singular Value Decomposition (SVD) techniques combining meshes of cascaded 2x2 Mach Zehnder Interferometers (MZIs), with the main challenges being the precision in the experimental representation of the targeted matrix, referred to as fidelity, and the overall insertion loss. We demonstrate a novel interferometric coherent photonic crossbar architecture (Xbar) that demarcates from state-of-the-art SVD implementations and can realize any linear operator, supporting full restoration of the loss-induced fidelity. Its novel interferometric design allows for the direct mapping of each matrix element to a single, designated Xbar node, bringing down the number of programming steps to only one. We present the theoretical foundations of the Xbar, proving that its insertion losses scale linearly with the node losses as opposed to the exponential scaling witnessed by the SVD counterparts. This leads to a matrix design with significantly lower overall insertion losses compared to SVD-based schemes when utilizing state-of-the-art silicon photonic fabrication metrics, allowing for alternative node technologies with lower energy consumption and higher operational speed credentials to be employed. Finally, we validate that our Xbar architecture is the first linear operator that supports fidelity restoration, outperforming SVD schemes in loss- and phase-error fidelity performance and forming a significantly more robust layout to loss and phase deviations.
A Study on Broadcast Networks for Music Genre Classification
Venue: arXiv cs.SD 2022
Year: 2022

Due to the increased demand for music streaming/recommender services and the recent developments of music information retrieval frameworks, Music Genre Classification (MGC) has attracted the community's attention. However, convolutional-based approaches are known to lack the ability to efficiently encode and localize temporal features. In this paper, we study the broadcast-based neural networks aiming to improve the localization and generalizability under a small set of parameters (about 180k) and investigate twelve variants of broadcast networks discussing the effect of block configuration, pooling method, activation function, normalization mechanism, label smoothing, channel interdependency, LSTM block inclusion, and variants of inception schemes. Our computational experiments using relevant datasets such as GTZAN, Extended Ballroom, HOMBURG, and Free Music Archive (FMA) show state-of-the-art classification accuracies in Music Genre Classification. Our approach offers insights and the potential to enable compact and generalizable broadcast networks for music and audio classification.
A Survey on Temporal Graph Representation Learning and Generative Modeling
Venue: arXiv cs.LG 2022
Year: 2022

Temporal graphs represent the dynamic relationships among entities and occur in many real life application like social networks, e commerce, communication, road networks, biological systems, and many more. They necessitate research beyond the work related to static graphs in terms of their generative modeling and representation learning. In this survey, we comprehensively review the neural time dependent graph representation learning and generative modeling approaches proposed in recent times for handling temporal graphs. Finally, we identify the weaknesses of existing approaches and discuss the research proposal of our recently published paper TIGGER[24].
An analysis of load-balancing algorithms on edge-Markovian evolving graphs
Venue: arXiv cs.DM 2022
Year: 2022

Analysis of algorithms on time-varying networks (often called evolving graphs) is a modern challenge in theoretical computer science. The edge-Markovian is a relatively simple and comprehensive model of evolving graphs: every pair of vertices which is not a current edge independently becomes an edge with probability $p$ at each time-step, as well as every edge disappears with probability $q$. Clearly, the edge-Markovian graph changes its shape depending on the current shape, and the dependency refuses some useful techniques for an independent sequence of random graphs which often behaves similarly to a static random graph. It motivates this paper to develop a new technique for analysis of algorithms on edge-Markovian evolving graphs. Specifically speaking, this paper is concerned with load-balancing, which is a popular subject in distributed computing, and we analyze the so-called random matching algorithms, which is a standard scheme for load-balancing. We prove that major random matching algorithms achieve nearly optimal load balance in $O(r \log (Δn))$ steps on edge-Markovian evolving graphs, where $r = \max\{p/(1-q), (1-q)/p\}$, $n$ is the number of vertices (i.e., processors) and $Δ$ denotes the initial gap of loads unbalance. We remark that the independent sequences of random graphs correspond to $r=1$. To avoid the difficulty of an analysis caused by a complex correlation with the history of an execution, we develop a simple proof technique based on history-independent bounds. As far as we know, this is the first theoretical analysis of load-balancing on randomly evolving graphs, not only for the edge-Markovian but also for the independent sequences of random graphs.
Image Based Food Energy Estimation With Depth Domain Adaptation
Venue: arXiv cs.CV 2022
Year: 2022

Assessment of dietary intake has primarily relied on self-report instruments, which are prone to measurement errors. Dietary assessment methods have increasingly incorporated technological advances particularly mobile, image based approaches to address some of these limitations and further automation. Mobile, image-based methods can reduce user burden and bias by automatically estimating dietary intake from eating occasion images that are captured by mobile devices. In this paper, we propose an "Energy Density Map" which is a pixel-to-pixel mapping from the RGB image to the energy density of the food. We then incorporate the "Energy Density Map" with an associated depth map that is captured by a depth sensor to estimate the food energy. The proposed method is evaluated on the Nutrition5k dataset. Experimental results show improved results compared to baseline methods with an average error of 13.29 kCal and an average percentage error of 13.57% between the ground-truth and the estimated energy of the food.
CAS4DL: Christoffel Adaptive Sampling for function approximation via Deep Learning
Venue: arXiv cs.LG 2022
Year: 2022

The problem of approximating smooth, multivariate functions from sample points arises in many applications in scientific computing, e.g., in computational Uncertainty Quantification (UQ) for science and engineering. In these applications, the target function may represent a desired quantity of interest of a parameterized Partial Differential Equation (PDE). Due to the large cost of solving such problems, where each sample is computed by solving a PDE, sample efficiency is a key concerning these applications. Recently, there has been increasing focus on the use of Deep Neural Networks (DNN) and Deep Learning (DL) for learning such functions from data. In this work, we propose an adaptive sampling strategy, CAS4DL (Christoffel Adaptive Sampling for Deep Learning) to increase the sample efficiency of DL for multivariate function approximation. Our novel approach is based on interpreting the second to last layer of a DNN as a dictionary of functions defined by the nodes on that layer. With this viewpoint, we then define an adaptive sampling strategy motivated by adaptive sampling schemes recently proposed for linear approximation schemes, wherein samples are drawn randomly with respect to the Christoffel function of the subspace spanned by this dictionary. We present numerical experiments comparing CAS4DL with standard Monte Carlo (MC) sampling. Our results demonstrate that CAS4DL often yields substantial savings in the number of samples required to achieve a given accuracy, particularly in the case of smooth activation functions, and it shows a better stability in comparison to MC. These results therefore are a promising step towards fully adapting DL towards scientific computing applications.
Masked Autoencoders Enable Efficient Knowledge Distillers
Venue: arXiv cs.CV 2022
Year: 2022

This paper studies the potential of distilling knowledge from pre-trained models, especially Masked Autoencoders. Our approach is simple: in addition to optimizing the pixel reconstruction loss on masked inputs, we minimize the distance between the intermediate feature map of the teacher model and that of the student model. This design leads to a computationally efficient knowledge distillation framework, given 1) only a small visible subset of patches is used, and 2) the (cumbersome) teacher model only needs to be partially executed, ie, forward propagate inputs through the first few layers, for obtaining intermediate feature maps. Compared to directly distilling fine-tuned models, distilling pre-trained models substantially improves downstream performance. For example, by distilling the knowledge from an MAE pre-trained ViT-L into a ViT-B, our method achieves 84.0% ImageNet top-1 accuracy, outperforming the baseline of directly distilling a fine-tuned ViT-L by 1.2%. More intriguingly, our method can robustly distill knowledge from teacher models even with extremely high masking ratios: e.g., with 95% masking ratio where merely TEN patches are visible during distillation, our ViT-B competitively attains a top-1 ImageNet accuracy of 83.6%; surprisingly, it can still secure 82.4% top-1 ImageNet accuracy by aggressively training with just FOUR visible patches (98% masking ratio). The code and models are publicly available at https://github.com/UCSC-VLAA/DMAE.
Synthetic End-User Testing: Modeling Realistic Agents Based on Behavioral Examples
Venue: arXiv cs.SE 2022
Year: 2022

For software interacting directly with real-world end-users, it is common practice to script scenario tests validating the system's compliance with a number of its features. However, these do not accommodate the replication of the type of end-user activity to which the system is required to respond in a live instance. It is especially true as compliance might also break in scenarios of interactions with external events or processes, such as other users. State-of-the-art approaches aim at inducing the software into runtime errors by generating tests that maximize some target metrics, such as code coverage. As a result, they suffer from targeting an infinitely large search space, are severely limited in recognizing error states that do not result in runtime errors, and the test cases they generate are often challenging to interpret. Other forms of testing, such as Record-Replay, instead fail to capture the end-users' decision-making process, hence producing largely scripted test scenarios. Therefore, it is impossible to test a software's compliance with unknown but otherwise plausible states. This paper introduces "Synthetic End-User Testing," a novel testing strategy for complex systems in which real-world users are synthesized into reusable agents and employed to test and validate the software in a simulation environment. Hence, it discusses how end-user behavioral examples can be obtained and used to create agents that operate the target software in a reduced search space of likely action sequences. The notion of action expectation, which allows agents to assert the learned compliance of the system, is also introduced. Finally, a prototype asserting the feasibility of such a strategy is presented.
Identifying and Overcoming Transformation Bias in Forecasting Models
Venue: arXiv cs.LG 2022
Year: 2022

Log and square root transformations of target variable are routinely used in forecasting models to predict future sales. These transformations often lead to better performing models. However, they also introduce a systematic negative bias (under-forecasting). In this paper, we demonstrate the existence of this bias, dive deep into its root cause and introduce two methods to correct for the bias. We conclude that the proposed bias correction methods improve model performance (by up to 50%) and make a case for incorporating bias correction in modeling workflow. We also experiment with `Tweedie' family of cost functions which circumvents the transformation bias issue by modeling directly on sales. We conclude that Tweedie regression gives the best performance so far when modeling on sales making it a strong alternative to working with a transformed target variable.
Modelling the Recommender Alignment Problem
Venue: arXiv cs.AI 2022
Year: 2022

Recommender systems (RS) mediate human experience online. Most RS act to optimize metrics that are imperfectly aligned with the best-interest of users but are easy to measure, like ad-clicks and user engagement. This has resulted in a host of hard-to-measure side-effects: political polarization, addiction, fake news. RS design faces a recommender alignment problem: that of aligning recommendations with the goals of users, system designers, and society as a whole. But how do we test and compare potential solutions to align RS? Their massive scale makes them costly and risky to test in deployment. We synthesized a simple abstract modelling framework to guide future work. To illustrate it, we construct a toy experiment where we ask: "How can we evaluate the consequences of using user retention as a reward function?" To answer the question, we learn recommender policies that optimize reward functions by controlling graph dynamics on a toy environment. Based on the effects that trained recommenders have on their environment, we conclude that engagement maximizers generally lead to worse outcomes than aligned recommenders but not always. After learning, we examine competition between RS as a potential solution to RS alignment. We find that it generally makes our toy-society better-off than it would be under the absence of recommendation or engagement maximizers. In this work, we aimed for a broad scope, touching superficially on many different points to shed light on how an end-to-end study of reward functions for recommender systems might be done. Recommender alignment is a pressing and important problem. Attempted solutions are sure to have far-reaching impacts. Here, we take a first step in developing methods to evaluating and comparing solutions with respect to their impacts on society.
Robustness, Scott Continuity, and Computability
Venue: arXiv cs.LO 2022
Year: 2022

Robustness is a property of system analyses, namely monotonic maps from the complete lattice of subsets of a (system's state) space to the two-point lattice. The definition of robustness requires the space to be a metric space. Robust analyses cannot discriminate between a subset of the metric space and its closure, therefore one can restrict to the complete lattice of closed subsets. When the metric space is compact, the complete lattice of closed subsets ordered by reverse inclusion is w-continuous and robust analyses are exactly the Scott continuous maps. Thus, one can also ask whether a robust analysis is computable (with respect to a countable base). The main result of this paper establishes a relation between robustness and Scott continuity, when the metric space is not compact. The key idea is to replace the metric space with a compact Hausdorff space, and relate robustness and Scott continuity by an adjunction between the complete lattice of closed subsets of the metric space and the w-continuous lattice of closed subsets of the compact Hausdorff space. We demonstrate the applicability of this result with several examples involving Banach spaces.
Consensusless Blockchain: A Promising High-Performance Blockchain without Consensus
Venue: arXiv cs.NI 2022
Year: 2022

Consensus is unnecessary when the truth is available. In this paper, we present a new perspective of rebuilding the blockchain without consensus. When the consensus phase is eliminated from a blockchain, transactions could be canonized quickly using a well-defined universal rule without consuming hashing power. Thus, the transactions per second(TPS) metric of such the consensusless blockchain can be largely boosted. Although consensus blockchain is promising, several technical challenges are also crucial. For example, double-spending attacks and frequent forking events must be prevented, the credit of block's minting must be carefully defined, and etc. To address those technical challenges, we propose several solutions for our consensusless blockchain (CB), including a naive monotonic scoring mechanism to calculate the ranking of each block in the chain, and a two-stage witness mechanism to add new blocks. The proposed CB chain is promising to offer a simplified and equipment-cheap infrastructure for rich real-world decentralized applications.
Cross-lingual Transfer Learning for Fake News Detector in a Low-Resource Language
Venue: arXiv cs.CL 2022
Year: 2022

Development of methods to detect fake news (FN) in low-resource languages has been impeded by a lack of training data. In this study, we solve the problem by using only training data from a high-resource language. Our FN-detection system permitted this strategy by applying adversarial learning that transfers the detection knowledge through languages. To assist the knowledge transfer, our system judges the reliability of articles by exploiting source information, which is a cross-lingual feature that represents the credibility of the speaker. In experiments, our system got 3.71% higher accuracy than a system that uses a machine-translated training dataset. In addition, our suggested cross-lingual feature exploitation for fake news detection improved accuracy by 3.03%.