OpenRaiser/Intern-Atlas huggingface dataset:
Uncertainty-Driven Active Vision for Implicit Scene Reconstruction
Venue: arXiv cs.CV 2022
Year: 2022

Multi-view implicit scene reconstruction methods have become increasingly popular due to their ability to represent complex scene details. Recent efforts have been devoted to improving the representation of input information and to reducing the number of views required to obtain high quality reconstructions. Yet, perhaps surprisingly, the study of which views to select to maximally improve scene understanding remains largely unexplored. We propose an uncertainty-driven active vision approach for implicit scene reconstruction, which leverages occupancy uncertainty accumulated across the scene using volume rendering to select the next view to acquire. To this end, we develop an occupancy-based reconstruction method which accurately represents scenes using either 2D or 3D supervision. We evaluate our proposed approach on the ABC dataset and the in the wild CO3D dataset, and show that: (1) we are able to obtain high quality state-of-the-art occupancy reconstructions; (2) our perspective conditioned uncertainty definition is effective to drive improvements in next best view selection and outperforms strong baseline approaches; and (3) we can further improve shape understanding by performing a gradient-based search on the view selection candidates. Overall, our results highlight the importance of view selection for implicit scene reconstruction, making it a promising avenue to explore further.
Uniqueness of logical connectives in a bilateralist setting
Venue: arXiv cs.LO 2022
Year: 2022

In this paper I will show the problems that are encountered when dealing with uniqueness of connectives in a bilateralist setting within the larger framework of proof-theoretic semantics and suggest a solution. Therefore, the logic 2Int is suitable, for which I introduce a sequent calculus system, displaying - just like the corresponding natural deduction system - a consequence relation for provability as well as one dual to provability. I will propose a modified characterization of uniqueness incorporating such a duality of consequence relations, with which we can maintain uniqueness in a bilateralist setting.
Visual Prompt Tuning for Generative Transfer Learning
Venue: arXiv cs.CV 2022
Year: 2022

Transferring knowledge from an image synthesis model trained on a large dataset is a promising direction for learning generative image models from various domains efficiently. While previous works have studied GAN models, we present a recipe for learning vision transformers by generative knowledge transfer. We base our framework on state-of-the-art generative vision transformers that represent an image as a sequence of visual tokens to the autoregressive or non-autoregressive transformers. To adapt to a new domain, we employ prompt tuning, which prepends learnable tokens called prompt to the image token sequence, and introduce a new prompt design for our task. We study on a variety of visual domains, including visual task adaptation benchmark~\cite{zhai2019large}, with varying amount of training images, and show effectiveness of knowledge transfer and a significantly better image generation quality over existing works.
Complexity of Polyadic Boolean Modal Logics: Model Checking and Satisfiability
Venue: arXiv cs.LO 2022
Year: 2022

We study the computational complexity of model checking and satisfiability problems of polyadic modal logics extended with permutations and Boolean operators on accessibility relations. First, we show that the combined complexity of the model checking problem for the resulting logic is PTime-complete. Secondly, we show that the satisfiability problem of polyadic modal logic extended with negation on accessibility relations is ExpTime-complete. Finally, we show that the satisfiability problem of polyadic modal logic with permutations and Boolean operators on accessibility relations is ExpTime-complete, under the necessary assumption that the number of accessibility relations that can be used is bounded by a constant.
Dual-former: Hybrid Self-attention Transformer for Efficient Image Restoration
Venue: arXiv cs.CV 2022
Year: 2022

Recently, image restoration transformers have achieved comparable performance with previous state-of-the-art CNNs. However, how to efficiently leverage such architectures remains an open problem. In this work, we present Dual-former whose critical insight is to combine the powerful global modeling ability of self-attention modules and the local modeling ability of convolutions in an overall architecture. With convolution-based Local Feature Extraction modules equipped in the encoder and the decoder, we only adopt a novel Hybrid Transformer Block in the latent layer to model the long-distance dependence in spatial dimensions and handle the uneven distribution between channels. Such a design eliminates the substantial computational complexity in previous image restoration transformers and achieves superior performance on multiple image restoration tasks. Experiments demonstrate that Dual-former achieves a 1.91dB gain over the state-of-the-art MAXIM method on the Indoor dataset for single image dehazing while consuming only 4.2% GFLOPs as MAXIM. For single image deraining, it exceeds the SOTA method by 0.1dB PSNR on the average results of five datasets with only 21.5% GFLOPs. Dual-former also substantially surpasses the latest desnowing method on various datasets, with fewer parameters.
Decompiling x86 Deep Neural Network Executables
Venue: arXiv cs.CR 2022
Year: 2022

Due to their widespread use on heterogeneous hardware devices, deep learning (DL) models are compiled into executables by DL compilers to fully leverage low-level hardware primitives. This approach allows DL computations to be undertaken at low cost across a variety of computing platforms, including CPUs, GPUs, and various hardware accelerators. We present BTD (Bin to DNN), a decompiler for deep neural network (DNN) executables. BTD takes DNN executables and outputs full model specifications, including types of DNN operators, network topology, dimensions, and parameters that are (nearly) identical to those of the input models. BTD delivers a practical framework to process DNN executables compiled by different DL compilers and with full optimizations enabled on x86 platforms. It employs learning-based techniques to infer DNN operators, dynamic analysis to reveal network architectures, and symbolic execution to facilitate inferring dimensions and parameters of DNN operators. Our evaluation reveals that BTD enables accurate recovery of full specifications of complex DNNs with millions of parameters (e.g., ResNet). The recovered DNN specifications can be re-compiled into a new DNN executable exhibiting identical behavior to the input executable. We show that BTD can boost two representative attacks, adversarial example generation and knowledge stealing, against DNN executables. We also demonstrate cross-architecture legacy code reuse using BTD, and envision BTD being used for other critical downstream tasks like DNN security hardening and patching.
An Almost Singularly Optimal Asynchronous Distributed MST Algorithm
Venue: arXiv cs.DC 2022
Year: 2022

A singularly (near) optimal distributed algorithm is one that is (near) optimal in \emph{two} criteria, namely, its time and message complexities. For \emph{synchronous} CONGEST networks, such algorithms are known for fundamental distributed computing problems such as leader election [Kutten et al., JACM 2015] and Minimum Spanning Tree (MST) construction [Pandurangan et al., STOC 2017, Elkin, PODC 2017]. However, it is open whether a singularly (near) optimal bound can be obtained for the MST construction problem in general \emph{asynchronous} CONGEST networks. We present a randomized distributed MST algorithm that, with high probability, computes an MST in \emph{asynchronous} CONGEST networks and takes $\tilde{O}(D^{1+ε} + \sqrt{n})$ time and $\tilde{O}(m)$ messages, where $n$ is the number of nodes, $m$ the number of edges, $D$ is the diameter of the network, and $ε>0$ is an arbitrarily small constant (both time and message bounds hold with high probability). Our algorithm is message optimal (up to a polylog$(n)$ factor) and almost time optimal (except for a $D^ε$ factor). Our result answers an open question raised in Mashregi and King [DISC 2019] by giving the first known asynchronous MST algorithm that has sublinear time (for all $D = O(n^{1-ε})$) and uses $\tilde{O}(m)$ messages. Using a result of Mashregi and King [DISC 2019], this also yields the first asynchronous MST algorithm that is sublinear in both time and messages in the $KT_1$ CONGEST model. A key tool in our algorithm is the construction of a low diameter rooted spanning tree in asynchronous CONGEST that has depth $\tilde{O}(D^{1+ε})$ (for an arbitrarily small constant $ε> 0$) in $\tilde{O}(D^{1+ε})$ time and $\tilde{O}(m)$ messages. To the best of our knowledge, this is the first such construction that is almost singularly optimal in the asynchronous setting.
Unveiling the Black Box of PLMs with Semantic Anchors: Towards Interpretable Neural Semantic Parsing
Venue: arXiv cs.CL 2022
Year: 2022

The recent prevalence of pretrained language models (PLMs) has dramatically shifted the paradigm of semantic parsing, where the mapping from natural language utterances to structured logical forms is now formulated as a Seq2Seq task. Despite the promising performance, previous PLM-based approaches often suffer from hallucination problems due to their negligence of the structural information contained in the sentence, which essentially constitutes the key semantics of the logical forms. Furthermore, most works treat PLM as a black box in which the generation process of the target logical form is hidden beneath the decoder modules, which greatly hinders the model's intrinsic interpretability. To address these two issues, we propose to incorporate the current PLMs with a hierarchical decoder network. By taking the first-principle structures as the semantic anchors, we propose two novel intermediate supervision tasks, namely Semantic Anchor Extraction and Semantic Anchor Alignment, for training the hierarchical decoders and probing the model intermediate representations in a self-adaptive manner alongside the fine-tuning process. We conduct intensive experiments on several semantic parsing benchmarks and demonstrate that our approach can consistently outperform the baselines. More importantly, by analyzing the intermediate representations of the hierarchical decoders, our approach also makes a huge step toward the intrinsic interpretability of PLMs in the domain of semantic parsing.
Shielding Federated Learning: Mitigating Byzantine Attacks with Less Constraints
Venue: arXiv cs.DC 2022
Year: 2022

Federated learning is a newly emerging distributed learning framework that facilitates the collaborative training of a shared global model among distributed participants with their privacy preserved. However, federated learning systems are vulnerable to Byzantine attacks from malicious participants, who can upload carefully crafted local model updates to degrade the quality of the global model and even leave a backdoor. While this problem has received significant attention recently, current defensive schemes heavily rely on various assumptions, such as a fixed Byzantine model, availability of participants' local data, minority attackers, IID data distribution, etc. To relax those constraints, this paper presents Robust-FL, the first prediction-based Byzantine-robust federated learning scheme where none of the assumptions is leveraged. The core idea of the Robust-FL is exploiting historical global model to construct an estimator based on which the local models will be filtered through similarity detection. We then cluster local models to adaptively adjust the acceptable differences between the local models and the estimator such that Byzantine users can be identified. Extensive experiments over different datasets show that our approach achieves the following advantages simultaneously: (i) independence of participants' local data, (ii) tolerance of majority attackers, (iii) generalization to variable Byzantine model.
Long-Term Localization using Semantic Cues in Floor Plan Maps
Venue: arXiv cs.RO 2022
Year: 2022

Lifelong localization in a given map is an essential capability for autonomous service robots. In this paper, we consider the task of long-term localization in a changing indoor environment given sparse CAD floor plans. The commonly used pre-built maps from the robot sensors may increase the cost and time of deployment. Furthermore, their detailed nature requires that they are updated when significant changes occur. We address the difficulty of localization when the correspondence between the map and the observations is low due to the sparsity of the CAD map and the changing environment. To overcome both challenges, we propose to exploit semantic cues that are commonly present in human-oriented spaces. These semantic cues can be detected using RGB cameras by utilizing object detection, and are matched against an easy-to-update, abstract semantic map. The semantic information is integrated into a Monte Carlo localization framework using a particle filter that operates on 2D LiDAR scans and camera data. We provide a long-term localization solution and a semantic map format, for environments that undergo changes to their interior structure and detailed geometric maps are not available. We evaluate our localization framework on multiple challenging indoor scenarios in an office environment, taken weeks apart. The experiments suggest that our approach is robust to structural changes and can run on an onboard computer. We released the open source implementation of our approach written in C++ together with a ROS wrapper.
Generative Models and Learning Algorithms for Core-Periphery Structured Graphs
Venue: arXiv cs.LG 2022
Year: 2022

We consider core-periphery structured graphs, which are graphs with a group of densely and sparsely connected nodes, respectively, referred to as core and periphery nodes. The so-called core score of a node is related to the likelihood of it being a core node. In this paper, we focus on learning the core scores of a graph from its node attributes and connectivity structure. To this end, we propose two classes of probabilistic graphical models: affine and nonlinear. First, we describe affine generative models to model the dependence of node attributes on its core scores, which determine the graph structure. Next, we discuss nonlinear generative models in which the partial correlations of node attributes influence the graph structure through latent core scores. We develop algorithms for inferring the model parameters and core scores of a graph when both the graph structure and node attributes are available. When only the node attributes of graphs are available, we jointly learn a core-periphery structured graph and its core scores. We provide results from numerical experiments on several synthetic and real-world datasets to demonstrate the efficacy of the developed models and algorithms.
On Background Bias in Deep Metric Learning
Venue: arXiv cs.CV 2022
Year: 2022

Deep Metric Learning trains a neural network to map input images to a lower-dimensional embedding space such that similar images are closer together than dissimilar images. When used for item retrieval, a query image is embedded using the trained model and the closest items from a database storing their respective embeddings are returned as the most similar items for the query. Especially in product retrieval, where a user searches for a certain product by taking a photo of it, the image background is usually not important and thus should not influence the embedding process. Ideally, the retrieval process always returns fitting items for the photographed object, regardless of the environment the photo was taken in. In this paper, we analyze the influence of the image background on Deep Metric Learning models by utilizing five common loss functions and three common datasets. We find that Deep Metric Learning networks are prone to so-called background bias, which can lead to a severe decrease in retrieval performance when changing the image background during inference. We also show that replacing the background of images during training with random background images alleviates this issue. Since we use an automatic background removal method to do this background replacement, no additional manual labeling work and model changes are required while inference time stays the same. Qualitative and quantitative analyses, for which we introduce a new evaluation metric, confirm that models trained with replaced backgrounds attend more to the main object in the image, benefitting item retrieval systems.
Location-aware green energy availability forecasting for multiple time frames in smart buildings: The case of Estonia
Venue: arXiv cs.LG 2022
Year: 2022

Renewable Energies (RE) have gained more attention in recent years since they offer clean and sustainable energy. One of the major sustainable development goals (SDG-7) set by the United Nations (UN) is to achieve affordable and clean energy for everyone. Among the world's all renewable resources, solar energy is considered as the most abundant and can certainly fulfill the target of SDGs. Solar energy is converted into electrical energy through Photovoltaic (PV) panels with no greenhouse gas emissions. However, power generated by PV panels is highly dependent on solar radiation received at a particular location over a given time period. Therefore, it is challenging to forecast the amount of PV output power. Predicting the output power of PV systems is essential since several public or private institutes generate such green energy, and need to maintain the balance between demand and supply. This research aims to forecast PV system output power based on weather and derived features using different machine learning models. The objective is to obtain the best-fitting model to precisely predict output power by inspecting the data. Moreover, different performance metrics are used to compare and evaluate the accuracy under different machine learning models such as random forest, XGBoost, KNN, etc.
Codeless App Development: Evaluating A Cloud-Native Domain-Specific Functions Approach
Venue: arXiv cs.SE 2022
Year: 2022

Mobile applications play an important role in the economy today and there is an increasing trend for app enablement on multiple platforms. However, creating, distributing, and maintaining an application remain expert tasks. Even for software developers, the process can be error-prone and resource-consuming, especially when targeting different platforms simultaneously. Researchers have proposed several frameworks to facilitate cross-platform app development, but little attention has been paid to non-technical users. In this paper, we described the Flow framework, which takes the advantage of domain-specific languages to enable no-code specification for app modeling. The cloud-native coordination mechanism further supports non-technical users to execute, monitor, and maintain apps for any target platforms. User evaluations were conducted to assess the usability and user experience with the system. The results indicated that users can develop apps in Flow with ease, but the prototype could be optimized to reduce learning time and workload.
Conceptual Design of Cellular Auxetic Systems with Passive Adaptation to Loading
Venue: arXiv cs.CE 2022
Year: 2022

Auxetics refer to a class of engineered structures which exhibit an overall negative Poisson's ratio. These structures open up various potential opportunities in impact resistance, high energy absorption, and flexible robotics, among others. Interestingly, auxetic structures could also be tailored to provide passive adaptation to changes in environmental stimuli -- an adaptation of this concept is explored in this paper in the context of designing a novel load-adaptive gripper system. Defining the design in terms of repeating parametric unit cells from which the finite structure can be synthesized presents an attractive computationally-efficient approach to designing auxetic structures. This approach also decouples the optimization cost and the size of the overall structure, and avoids the pitfalls of system-scale design e.g., via topology optimization. In this paper, a surrogate-based design optimization framework is presented to implement the concept of passively load-adaptive structures (of given outer shape) synthesized from auxetic unit cells. Open-source meshing, FEA and Bayesian Optimization tools are integrated to develop this computational framework, enhancing it adopt-ability and extensibility. Demonstration of the concept and the underlying framework is performed by designing a simplified robotic gripper, with the objective to maximize the ratio of towards-load (gripping) horizontal displacement to the load-affected vertical displacement. Optimal auxetic cell-based design generated thereof is found to be four times better in terms of exhibited contact reaction force when compared to a design obtained with topology optimization that is subjected to the same specified maximum loading.
Using Entropy Measures for Monitoring the Evolution of Activity Patterns
Venue: arXiv cs.LG 2022
Year: 2022

In this work, we apply information theory inspired methods to quantify changes in daily activity patterns. We use in-home movement monitoring data and show how they can help indicate the occurrence of healthcare-related events. Three different types of entropy measures namely Shannon's entropy, entropy rates for Markov chains, and entropy production rate have been utilised. The measures are evaluated on a large-scale in-home monitoring dataset that has been collected within our dementia care clinical study. The study uses Internet of Things (IoT) enabled solutions for continuous monitoring of in-home activity, sleep, and physiology to develop care and early intervention solutions to support people living with dementia (PLWD) in their own homes. Our main goal is to show the applicability of the entropy measures to time-series activity data analysis and to use the extracted measures as new engineered features that can be fed into inference and analysis models. The results of our experiments show that in most cases the combination of these measures can indicate the occurrence of healthcare-related events. We also find that different participants with the same events may have different measures based on one entropy measure. So using a combination of these measures in an inference model will be more effective than any of the single measures.
Modular Approach to Machine Reading Comprehension: Mixture of Task-Aware Experts
Venue: arXiv cs.CL 2022
Year: 2022

In this work we present a Mixture of Task-Aware Experts Network for Machine Reading Comprehension on a relatively small dataset. We particularly focus on the issue of common-sense learning, enforcing the common ground knowledge by specifically training different expert networks to capture different kinds of relationships between each passage, question and choice triplet. Moreover, we take inspi ration on the recent advancements of multitask and transfer learning by training each network a relevant focused task. By making the mixture-of-networks aware of a specific goal by enforcing a task and a relationship, we achieve state-of-the-art results and reduce over-fitting.