OpenRaiser/Intern-Atlas huggingface dataset:
Sat-NeRF: Learning Multi-View Satellite Photogrammetry With Transient Objects and Shadow Modeling Using RPC Cameras
Venue: arXiv cs.CV 2022
Year: 2022

We introduce the Satellite Neural Radiance Field (Sat-NeRF), a new end-to-end model for learning multi-view satellite photogrammetry in the wild. Sat-NeRF combines some of the latest trends in neural rendering with native satellite camera models, represented by rational polynomial coefficient (RPC) functions. The proposed method renders new views and infers surface models of similar quality to those obtained with traditional state-of-the-art stereo pipelines. Multi-date images exhibit significant changes in appearance, mainly due to varying shadows and transient objects (cars, vegetation). Robustness to these challenges is achieved by a shadow-aware irradiance model and uncertainty weighting to deal with transient phenomena that cannot be explained by the position of the sun. We evaluate Sat-NeRF using WorldView-3 images from different locations and stress the advantages of applying a bundle adjustment to the satellite camera models prior to training. This boosts the network performance and can optionally be used to extract additional cues for depth supervision.
SMARTmBOT: A ROS2-based Low-cost and Open-source Mobile Robot Platform
Venue: arXiv cs.RO 2022
Year: 2022

This paper introduces SMARTmBOT, an open-source mobile robot platform based on Robot Operating System 2 (ROS2). The characteristics of the SMARTmBOT, including low-cost, modular-typed, customizable and expandable design, make it an easily achievable and effective robot platform to support broad robotics research and education involving either single-robot or multi-robot systems. The total cost per robot is approximately $210, and most hardware components can be fabricated by a generic 3D printer, hence allowing users to build the robots or replace any broken parts conveniently. The SMARTmBot is also equipped with a rich range of sensors, making it competent for general task scenarios, such as point-to-point navigation and obstacle avoidance. We validated the mobility and function of SMARTmBOT through various robot navigation experiments and applications with tasks including go-to-goal, pure-pursuit, line following, and swarming. All source code necessary for reading sensors, streaming from an embedded camera, and controlling the robot including robot navigation controllers is available through an online repository that can be found at https://github.com/SMARTlab-Purdue/SMARTmBOT.
The Best of Both Worlds: Combining Learned Embeddings with Engineered Features for Accurate Prediction of Correct Patches
Venue: arXiv cs.SE 2022
Year: 2022

A large body of the literature on automated program repair develops approaches where patches are automatically generated to be validated against an oracle (e.g., a test suite). Because such an oracle can be imperfect, the generated patches, although validated by the oracle, may actually be incorrect. Our empirical work investigates different representation learning approaches for code changes to derive embeddings that are amenable to similarity computations of patch correctness identification, and assess the possibility of accurate classification of correct patch by combining learned embeddings with engineered features. Experimental results demonstrate the potential of learned embeddings to empower Leopard (a patch correctness predicting framework implemented in this work) with learning algorithms in reasoning about patch correctness: a machine learning predictor with BERT transformer-based learned embeddings associated with XGBoost achieves an AUC value of about 0.803 in the prediction of patch correctness on a new dataset of 2,147 labeled patches that we collected for the experiments. Our investigations show that deep learned embeddings can lead to complementary/better performance when comparing against the state-of-the-art, PATCH-SIM, which relies on dynamic information. By combining deep learned embeddings and engineered features, Panther (the upgraded version of Leopard implemented in this work) outperforms Leopard with higher scores in terms of AUC, +Recall and -Recall, and can accurately identify more (in)correct patches that cannot be predicted by the classifiers only with learned embeddings or engineered features. Finally, we use an explainable ML technique, SHAP, to empirically interpret how the learned embeddings and engineered features are contributed to the patch correctness prediction.
Any Way You Look At It: Semantic Crossview Localization and Mapping with LiDAR
Venue: arXiv cs.RO 2022
Year: 2022

Currently, GPS is by far the most popular global localization method. However, it is not always reliable or accurate in all environments. SLAM methods enable local state estimation but provide no means of registering the local map to a global one, which can be important for inter-robot collaboration or human interaction. In this work, we present a real-time method for utilizing semantics to globally localize a robot using only egocentric 3D semantically labelled LiDAR and IMU as well as top-down RGB images obtained from satellites or aerial robots. Additionally, as it runs, our method builds a globally registered, semantic map of the environment. We validate our method on KITTI as well as our own challenging datasets, and show better than 10 meter accuracy, a high degree of robustness, and the ability to estimate the scale of a top-down map on the fly if it is initially unknown.
$\ell_p$ Slack Norm Support Vector Data Description
Venue: arXiv cs.LG 2022
Year: 2022

The support vector data description (SVDD) approach serves as a de facto standard for one-class classification where the learning task entails inferring the smallest hyper-sphere to enclose target objects while linearly penalising any errors/slacks via an $\ell_1$-norm penalty term. In this study, we generalise this modelling formalism to a general $\ell_p$-norm ($p\geq1$) slack penalty function. By virtue of an $\ell_p$ slack norm, the proposed approach enables formulating a non-linear cost function with respect to slacks. From a dual problem perspective, the proposed method introduces a sparsity-inducing dual norm into the objective function, and thus, possesses a higher capacity to tune into the inherent sparsity of the problem for enhanced descriptive capability. A theoretical analysis based on Rademacher complexities characterises the generalisation performance of the proposed approach in terms of parameter $p$ while the experimental results on several datasets confirm the merits of the proposed method compared to other alternatives.
ABN: Agent-Aware Boundary Networks for Temporal Action Proposal Generation
Venue: arXiv cs.CV 2022
Year: 2022

Temporal action proposal generation (TAPG) aims to estimate temporal intervals of actions in untrimmed videos, which is a challenging yet plays an important role in many tasks of video analysis and understanding. Despite the great achievement in TAPG, most existing works ignore the human perception of interaction between agents and the surrounding environment by applying a deep learning model as a black-box to the untrimmed videos to extract video visual representation. Therefore, it is beneficial and potentially improve the performance of TAPG if we can capture these interactions between agents and the environment. In this paper, we propose a novel framework named Agent-Aware Boundary Network (ABN), which consists of two sub-networks (i) an Agent-Aware Representation Network to obtain both agent-agent and agents-environment relationships in the video representation, and (ii) a Boundary Generation Network to estimate the confidence score of temporal intervals. In the Agent-Aware Representation Network, the interactions between agents are expressed through local pathway, which operates at a local level to focus on the motions of agents whereas the overall perception of the surroundings are expressed through global pathway, which operates at a global level to perceive the effects of agents-environment. Comprehensive evaluations on 20-action THUMOS-14 and 200-action ActivityNet-1.3 datasets with different backbone networks (i.e C3D, SlowFast and Two-Stream) show that our proposed ABN robustly outperforms state-of-the-art methods regardless of the employed backbone network on TAPG. We further examine the proposal quality by leveraging proposals generated by our method onto temporal action detection (TAD) frameworks and evaluate their detection performances. The source code can be found in this URL https://github.com/vhvkhoa/TAPG-AgentEnvNetwork.git.
CachePerf: A Unified Cache Miss Classifier via Hybrid Hardware Sampling
Venue: arXiv cs.PF 2022
Year: 2022

The cache plays a key role in determining the performance of applications, no matter for sequential or concurrent programs on homogeneous and heterogeneous architecture. Fixing cache misses requires to understand the origin and the type of cache misses. However, this remains to be an unresolved issue even after decades of research. This paper proposes a unified profiling tool--CachePerf--that could correctly identify different types of cache misses, differentiate allocator-induced issues from those of applications, and exclude minor issues without much performance impact. The core idea behind CachePerf is a hybrid sampling scheme: it employs the PMU-based coarse-grained sampling to select very few susceptible instructions (with frequent cache misses) and then employs the breakpoint-based fine-grained sampling to collect the memory access pattern of these instructions. Based on our evaluation, CachePerf only imposes 14% performance overhead and 19% memory overhead (for applications with large footprints), while identifying the types of cache misses correctly. CachePerf detected 9 previous-unknown bugs. Fixing the reported bugs achieves from 3% to 3788% performance speedup. CachePerf will be an indispensable complementary to existing profilers due to its effectiveness and low overhead.
Adaptive n-ary Activation Functions for Probabilistic Boolean Logic
Venue: arXiv cs.LG 2022
Year: 2022

Balancing model complexity against the information contained in observed data is the central challenge to learning. In order for complexity-efficient models to exist and be discoverable in high dimensions, we require a computational framework that relates a credible notion of complexity to simple parameter representations. Further, this framework must allow excess complexity to be gradually removed via gradient-based optimization. Our n-ary, or n-argument, activation functions fill this gap by approximating belief functions (probabilistic Boolean logic) using logit representations of probability. Just as Boolean logic determines the truth of a consequent claim from relationships among a set of antecedent propositions, probabilistic formulations generalize predictions when antecedents, truth tables, and consequents all retain uncertainty. Our activation functions demonstrate the ability to learn arbitrary logic, such as the binary exclusive disjunction (p xor q) and ternary conditioned disjunction ( c ? p : q ), in a single layer using an activation function of matching or greater arity. Further, we represent belief tables using a basis that directly associates the number of nonzero parameters to the effective arity of the belief function, thus capturing a concrete relationship between logical complexity and efficient parameter representations. This opens optimization approaches to reduce logical complexity by inducing parameter sparsity.
Measuring Consumer Perceived Warm-Glow for Technology Adoption Modeling
Venue: arXiv cs.CY 2022
Year: 2022

In this paper, we adapt and validate two constructs-perceived extrinsic warm-glow (PEWG) and perceived intrinsic warm-glow (PIWG)-to measure the two dimensions of consumer perceived warm-glow (i.e., extrinsic and intrinsic) for use with the practice of technology adoption modeling. Taking an experimental approach, participants were exposed to one of four vignettes designed to simulate either the absence or the presence of warm-glow (specifically, extrinsic warm-glow, intrinsic warm-glow, and concurrently extrinsic and intrinsic warm-glow). The results revealed that both constructs measured their respective forms of warm-glow with two caveats. Firstly, singularly trying to evoke extrinsic warm-glow led to only a slight increase in consumer perception of extrinsic warm-glow. We attributed this finding to individuals not being attracted to technology products that overtly target and seek to satisfy their vanity, instead preferring technology that does so in a more subtle way. The second is that singularly trying to evoke intrinsic warm-glow also resulted in the manifestation of extrinsic warm-glow. Thus, warm-glow appears as a blend of extrinsic and intrinsic dimensions. This finding serves to reinforce what has already been reported in existing literature regarding warm-glow and the idea of impure altruism.
Sub-Terahertz Wireless Coverage Analysis at 142 GHz in Urban Microcell
Venue: arXiv cs.IT 2022
Year: 2022

Small-cell cellular base stations are going to be used for mmWave and sub-THz communication systems to provide multi-Gbps data rates and reliable coverage to mobile users. This paper analyzes the base station coverage of sub-THz communication systems and the system performance in terms of spectral efficiency through Monte Carlo simulations for both single-cell and multi-cell cases. The simulations are based on realistic channel models derived from outdoor field measurements at 142 GHz in urban microcell (UMi) environments conducted in downtown Brooklyn, New York. The single-cell base station can provide a downlink coverage area with a radius of 200 m and the 7-cell system can provide a downlink coverage area with a radius of 400 m at 142 GHz. Using a 1 GHz downlink bandwidth and 100 MHz uplink bandwidth, the 7-cell system can provide about 4.5 Gbps downlink average data rate and 410 Mbps uplink average data rate at 142 GHz.
Insights for post-pandemic pedagogy across one CS department
Venue: arXiv cs.CY 2022
Year: 2022

Adaptive remote instruction has led to important lessons for the future, including rediscovery of known pedagogical principles in new contexts and new insights for supporting remote learning. Studying one computer science department that serves residential and remote undergraduate and graduate students, we conducted interviews with stakeholders in the department (n=26) and ran a department-wide student survey (n=102) during the four academic quarters from spring 2020 to spring 2021. Our case study outlines what the instructors did, summarizes what instructors and students say about courses during this period, and provides recommendations for CS departments with similar scope going forward. Specific insights address: (1) how instructional components are best structured for students; (2) how students are assessed for their learning; and (3) how students are supported in student-initiated components of learning. The institution is a large U.S. research university that has a history of online programs including online enrollment in regular on-campus courses and large-scale open enrollment courses. Our recommendations to instructors across the scope of this department may also be applicable to other institutions that provide technology-supported in-person instruction, remote enrollment, and hybrid courses combining both modalities.
Robust Table Detection and Structure Recognition from Heterogeneous Document Images
Venue: arXiv cs.CV 2022
Year: 2022

We introduce a new table detection and structure recognition approach named RobusTabNet to detect the boundaries of tables and reconstruct the cellular structure of each table from heterogeneous document images. For table detection, we propose to use CornerNet as a new region proposal network to generate higher quality table proposals for Faster R-CNN, which has significantly improved the localization accuracy of Faster R-CNN for table detection. Consequently, our table detection approach achieves state-of-the-art performance on three public table detection benchmarks, namely cTDaR TrackA, PubLayNet and IIIT-AR-13K, by only using a lightweight ResNet-18 backbone network. Furthermore, we propose a new split-and-merge based table structure recognition approach, in which a novel spatial CNN based separation line prediction module is proposed to split each detected table into a grid of cells, and a Grid CNN based cell merging module is applied to recover the spanning cells. As the spatial CNN module can effectively propagate contextual information across the whole table image, our table structure recognizer can robustly recognize tables with large blank spaces and geometrically distorted (even curved) tables. Thanks to these two techniques, our table structure recognition approach achieves state-of-the-art performance on three public benchmarks, including SciTSR, PubTabNet and cTDaR TrackB2-Modern. Moreover, we have further demonstrated the advantages of our approach in recognizing tables with complex structures, large blank spaces, as well as geometrically distorted or even curved shapes on a more challenging in-house dataset.
STPLS3D: A Large-Scale Synthetic and Real Aerial Photogrammetry 3D Point Cloud Dataset
Venue: arXiv cs.CV 2022
Year: 2022

Although various 3D datasets with different functions and scales have been proposed recently, it remains challenging for individuals to complete the whole pipeline of large-scale data collection, sanitization, and annotation. Moreover, the created datasets usually suffer from extremely imbalanced class distribution or partial low-quality data samples. Motivated by this, we explore the procedurally synthetic 3D data generation paradigm to equip individuals with the full capability of creating large-scale annotated photogrammetry point clouds. Specifically, we introduce a synthetic aerial photogrammetry point clouds generation pipeline that takes full advantage of open geospatial data sources and off-the-shelf commercial packages. Unlike generating synthetic data in virtual games, where the simulated data usually have limited gaming environments created by artists, the proposed pipeline simulates the reconstruction process of the real environment by following the same UAV flight pattern on different synthetic terrain shapes and building densities, which ensure similar quality, noise pattern, and diversity with real data. In addition, the precise semantic and instance annotations can be generated fully automatically, avoiding the expensive and time-consuming manual annotation. Based on the proposed pipeline, we present a richly-annotated synthetic 3D aerial photogrammetry point cloud dataset, termed STPLS3D, with more than 16 $km^2$ of landscapes and up to 18 fine-grained semantic categories. For verification purposes, we also provide a parallel dataset collected from four areas in the real environment. Extensive experiments conducted on our datasets demonstrate the effectiveness and quality of the proposed synthetic dataset.
UNIMO-2: End-to-End Unified Vision-Language Grounded Learning
Venue: arXiv cs.CV 2022
Year: 2022

Vision-Language Pre-training (VLP) has achieved impressive performance on various cross-modal downstream tasks. However, most existing methods can only learn from aligned image-caption data and rely heavily on expensive regional features, which greatly limits their scalability and performance. In this paper, we propose an end-to-end unified-modal pre-training framework, namely UNIMO-2, for joint learning on both aligned image-caption data and unaligned image-only and text-only corpus. We build a unified Transformer model to jointly learn visual representations, textual representations and semantic alignment between images and texts. In particular, we propose to conduct grounded learning on both images and texts via a sharing grounded space, which helps bridge unaligned images and texts, and align the visual and textual semantic spaces on different types of corpora. The experiments show that our grounded learning method can improve textual and visual semantic alignment for improving performance on various cross-modal tasks. Moreover, benefiting from effective joint modeling of different types of corpora, our model also achieves impressive performance on single-modal visual and textual tasks. Our code and models are public at the UNIMO project page https://unimo-ptm.github.io/.
Time and the Value of Data
Venue: arXiv cs.LG 2022
Year: 2022

Managers often believe that collecting more data will continually improve the accuracy of their machine learning models. However, we argue in this paper that when data lose relevance over time, it may be optimal to collect a limited amount of recent data instead of keeping around an infinite supply of older (less relevant) data. In addition, we argue that increasing the stock of data by including older datasets may, in fact, damage the model's accuracy. Expectedly, the model's accuracy improves by increasing the flow of data (defined as data collection rate); however, it requires other tradeoffs in terms of refreshing or retraining machine learning models more frequently. Using these results, we investigate how the business value created by machine learning models scales with data and when the stock of data establishes a sustainable competitive advantage. We argue that data's time-dependency weakens the barrier to entry that the stock of data creates. As a result, a competing firm equipped with a limited (yet sufficient) amount of recent data can develop more accurate models. This result, coupled with the fact that older datasets may deteriorate models' accuracy, suggests that created business value doesn't scale with the stock of available data unless the firm offloads less relevant data from its data repository. Consequently, a firm's growth policy should incorporate a balance between the stock of historical data and the flow of new data. We complement our theoretical results with an experiment. In the experiment, we empirically measure the loss in the accuracy of a next word prediction model trained on datasets from various time periods. Our empirical measurements confirm the economic significance of the value decline over time. For example, 100MB of text data, after seven years, becomes as valuable as 50MB of current data for the next word prediction task.
A Survey on Brain-Computer Interface and Related Applications
Venue: arXiv cs.HC 2022
Year: 2022

BCI systems are able to communicate directly between the brain and computer using neural activity measurements without the involvement of muscle movements. For BCI systems to be widely used by people with severe disabilities, long-term studies of their real-world use are needed, along with effective and feasible dissemination models. In addition, the robustness of the BCI systems' performance should be improved so they reach the same level of robustness as natural muscle-based health monitoring. In this chapter, we review the recent BCI related studies, followed by the most relevant applications of BCI systems. We also present the key issues and challenges which exist in regard to the BCI systems and also provide future directions.
Recurrent Neural Networks for Forecasting Time Series with Multiple Seasonality: A Comparative Study
Venue: arXiv cs.LG 2022
Year: 2022

This paper compares recurrent neural networks (RNNs) with different types of gated cells for forecasting time series with multiple seasonality. The cells we compare include classical long short term memory (LSTM), gated recurrent unit (GRU), modified LSTM with dilation, and two new cells we proposed recently, which are equipped with dilation and attention mechanisms. To model the temporal dependencies of different scales, our RNN architecture has multiple dilated recurrent layers stacked with hierarchical dilations. The proposed RNN produces both point forecasts and predictive intervals (PIs) for them. An empirical study concerning short-term electrical load forecasting for 35 European countries confirmed that the new gated cells with dilation and attention performed best.