OpenRaiser/Intern-Atlas huggingface dataset:
Still Unsolved High-Performance Computing Challenges for up to Pre-Petascale Homogeneous Supercomputers
Venue: arXiv cs.DC 2022
Year: 2022

Pre-exascale High Performance Computers (HPC) can reach more than 400 Pflop/s real perfor-mance according the HPLinpack benchmarks. For nanoscience and quantum biology there are requirements for those program codes based on quantum physics algorithms which is difficult to ideally parallelize. Such parallel codes reach their limitations at terascale performance clus-ters. The standard Amdahl's law suggestions for code parallelization complicates focusing and planning for the next step the parallel code developments. In this report we focused on a three key applications domain which are highly parallelizable: HPC benchmarks, quantum compu-ting simulators and Car-Parinello molecular dynamics. According the results we summarize the Amdahl's Law & Parallel Speedup performance achievements with supercomputer with pre-petascale homogeneous HPC hardware. We conclude as an universal computer the pre-petascale supercomputing performance homogeneous hardware still has the basic challeng-es which must be addressed by the researchers or developer in order efficiently to use them.
NCVX: A General-Purpose Optimization Solver for Constrained Machine and Deep Learning
Venue: arXiv cs.LG 2022
Year: 2022

Imposing explicit constraints is relatively new but increasingly pressing in deep learning, stimulated by, e.g., trustworthy AI that performs robust optimization over complicated perturbation sets and scientific applications that need to respect physical laws and constraints. However, it can be hard to reliably solve constrained deep learning problems without optimization expertise. The existing deep learning frameworks do not admit constraints. General-purpose optimization packages can handle constraints but do not perform auto-differentiation and have trouble dealing with nonsmoothness. In this paper, we introduce a new software package called NCVX, whose initial release contains the solver PyGRANSO, a PyTorch-enabled general-purpose optimization package for constrained machine/deep learning problems, the first of its kind. NCVX inherits auto-differentiation, GPU acceleration, and tensor variables from PyTorch, and is built on freely available and widely used open-source frameworks. NCVX is available at https://ncvx.org, with detailed documentation and numerous examples from machine/deep learning and other fields.
Data Budgeting for Machine Learning
Venue: arXiv cs.LG 2022
Year: 2022

Data is the fuel powering AI and creates tremendous value for many domains. However, collecting datasets for AI is a time-consuming, expensive, and complicated endeavor. For practitioners, data investment remains to be a leap of faith in practice. In this work, we study the data budgeting problem and formulate it as two sub-problems: predicting (1) what is the saturating performance if given enough data, and (2) how many data points are needed to reach near the saturating performance. Different from traditional dataset-independent methods like PowerLaw, we proposed a learning method to solve data budgeting problems. To support and systematically evaluate the learning-based method for data budgeting, we curate a large collection of 383 tabular ML datasets, along with their data vs performance curves. Our empirical evaluation shows that it is possible to perform data budgeting given a small pilot study dataset with as few as $50$ data points.
Mutual Information Learned Classifiers: an Information-theoretic Viewpoint of Training Deep Learning Classification Systems
Venue: arXiv cs.LG 2022
Year: 2022

Deep learning systems have been reported to acheive state-of-the-art performances in many applications, and one of the keys for achieving this is the existence of well trained classifiers on benchmark datasets which can be used as backbone feature extractors in downstream tasks. As a main-stream loss function for training deep neural network (DNN) classifiers, the cross entropy loss can easily lead us to find models which demonstrate severe overfitting behavior when no other techniques are used for alleviating it such as data augmentation. In this paper, we prove that the existing cross entropy loss minimization for training DNN classifiers essentially learns the conditional entropy of the underlying data distribution of the dataset, i.e., the information or uncertainty remained in the labels after revealing the input. In this paper, we propose a mutual information learning framework where we train DNN classifiers via learning the mutual information between the label and input. Theoretically, we give the population error probability lower bound in terms of the mutual information. In addition, we derive the mutual information lower and upper bounds for a concrete binary classification data model in $\mbR^n$, and also the error probability lower bound in this scenario. Besides, we establish the sample complexity for accurately learning the mutual information from empirical data samples drawn from the underlying data distribution. Empirically, we conduct extensive experiments on several benchmark datasets to support our theory. Without whistles and bells, the proposed mutual information learned classifiers (MILCs) acheive far better generalization performances than the state-of-the-art classifiers with an improvement which can exceed more than 10\% in testing accuracy.
Extrinsic calibration for highly accurate trajectories reconstruction
Venue: arXiv cs.RO 2022
Year: 2022

In the context of robotics, accurate ground-truth positioning is the cornerstone for the development of mapping and localization algorithms. In outdoor environments and over long distances, total stations provide accurate and precise measurements, that are unaffected by the usual factors that deteriorate the accuracy of Global Navigation Satellite System (GNSS). While a single robotic total station can track the position of a target in three Degrees Of Freedom (DOF), three robotic total stations and three targets are necessary to yield the full six DOF pose reference. Since it is crucial to express the position of targets in a common coordinate frame, we present a novel extrinsic calibration method of multiple robotic total stations with field deployment in mind. The proposed method does not require the manual collection of ground control points during the system setup, nor does it require tedious synchronous measurement on each robotic total station. Based on extensive experimental work, we compare our approach to the classical extrinsic calibration methods used in geomatics for surveying and demonstrate that our approach brings substantial time savings during the deployment. Tested on more than 30 km of trajectories, our new method increases the precision of the extrinsic calibration by 25 % compared to the best state-of-the-art method, which is the one taking manually static ground control points.
Doing data science with platforms crumbs: an investigation into fakes views on YouTube
Venue: arXiv cs.SI 2022
Year: 2022

This paper contributes to the ongoing discussions on the scholarly access to social media data, discussing a case where this access is barred despite its value for understanding and countering online disinformation and despite the absence of privacy or copyright issues. Our study concerns YouTube's engagement metrics and, more specifically, the way in which the platform removes "fake views" (i.e., views considered as artificial or illegitimate by the platform). Working with one and a half year of data extracted from a thousand French YouTube channels, we show the massive extent of this phenomenon, which concerns the large majority of the channels and more than half the videos in our corpus. Our analysis indicates that most fakes news are corrected relatively late in the life of the videos and that the final view counts of the videos are not independent from the fake views they received. We discuss the potential harm that delays in corrections could produce in content diffusion: by inflating views counts, illegitimate views could make a video appear more popular than it is and unwarrantedly encourage its human and algorithmic recommendation. Unfortunately, we cannot offer a definitive assessment of this phenomenon, because YouTube provides no information on fake views in its API or interface. This paper is, therefore, also a call for greater transparency by YouTube and other online platforms about information that can have crucial implications for the quality of online public debate.
M-LIO: Multi-lidar, multi-IMU odometry with sensor dropout tolerance
Venue: arXiv cs.RO 2022
Year: 2022

We present a robust system for state estimation that fuses measurements from multiple lidars and inertial sensors with GNSS data. To initiate the method, we use the prior GNSS pose information. We then perform incremental motion in real-time, which produces robust motion estimates in a global frame by fusing lidar and IMU signals with GNSS translation components using a factor graph framework. We also propose methods to account for signal loss with a novel synchronization and fusion mechanism. To validate our approach extensive tests were carried out on data collected using Scania test vehicles (5 sequences for a total of ~ 7 Km). From our evaluations, we show an average improvement of 61% in relative translation and 42% rotational error compared to a state-of-the-art estimator fusing a single lidar/inertial sensor pair.
A Reproducible and Realistic Evaluation of Partial Domain Adaptation Methods
Venue: arXiv cs.CV 2022
Year: 2022

Unsupervised Domain Adaptation (UDA) aims at classifying unlabeled target images leveraging source labeled ones. In this work, we consider the Partial Domain Adaptation (PDA) variant, where we have extra source classes not present in the target domain. Most successful algorithms use model selection strategies that rely on target labels to find the best hyper-parameters and/or models along training. However, these strategies violate the main assumption in PDA: only unlabeled target domain samples are available. Moreover, there are also inconsistencies in the experimental settings - architecture, hyper-parameter tuning, number of runs - yielding unfair comparisons. The main goal of this work is to provide a realistic evaluation of PDA methods with the different model selection strategies under a consistent evaluation protocol. We evaluate 7 representative PDA algorithms on 2 different real-world datasets using 7 different model selection strategies. Our two main findings are: (i) without target labels for model selection, the accuracy of the methods decreases up to 30 percentage points; (ii) only one method and model selection pair performs well on both datasets. Experiments were performed with our PyTorch framework, BenchmarkPDA, which we open source.
Immunological Approaches to Load Balancing in MIMD Systems
Venue: arXiv cs.DC 2022
Year: 2022

Effective utilization of Multiple-Instruction-Multiple-Data (MIMD) parallel computers requires the application of good load balancing techniques. In this paper we show that heuristics derived from observation of complex natural systems, such as the mammalian immune system, can lead to effective load balancing strategies. In particular, the immune system processes of regulation, suppression, tolerance, and memory are seen to be powerful load balancing mechanisms. We provide a detailed example of our approach applied to parallelization of an image processing task, that of extracting the circuit design from the images of the layers of a CMOS integrated circuit. The results of this experiment show that good speedup characteristics can be obtained when using immune system derived load balancing strategies.
CaiRL: A High-Performance Reinforcement Learning Environment Toolkit
Venue: arXiv cs.LG 2022
Year: 2022

This paper addresses the dire need for a platform that efficiently provides a framework for running reinforcement learning (RL) experiments. We propose the CaiRL Environment Toolkit as an efficient, compatible, and more sustainable alternative for training learning agents and propose methods to develop more efficient environment simulations. There is an increasing focus on developing sustainable artificial intelligence. However, little effort has been made to improve the efficiency of running environment simulations. The most popular development toolkit for reinforcement learning, OpenAI Gym, is built using Python, a powerful but slow programming language. We propose a toolkit written in C++ with the same flexibility level but works orders of magnitude faster to make up for Python's inefficiency. This would drastically cut climate emissions. CaiRL also presents the first reinforcement learning toolkit with a built-in JVM and Flash support for running legacy flash games for reinforcement learning research. We demonstrate the effectiveness of CaiRL in the classic control benchmark, comparing the execution speed to OpenAI Gym. Furthermore, we illustrate that CaiRL can act as a drop-in replacement for OpenAI Gym to leverage significantly faster training speeds because of the reduced environment computation time.
Relative Entropy (RE) Based LTI System Modeling Equipped with time delay Estimation and Online Modeling
Venue: arXiv cs.IT 2022
Year: 2022

This paper proposes an impulse response modeling in presence of input and noisy output of a linear time-invariant (LTI) system. The approach utilizes Relative Entropy (RE) to choose the optimum impulse response estimate, optimum time delay and optimum impulse response length. The desired RE is the Kulback-Lielber divergence of the estimated distribution from its unknown true distribution. A unique probabilistic validation approach estimates the desired relative entropy and minimizes this criterion to provide the impulse response estimate. Classical methods have approached this system modeling problem from two separate angles for the time delay estimation and for the order selection. Time delay methods focus on time delay estimate minimizing various proposed criteria, while the existing order selection approaches choose the optimum impulse response length based on their proposed criteria. The strength of the proposed RE based method is in using the RE based criterion to estimate both the time delay and impulse response length simultaneously. In addition, estimation of the noise variance, when the Signal to Noise Ratio (SNR) is unknown is also concurrent and is based on optimizing the same RE based criterion. The RE based approach is also extended for online impulse response estimations. The online method reduces the model estimation computational complexity upon the arrival of a new sample. The introduced efficient stopping criteria for this online approaches is extremely valuable in practical applications. Simulation results illustrate precision and efficiency of the proposed method compared to the conventional time delay or order selection approaches.
OpBoost: A Vertical Federated Tree Boosting Framework Based on Order-Preserving Desensitization
Venue: arXiv cs.LG 2022
Year: 2022

Vertical Federated Learning (FL) is a new paradigm that enables users with non-overlapping attributes of the same data samples to jointly train a model without directly sharing the raw data. Nevertheless, recent works show that it's still not sufficient to prevent privacy leakage from the training process or the trained model. This paper focuses on studying the privacy-preserving tree boosting algorithms under the vertical FL. The existing solutions based on cryptography involve heavy computation and communication overhead and are vulnerable to inference attacks. Although the solution based on Local Differential Privacy (LDP) addresses the above problems, it leads to the low accuracy of the trained model. This paper explores to improve the accuracy of the widely deployed tree boosting algorithms satisfying differential privacy under vertical FL. Specifically, we introduce a framework called OpBoost. Three order-preserving desensitization algorithms satisfying a variant of LDP called distance-based LDP (dLDP) are designed to desensitize the training data. In particular, we optimize the dLDP definition and study efficient sampling distributions to further improve the accuracy and efficiency of the proposed algorithms. The proposed algorithms provide a trade-off between the privacy of pairs with large distance and the utility of desensitized values. Comprehensive evaluations show that OpBoost has a better performance on prediction accuracy of trained models compared with existing LDP approaches on reasonable settings. Our code is open source.
Wi-Closure: Reliable and Efficient Search of Inter-robot Loop Closures Using Wireless Sensing
Venue: arXiv cs.RO 2022
Year: 2022

In this paper we propose a novel algorithm, Wi-Closure, to improve computational efficiency and robustness of loop closure detection in multi-robot SLAM. Our approach decreases the computational overhead of classical approaches by pruning the search space of potential loop closures, prior to evaluation by a typical multi-robot SLAM pipeline. Wi-Closure achieves this by identifying candidates that are spatially close to each other by using sensing over the wireless communication signal between robots, even when they are operating in non-line-of-sight or in remote areas of the environment from one another. We demonstrate the validity of our approach in simulation and hardware experiments. Our results show that using Wi-closure greatly reduces computation time, by 54% in simulation and by 77% in hardware compared, with a multi-robot SLAM baseline. Importantly, this is achieved without sacrificing accuracy. Using Wi-Closure reduces absolute trajectory estimation error by 99% in simulation and 89.2% in hardware experiments. This improvement is due in part to Wi-Closure's ability to avoid catastrophic optimization failure that typically occurs with classical approaches in challenging repetitive environments.
Doubly-Irregular Repeat-Accumulate Codes over Integer Rings for Multi-user Communications
Venue: arXiv cs.IT 2022
Year: 2022

Structured codes based on lattices were shown to provide enlarged capacity for multi-user communication networks. In this paper, we study capacity-approaching irregular repeat accumulate (IRA) codes over integer rings $\mathbb{Z}_{2^{m}}$ for $2^m$-PAM signaling, $m=1,2,\cdots$. Such codes feature the property that the integer sum of $K$ codewords belongs to the extended codebook (or lattice) w.r.t. the base code. With it, \emph{% structured binning} can be utilized and the gains promised in lattice based network information theory can be materialized in practice. In designing IRA ring codes, we first analyze the effect of zero-divisors of integer ring on the iterative belief-propagation (BP) decoding, and show the invalidity of symmetric Gaussian approximation. Then we propose a doubly IRA (D-IRA) ring code structure, consisting of \emph{irregular multiplier distribution} and \emph{irregular node-degree distribution}, that can restore the symmetry and optimize the BP decoding threshold. For point-to-point AWGN channel with $% 2^m $-PAM inputs, D-IRA ring codes perform as low as 0.29 dB to the capacity limits, outperforming existing bit-interleaved coded-modulation (BICM) and IRA modulation codes over GF($2^m$). We then proceed to design D-IRA ring codes for two important multi-user communication setups, namely compute-forward (CF) and dirty paper coding (DPC), with $2^m$-PAM signaling. With it, a physical-layer network coding scheme yields a gap to the CF limit by 0.24 dB, and a simple linear DPC scheme exhibits a gap to the capacity by 0.91 dB.
Pay Self-Attention to Audio-Visual Navigation
Venue: arXiv cs.SD 2022
Year: 2022

Audio-visual embodied navigation, as a hot research topic, aims training a robot to reach an audio target using egocentric visual (from the sensors mounted on the robot) and audio (emitted from the target) input. The audio-visual information fusion strategy is naturally important to the navigation performance, but the state-of-the-art methods still simply concatenate the visual and audio features, potentially ignoring the direct impact of context. Moreover, the existing approaches requires either phase-wise training or additional aid (e.g. topology graph and sound semantics). Up till this date, the work that deals with the more challenging setup with moving target(s) is still rare. As a result, we propose an end-to-end framework FSAAVN (feature self-attention audio-visual navigation) to learn chasing after a moving audio target using a context-aware audio-visual fusion strategy implemented as a self-attention module. Our thorough experiments validate the superior performance (both quantitatively and qualitatively) of FSAAVN in comparison with the state-of-the-arts, and also provide unique insights about the choice of visual modalities, visual/audio encoder backbones and fusion patterns.
UltraBots: Large-Area Mid-Air Haptics for VR with Robotically Actuated Ultrasound Transducers
Venue: arXiv cs.HC 2022
Year: 2022

We introduce UltraBots, a system that combines ultrasound haptic feedback and robotic actuation for large-area mid-air haptics for VR. Ultrasound haptics can provide precise mid-air haptic feedback and versatile shape rendering, but the interaction area is often limited by the small size of the ultrasound devices, restricting the possible interactions for VR. To address this problem, this paper introduces a novel approach that combines robotic actuation with ultrasound haptics. More specifically, we will attach ultrasound transducer arrays to tabletop mobile robots or robotic arms for scalable, extendable, and translatable interaction areas. We plan to use Sony Toio robots for 2D translation and/or commercially available robotic arms for 3D translation. Using robotic actuation and hand tracking measured by a VR HMD (e.g., Oculus Quest), our system can keep the ultrasound transducers underneath the user's hands to provide on-demand haptics. We demonstrate applications with workspace environments, medical training, education and entertainment.
Learning an Invertible Output Mapping Can Mitigate Simplicity Bias in Neural Networks
Venue: arXiv cs.LG 2022
Year: 2022

Deep Neural Networks are known to be brittle to even minor distribution shifts compared to the training distribution. While one line of work has demonstrated that Simplicity Bias (SB) of DNNs - bias towards learning only the simplest features - is a key reason for this brittleness, another recent line of work has surprisingly found that diverse/ complex features are indeed learned by the backbone, and their brittleness is due to the linear classification head relying primarily on the simplest features. To bridge the gap between these two lines of work, we first hypothesize and verify that while SB may not altogether preclude learning complex features, it amplifies simpler features over complex ones. Namely, simple features are replicated several times in the learned representations while complex features might not be replicated. This phenomenon, we term Feature Replication Hypothesis, coupled with the Implicit Bias of SGD to converge to maximum margin solutions in the feature space, leads the models to rely mostly on the simple features for classification. To mitigate this bias, we propose Feature Reconstruction Regularizer (FRR) to ensure that the learned features can be reconstructed back from the logits. The use of {\em FRR} in linear layer training (FRR-L) encourages the use of more diverse features for classification. We further propose to finetune the full network by freezing the weights of the linear layer trained using FRR-L, to refine the learned features, making them more suitable for classification. Using this simple solution, we demonstrate up to 15% gains in OOD accuracy on the recently introduced semi-synthetic datasets with extreme distribution shifts. Moreover, we demonstrate noteworthy gains over existing SOTA methods on the standard OOD benchmark DomainBed as well.