OpenRaiser/Intern-Atlas huggingface dataset:
Semi-Supervised Learning for Sparsely-Labeled Sequential Data: Application to Healthcare Video Processing
Venue: arXiv cs.CV 2022
Year: 2022

Labeled data is a critical resource for training and evaluating machine learning models. However, many real-life datasets are only partially labeled. We propose a semi-supervised machine learning training strategy to improve event detection performance on sequential data, such as video recordings, when only sparse labels are available, such as event start times without their corresponding end times. Our method uses noisy guesses of the events' end times to train event detection models. Depending on how conservative these guesses are, mislabeled samples may be introduced into the training set. We further propose a mathematical model for explaining and estimating the evolution of the classification performance for increasingly noisier end time estimates. We show that neural networks can improve their detection performance by leveraging more training data with less conservative approximations despite the higher proportion of incorrect labels. We adapt sequential versions of CIFAR-10 and MNIST, and use the Berkeley MHAD and HMBD51 video datasets to empirically evaluate our method, and find that our risk-tolerant strategy outperforms conservative estimates by 3.5 points of mean average precision for CIFAR, 30 points for MNIST, 3 points for MHAD, and 14 points for HMBD51. Then, we leverage the proposed training strategy to tackle a real-life application: processing continuous video recordings of epilepsy patients, and show that our method outperforms baseline labeling methods by 17 points of average precision, and reaches a classification performance similar to that of fully supervised models. We share part of the code for this article.
Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules
Venue: arXiv cs.LG 2022
Year: 2022

Many important tasks in chemistry revolve around molecules during reactions. This requires predictions far from the equilibrium, while most recent work in machine learning for molecules has been focused on equilibrium or near-equilibrium states. In this paper we aim to extend this scope in three ways. First, we propose the DimeNet++ model, which is 8x faster and 10% more accurate than the original DimeNet on the QM9 benchmark of equilibrium molecules. Second, we validate DimeNet++ on highly reactive molecules by developing the challenging COLL dataset, which contains distorted configurations of small molecules during collisions. Finally, we investigate ensembling and mean-variance estimation for uncertainty quantification with the goal of accelerating the exploration of the vast space of non-equilibrium structures. Our DimeNet++ implementation as well as the COLL dataset are available online.
Verifying liquidity of recursive Bitcoin contracts
Venue: arXiv cs.CR 2022
Year: 2022

Smart contracts - computer protocols that regulate the exchange of crypto-assets in trustless environments - have become popular with the spread of blockchain technologies. A landmark security property of smart contracts is liquidity: in a non-liquid contract, it may happen that some assets remain frozen, i.e. not redeemable by anyone. The relevance of this issue is witnessed by recent liquidity attacks to Ethereum, which have frozen hundreds of USD millions. We address the problem of verifying liquidity on BitML, a DSL for smart contracts with a secure compiler to Bitcoin, featuring primitives for currency transfers, contract renegotiation and consensual recursion. Our main result is a verification technique for liquidity. We first transform the infinite-state semantics of BitML into a finite-state one, which focusses on the behaviour of a chosen set of contracts, abstracting from the moves of the context. With respect to the chosen contracts, this abstraction is sound, i.e. if the abstracted contract is liquid, then also the concrete one is such. We then verify liquidity by model-checking the finite-state abstraction. We implement a toolchain that automatically verifies liquidity of BitML contracts and compiles them to Bitcoin, and we assess it through a benchmark of representative contracts.
Feature Learning in Infinite-Width Neural Networks
Venue: arXiv cs.LG 2022
Year: 2022

As its width tends to infinity, a deep neural network's behavior under gradient descent can become simplified and predictable (e.g. given by the Neural Tangent Kernel (NTK)), if it is parametrized appropriately (e.g. the NTK parametrization). However, we show that the standard and NTK parametrizations of a neural network do not admit infinite-width limits that can learn features, which is crucial for pretraining and transfer learning such as with BERT. We propose simple modifications to the standard parametrization to allow for feature learning in the limit. Using the *Tensor Programs* technique, we derive explicit formulas for such limits. On Word2Vec and few-shot learning on Omniglot via MAML, two canonical tasks that rely crucially on feature learning, we compute these limits exactly. We find that they outperform both NTK baselines and finite-width networks, with the latter approaching the infinite-width feature learning performance as width increases. More generally, we classify a natural space of neural network parametrizations that generalizes standard, NTK, and Mean Field parametrizations. We show 1) any parametrization in this space either admits feature learning or has an infinite-width training dynamics given by kernel gradient descent, but not both; 2) any such infinite-width limit can be computed using the Tensor Programs technique. Code for our experiments can be found at github.com/edwardjhu/TP4.
Topologically Driven Methods for Construction Of Multi-Edge Type (Multigraph with nodes puncturing) Quasi-Cyclic Low-density Parity-check Codes for Wireless Channel, WDM Long-Haul and Archival Holographic Memory
Venue: arXiv cs.IT 2022
Year: 2022

In this Phd thesis discusses modern methods for constructing MET QC-LDPC codes with a given error correction ("waterfall, error-floor") and complexity (parallelism level according circulant size plus scheduler orthogonality of checks) profiles: 1. weight enumerators optimization, protograph construction using Density Evolution, MI (P/Exit-chart) and it approximation: Gaussian Approximation, Reciprocal-channel approximation and etc; 2. Covariance evolution and it approximation; 3. Lifting methods for QC codes construction:PEG, Guest-and-Test, Hill-Climbing with girth, EMD, ACE optimization; 4. Upper and lower bounds on code distance estimation and its parallel implementation using CPU/GPU; 5. Brouwer-Zimmerman and Number Geometry code distance estimation methods; 6. Importance Sampling for error-floor estimation; 7. Length and rate adaption methods for QC codes based on cyclic group decomposition; 8. Methods for interaction screening which allow to improve performance (decorrelate variables) under BP and it's approximation. We proposed several state-of-the-art methods: Simulated Annealing lifting for MET QC-LDPC codes construction; fast EMD and code distance estimation; floor scale modular lifting for lenght adaption; fast finite-length covariance evolution rate penalty from threshold for code construction and it hardware friendly compression for fast decoder's LLRs unbiasing due SNR's estimation error. We found topology reason's of efficient of such methods using topology thickening (homotopy of continuous and discrete curvature) under matched metric space which allow to generalize this idea to a class of nonlinear codes for Signal Processing and Machine Learning. Using the proposed algorithms several generations of WDM Long-Haul error-correction codes were built. It was applied for "5G eMBB" 3GPP TS38.212 and other applications like Flash storage, Compressed sensing measurement matrix.
Performance and Optimization of Reconfigurable Intelligent Surface Aided THz Communications
Venue: arXiv cs.IT 2022
Year: 2022

TeraHertz (THz) communications can satisfy the high data rate demand with massive bandwidth. However, severe path attenuation and hardware imperfection greatly alleviate its performance. Therefore, we utilize the reconfigurable intelligent surface (RIS) technology and investigate the RIS-aided THz communications. We first prove that the small-scale amplitude fading of THz signals can be accurately modeled by the fluctuating two-ray distribution based on two THz signal measurement experiments conducted in a variety of different scenarios. To optimize the phase-shifts at the RIS elements, we propose a novel swarm intelligence-based method that does not require full channel estimation. We then derive exact statistical characterizations of end-to-end signal-to-noise plus distortion ratio (SNDR) and signal-to-noise ratio (SNR). Moreover, we present asymptotic analysis to obtain more insights when the SNDR or the number of RIS's elements is high. Finally, we derive analytical expressions for the outage probability and ergodic capacity. The tight upper bounds of ergodic capacity for both ideal and nonideal radio frequency chains are obtained. It is interesting to find that increasing the number of RIS's elements can significantly improve the THz communications system performance. For example, the ergodic capacity can increase up to 25% when the number of elements increases from 40 to 80, which incurs only insignificant costs to the system.
MAUIL: Multi-level Attribute Embedding for Semi-supervised User Identity Linkage
Venue: arXiv cs.SI 2022
Year: 2022

User identity linkage (UIL) across social networks has recently attracted an increasing amount of attention due to its significant research challenges and practical value. Most of the existing methods use a single method to express different types of attribute features. However, the simplex pattern can neither cover the entire set of different attribute features nor capture the higher-level semantic features in the attribute text. This paper establishes a novel semisupervised model, namely the multilevel attribute embedding for semisupervised user identity linkage (MAUIL), to seek the common user identity across social networks. MAUIL includes two components: multilevel attribute embedding and regularized canonical correlation analysis (RCCA)-based linear projection. Specifically, the text attributes for each network are first divided into three types: character-level, word-level, and topic-level attributes. Second, unsupervised approaches are employed to extract the corresponding three types of text attribute features, and user relationships are embedded as a complimentary feature. All the resultant features are combined to form the final representation of each user. Finally, target social networks are projected into a common correlated space by RCCA with the help of a small number of prematched user pairs. We demonstrate the superiority of the proposed method over state-of-the-art methods through extensive experiments on two real-world datasets.
Temporal Representation Learning on Monocular Videos for 3D Human Pose Estimation
Venue: arXiv cs.CV 2022
Year: 2022

In this paper we propose an unsupervised feature extraction method to capture temporal information on monocular videos, where we detect and encode subject of interest in each frame and leverage contrastive self-supervised (CSS) learning to extract rich latent vectors. Instead of simply treating the latent features of nearby frames as positive pairs and those of temporally-distant ones as negative pairs as in other CSS approaches, we explicitly disentangle each latent vector into a time-variant component and a time-invariant one. We then show that applying contrastive loss only to the time-variant features and encouraging a gradual transition on them between nearby and away frames while also reconstructing the input, extract rich temporal features, well-suited for human pose estimation. Our approach reduces error by about 50% compared to the standard CSS strategies, outperforms other unsupervised single-view methods and matches the performance of multi-view techniques. When 2D pose is available, our approach can extract even richer latent features and improve the 3D pose estimation accuracy, outperforming other state-of-the-art weakly supervised methods.
Scalable Transfer Evolutionary Optimization: Coping with Big Task Instances
Venue: arXiv cs.NE 2022
Year: 2022

In today's digital world, we are faced with an explosion of data and models produced and manipulated by numerous large-scale cloud-based applications. Under such settings, existing transfer evolutionary optimization frameworks grapple with simultaneously satisfying two important quality attributes, namely (1) scalability against a growing number of source tasks and (2) online learning agility against sparsity of relevant sources to the target task of interest. Satisfying these attributes shall facilitate practical deployment of transfer optimization to scenarios with big task-instances, while curbing the threat of negative transfer. While applications of existing algorithms are limited to tens of source tasks, in this paper, we take a quantum leap forward in enabling more than two orders of magnitude scale-up in the number of tasks; i.e., we efficiently handle scenarios beyond 1000 source task-instances. We devise a novel transfer evolutionary optimization framework comprising two co-evolving species for joint evolutions in the space of source knowledge and in the search space of solutions to the target problem. In particular, co-evolution enables the learned knowledge to be orchestrated on the fly, expediting convergence in the target optimization task. We have conducted an extensive series of experiments across a set of practically motivated discrete and continuous optimization examples comprising a large number of source task-instances, of which only a small fraction indicate source-target relatedness. The experimental results show that not only does our proposed framework scale efficiently with a growing number of source tasks but is also effective in capturing relevant knowledge against sparsity of related sources, fulfilling the two salient features of scalability and online learning agility.
DeepSym: Deep Symbol Generation and Rule Learning from Unsupervised Continuous Robot Interaction for Planning
Venue: arXiv cs.RO 2022
Year: 2022

We propose a novel general method that finds action-grounded, discrete object and effect categories and builds probabilistic rules over them for non-trivial action planning. Our robot interacts with objects using an initial action repertoire that is assumed to be acquired earlier and observes the effects it can create in the environment. To form action-grounded object, effect, and relational categories, we employ a binary bottleneck layer in a predictive, deep encoder-decoder network that takes the image of the scene and the action applied as input, and generates the resulting effects in the scene in pixel coordinates. After learning, the binary latent vector represents action-driven object categories based on the interaction experience of the robot. To distill the knowledge represented by the neural network into rules useful for symbolic reasoning, a decision tree is trained to reproduce its decoder function. Probabilistic rules are extracted from the decision paths of the tree and are represented in the Probabilistic Planning Domain Definition Language (PPDDL), allowing off-the-shelf planners to operate on the knowledge extracted from the sensorimotor experience of the robot. The deployment of the proposed approach for a simulated robotic manipulator enabled the discovery of discrete representations of object properties such as `rollable' and `insertable'. In turn, the use of these representations as symbols allowed the generation of effective plans for achieving goals, such as building towers of the desired height, demonstrating the effectiveness of the approach for multi-step object manipulation. Finally, we demonstrate that the system is not only restricted to the robotics domain by assessing its applicability to the MNIST 8-puzzle domain in which learned symbols allow for the generation of plans that move the empty tile into any given position.
Spatio-Temporal Analysis of On Demand Transit: A Case Study of Belleville, Canada
Venue: arXiv cs.CY 2022
Year: 2022

The rapid increase in the cyber-physical nature of transportation, availability of GPS data, mobile applications, and effective communication technologies have led to the emergence of On-Demand Transit (ODT) systems. In September 2018, the City of Belleville in Canada started an on-demand public transit pilot project, where the late-night fixed-route (RT 11) was substituted with the ODT providing a real-time ride-hailing service. We present an in-depth analysis of the spatio-temporal demand and supply, level of service, and origin and destination patterns of Belleville ODT users, based on the data collected from September 2018 till May 2019. The independent and combined effects of the demographic characteristics (population density, working-age, and median income) on the ODT trip production and attraction levels were studied using GIS and the K-means machine learning clustering algorithm. The results indicate that ODT trips demand is highest for 11:00 pm-11:45 pm during the weekdays and 8:00 pm-8:30 pm during the weekends. We expect this to be the result of users returning home from work or shopping. Results showed that 39% of the trips were found to have a waiting time of smaller than 15 minutes, while 28% of trips had a waiting time of 15-30 minutes. The dissemination areas with higher population density, lower median income, or higher working-age percentages tend to have higher ODT trip attraction levels, except for the dissemination areas that have highly attractive places like commercial areas.
A three layer neural network can represent any multivariate function
Venue: arXiv cs.LG 2022
Year: 2022

In 1987, Hecht-Nielsen showed that any continuous multivariate function can be implemented by a certain type three-layer neural network. This result was very much discussed in neural network literature. In this paper we prove that not only continuous functions but also all discontinuous functions can be implemented by such neural networks.
How to Deploy Intelligent Reflecting Surfaces in Wireless Network: BS-side, User-side, or Both Sides?
Venue: arXiv cs.IT 2022
Year: 2022

The performance of wireless communication systems is fundamentally constrained by the random and uncontrollable wireless channel. By leveraging the recent advance in digitally-controlled metasurface, intelligent reflecting surface (IRS) has emerged as a promising solution to enhance the wireless network performance by smartly reconfiguring the radio propagation environment. Despite the substantial research on IRS-aided communications, this article addresses the important issue of how to deploy IRSs in a wireless network to achieve its optimum performance. We first compare the two conventional strategies of deploying IRS at the side of base station or distributed users in terms of various communication performance metrics, and then propose a new hybrid IRS deployment strategy by combining their complementary advantages. Moreover, the main challenges in optimizing IRS deployment as well as their promising solutions are discussed. A case study is also presented to compare the performance of different IRS deployment strategies and draw useful insights for practical design.
The Neural Coding Framework for Learning Generative Models
Venue: arXiv cs.LG 2022
Year: 2022

Neural generative models can be used to learn complex probability distributions from data, to sample from them, and to produce probability density estimates. We propose a computational framework for developing neural generative models inspired by the theory of predictive processing in the brain. According to predictive processing theory, the neurons in the brain form a hierarchy in which neurons in one level form expectations about sensory inputs from another level. These neurons update their local models based on differences between their expectations and the observed signals. In a similar way, artificial neurons in our generative models predict what neighboring neurons will do, and adjust their parameters based on how well the predictions matched reality. In this work, we show that the neural generative models learned within our framework perform well in practice across several benchmark datasets and metrics and either remain competitive with or significantly outperform other generative models with similar functionality (such as the variational auto-encoder).
Generative Adversarial User Privacy in Lossy Single-Server Information Retrieval
Venue: arXiv cs.LG 2022
Year: 2022

We propose to extend the concept of private information retrieval by allowing for distortion in the retrieval process and relaxing the perfect privacy requirement at the same time. In particular, we study the trade-off between download rate, distortion, and user privacy leakage, and show that in the limit of large file sizes this trade-off can be captured via a novel information-theoretical formulation for datasets with a known distribution. Moreover, for scenarios where the statistics of the dataset is unknown, we propose a new deep learning framework by leveraging a generative adversarial network approach, which allows the user to learn efficient schemes from the data itself. We evaluate the performance of the scheme on a synthetic Gaussian dataset as well as on the MNIST, CIFAR-10, and LSUN datasets. For the MNIST, CIFAR-10, and LSUN datasets, the data-driven approach significantly outperforms a nonlearning-based scheme which combines source coding with the download of multiple files.
Local Dampening: Differential Privacy for Non-numeric Queries via Local Sensitivity
Venue: arXiv cs.CR 2022
Year: 2022

Differential privacy is the state-of-the-art formal definition for data release under strong privacy guarantees. A variety of mechanisms have been proposed in the literature for releasing the output of numeric queries (e.g., the Laplace mechanism and smooth sensitivity mechanism). Those mechanisms guarantee differential privacy by adding noise to the true query's output. The amount of noise added is calibrated by the notions of global sensitivity and local sensitivity of the query that measure the impact of the addition or removal of an individual on the query's output. Mechanisms that use local sensitivity add less noise and, consequently, have a more accurate answer. However, although there has been some work on generic mechanisms for releasing the output of non-numeric queries using global sensitivity (e.g., the Exponential mechanism), the literature lacks generic mechanisms for releasing the output of non-numeric queries using local sensitivity to reduce the noise in the query's output. In this work, we remedy this shortcoming and present the local dampening mechanism. We adapt the notion of local sensitivity for the non-numeric setting and leverage it to design a generic non-numeric mechanism. We provide theoretical comparisons to the exponential mechanism and show under which conditions the local dampening mechanism is more accurate than the exponential mechanism. We illustrate the effectiveness of the local dampening mechanism by applying it to three diverse problems: (i) percentile selection problem. We report the p-th element in the database; (ii) Influential node analysis. Given an influence metric, we release the top-k most influential nodes while preserving the privacy of the relationship between nodes in the network; (iii) Decision tree induction. We provide a private adaptation to the ID3 algorithm to build decision trees from a given tabular dataset.