Miracle-dz/UniFAR-FTUs huggingface dataset:
On instabilities of deep learning in image reconstruction and the potential costs of AI

Deep learning, due to its unprecedented success in tasks such as image classification, has emerged as a new tool in image reconstruction with potential to change the field. In this paper, we demonstrate a crucial phenomenon: Deep learning typically yields unstable methods for image reconstruction. The instabilities usually occur in several forms: 1) Certain tiny, almost undetectable perturbations, both in the image and sampling domain, may result in severe artefacts in the reconstruction; 2) a small structural change, for example, a tumor, may not be captured in the reconstructed image; and 3) (a counterintuitive type of instability) more samples may yield poorer performance. Our stability test with algorithms and easy-to-use software detects the instability phenomena. The test is aimed at researchers, to test their networks for instabilities, and for government agencies, such as the Food and Drug Administration (FDA), to secure safe use of deep learning methods.
Learning to see across Domains and Modalities

Deep learning has raised hopes and expectations as a general solution for many applications; indeed it has proven effective, but it also showed a strong dependence on large quantities of data. Luckily, it has been shown that, even when data is scarce, a successful model can be trained by reusing prior knowledge. Thus, developing techniques for transfer learning, in its broadest definition, is a crucial element towards the deployment of effective and accurate intelligent systems. This thesis will focus on a family of transfer learning methods applied to the task of visual object recognition, specifically image classification. Transfer learning is a general term, and specific settings have been given specific names: when the learner has only access to unlabeled data from the a target domain and labeled data from a different domain (the source), the problem is known as that of "unsupervised domain adaptation" (DA). The first part of this work will focus on three methods for this setting: one of these methods deals with features, one with images while the third one uses both. The second part will focus on the real life issues of robotic perception, specifically RGB-D recognition. Robotic platforms are usually not limited to color perception; very often they also carry a Depth camera. Unfortunately, the depth modality is rarely used for visual recognition due to the lack of pretrained models from which to transfer and little data to train one on from scratch. Two methods for dealing with this scenario will be presented: one using synthetic data and the other exploiting cross-modality transfer learning.
Simultaneously Learning Vision and Feature-based Control Policies for Real-world Ball-in-a-Cup

We present a method for fast training of vision based control policies on real robots. The key idea behind our method is to perform multi-task Reinforcement Learning with auxiliary tasks that differ not only in the reward to be optimized but also in the state-space in which they operate. In particular, we allow auxiliary task policies to utilize task features that are available only at training-time. This allows for fast learning of auxiliary policies, which subsequently generate good data for training the main, vision-based control policies. This method can be seen as an extension of the Scheduled Auxiliary Control (SAC-X) framework. We demonstrate the efficacy of our method by using both a simulated and real-world Ball-in-a-Cup game controlled by a robot arm. In simulation, our approach leads to significant learning speed-ups when compared to standard SAC-X. On the real robot we show that the task can be learned from-scratch, i.e., with no transfer from simulation and no imitation learning. Videos of our learned policies running on the real robot can be found at this https URL.
Extreme Tensoring for Low-Memory Preconditioning

State-of-the-art models are now trained with billions of parameters, reaching hardware limits in terms of memory consumption. This has created a recent demand for memory-efficient optimizers. To this end, we investigate the limits and performance tradeoffs of memory-efficient adaptively preconditioned gradient methods. We propose extreme tensoring for high-dimensional stochastic optimization, showing that an optimizer needs very little memory to benefit from adaptive preconditioning. Our technique applies to arbitrary models (not necessarily with tensor-shaped parameters), and is accompanied by regret and convergence guarantees, which shed light on the tradeoffs between preconditioner quality and expressivity. On a large-scale NLP model, we reduce the optimizer memory overhead by three orders of magnitude, without degrading performance.
Rethinking Generative Mode Coverage: A Pointwise Guaranteed Approach

Many generative models have to combat $\textit{missing modes}$. The conventional wisdom to this end is by reducing through training a statistical distance (such as $f$-divergence) between the generated distribution and provided data distribution. But this is more of a heuristic than a guarantee. The statistical distance measures a $\textit{global}$, but not $\textit{local}$, similarity between two distributions. Even if it is small, it does not imply a plausible mode coverage. Rethinking this problem from a game-theoretic perspective, we show that a complete mode coverage is firmly attainable. If a generative model can approximate a data distribution moderately well under a global statistical distance measure, then we will be able to find a mixture of generators that collectively covers $\textit{every}$ data point and thus $\textit{every}$ mode, with a lower-bounded generation probability. Constructing the generator mixture has a connection to the multiplicative weights update rule, upon which we propose our algorithm. We prove that our algorithm guarantees complete mode coverage. And our experiments on real and synthetic datasets confirm better mode coverage over recent approaches, ones that also use generator mixtures but rely on global statistical distances.
Progressively Growing Generative Adversarial Networks for High Resolution Semantic Segmentation of Satellite Images

Machine learning has proven to be useful in classification and segmentation of images. In this paper, we evaluate a training methodology for pixel-wise segmentation on high resolution satellite images using progressive growing of generative adversarial networks. We apply our model to segmenting building rooftops and compare these results to conventional methods for rooftop segmentation. We present our findings using the SpaceNet version 2dataset. Progressive GAN training achieved a test accuracy of 93% compared to 89% for traditional GAN training.
Proving Expected Sensitivity of Probabilistic Programs with Randomized Execution Time

The notion of program sensitivity (aka Lipschitz continuity) specifies that changes in the program input result in proportional changes to the program output. For probabilistic programs the notion is naturally extended to expected sensitivity. Previous approach develops a nice relational program logic framework for expected sensitivity of probabilistic while loops, where the number of iterations is fixed and bounded. In this work we present a sound approach for sensitivity analysis of probabilistic while loops, where the number of iterations is not fixed, but is randomized and only the expected number of iterations is finite. We demonstrate the effectiveness of our approach on several classical examples, e.g., mini-roulette and regularized stochastic gradient descent algorithm.
Throughput–Outage Analysis and Evaluation of Cache-Aided D2D Networks With Measured Popularity Distributions

Caching of video files on user devices, combined with file exchange through device-to-device (D2D) communications is a promising method for increasing the throughput of wireless networks. Previous theoretical investigations showed that throughput can be increased by orders of magnitude, but assumed a Zipf distribution for modeling the popularity distribution, which was based on observations in wired networks. Thus the question whether cache-aided D2D video distribution can provide in practice the benefits promised by existing theoretical literature remains open. To answer this question, we provide new results specifically for popularity distributions of video requests of mobile users. Based on an extensive real-world dataset, we adopt a generalized distribution, known as Mandelbrot-Zipf (MZipf) distribution. We first show that this popularity distribution can fit the practical data well. Using this distribution, we analyze the throughput–outage tradeoff of the cache-aided D2D network and show that the scaling law is identical to the case of Zipf popularity distribution when the MZipf distribution is sufficiently skewed, implying that the benefits previously promised in the literature could indeed be realized in practice. To support the theory, practical evaluations using numerical experiments are provided, and show that the cache-aided D2D can outperform the conventional unicasting from base stations.
Value constrained model-free continuous control

The naive application of Reinforcement Learning algorithms to continuous control problems -- such as locomotion and manipulation -- often results in policies which rely on high-amplitude, high-frequency control signals, known colloquially as bang-bang control. Although such solutions may indeed maximize task reward, they can be unsuitable for real world systems. Bang-bang control may lead to increased wear and tear or energy consumption, and tends to excite undesired second-order dynamics. To counteract this issue, multi-objective optimization can be used to simultaneously optimize both the reward and some auxiliary cost that discourages undesired (e.g. high-amplitude) control. In principle, such an approach can yield the sought after, smooth, control policies. It can, however, be hard to find the correct trade-off between cost and return that results in the desired behavior. In this paper we propose a new constraint-based reinforcement learning approach that ensures task success while minimizing one or more auxiliary costs (such as control effort). We employ Lagrangian relaxation to learn both (a) the parameters of a control policy that satisfies the desired constraints and (b) the Lagrangian multipliers for the optimization. Moreover, we demonstrate that we can satisfy constraints either in expectation or in a per-step fashion, and can even learn a single policy that is able to dynamically trade-off between return and cost. We demonstrate the efficacy of our approach using a number of continuous control benchmark tasks, a realistic, energy-optimized quadruped locomotion task, as well as a reaching task on a real robot arm.
Structured Bayesian Compression for Deep Models in Mobile-Enabled Devices for Connected Healthcare

Deep models, typically deep neural networks, have millions of parameters, analyze medical data accurately, yet in a time-consuming method. However, energy cost effectiveness and computational efficiency are important for prerequisites developing and deploying mobile-enabled devices, the mainstream trend in connected healthcare. Therefore, deep models’ compression has become a problem of great significance for real-time health services. In this article, we first emphasize the use of Bayesian learning for model sparsity, effectively reducing the number of parameters while maintaining model performance. Specifically, with sparsity inducing priors, large parts of the network can be pruned with a simple retraining of arbitrary datasets. Then, we propose a novel structured Bayesian compression architecture by adaptively learning both group sparse and block sparse while also designing sparse-oriented mixture priors to improve the expandability of the compression model. Experimental results from both simulated datasets (MNIST) as well as practical medical datasets (Histopathologic Cancer) demonstrate the effectiveness and good performance of our framework on deep model compression.
Contrastive Variational Autoencoder Enhances Salient Features

Variational autoencoders are powerful algorithms for identifying dominant latent structure in a single dataset. In many applications, however, we are interested in modeling latent structure and variation that are enriched in a target dataset compared to some background---e.g. enriched in patients compared to the general population. Contrastive learning is a principled framework to capture such enriched variation between the target and background, but state-of-the-art contrastive methods are limited to linear models. In this paper, we introduce the contrastive variational autoencoder (cVAE), which combines the benefits of contrastive learning with the power of deep generative models. The cVAE is designed to identify and enhance salient latent features. The cVAE is trained on two related but unpaired datasets, one of which has minimal contribution from the salient latent features. The cVAE explicitly models latent features that are shared between the datasets, as well as those that are enriched in one dataset relative to the other, which allows the algorithm to isolate and enhance the salient latent features. The algorithm is straightforward to implement, has a similar run-time to the standard VAE, and is robust to noise and dataset purity. We conduct experiments across diverse types of data, including gene expression and facial images, showing that the cVAE effectively uncovers latent structure that is salient in a particular analysis.
Accelerating Stochastic Training for Over-parametrized Learning

We introduce MaSS (Momentum-added Stochastic Solver), an accelerated SGD method for optimizing over-parametrized models. Our method is simple and efficient to implement and does not require adapting hyper-parameters or computing full gradients in the course of optimization. Experimental evaluation of MaSS for several standard architectures of deep networks, including ResNet and convolutional networks, shows improved performance over Adam and SGD both in optimization and generalization. We prove accelerated convergence of MaSS over SGD and provide analysis for hyper-parameter selection in the quadratic case as well as some results in general strongly convex setting. In contrast, we show theoretically and verify empirically that the standard SGD+Nesterov can diverge for common choices of hyper-parameter values. We also analyze the practically important question of the dependence of the convergence rate and optimal hyper-parameters as functions of the mini-batch size, demonstrating three distinct regimes: linear scaling, diminishing returns and saturation.
Do Subsampled Newton Methods Work for High-Dimensional Data?

Subsampled Newton methods approximate Hessian matrices through subsampling techniques to alleviate the per-iteration cost. Previous results require Ω (d) samples to approximate Hessians, where d is the dimension of data points, making it less practical for high-dimensional data. The situation is deteriorated when d is comparably as large as the number of data points n, which requires to take the whole dataset into account, making subsampling not useful. This paper theoretically justifies the effectiveness of subsampled Newton methods on strongly convex empirical risk minimization with high dimensional data. Specifically, we provably require only Θ˜(deffγ) samples for approximating the Hessian matrices, where deffγ is the γ-ridge leverage and can be much smaller than d as long as nγ ≫ 1. Our theories work for three types of Newton methods: subsampled Netwon, distributed Newton, and proximal Newton.
Influence of nicotine and alcohol on sleep and implications on insomnia: the reward-attention circuit model

Dopamine, a neurotransmitter well known for regulating movement, reward, and learning, is emerging as one of the neuromodulators of wakefulness. Drugs that increase the level of dopamine in the brain (including, but not limited to, nicotine) also increase feelings of wakefulness. Diseases that are characterized by low dopamine levels, like Parkinson's disease, also are related to sleep disorders. In this work we investigate the influence that nicotine and alcohol exert on sleep, modeling the coupling of reward and thalamocortical circuits in a reward-attention circuit. Computer simulations of the circuit reflect the spiking behavior of neurons in the network under the presence or absence of nicotine or alcohol. Each neuron in the reward-attention model represents a population of cells in the circuit, and is described by a coupled system of nonlinear differential equations that replicates essential neurophysiological properties of the population. The computational simulations highlight aspects of clinical insomnia symptoms in Parkinson's disease, attention deficit hyperactivity disorder and autism spectrum disorder. Our results imply that nicotine can disrupt sleep, promoting wakefulness. In contrast, alcohol increases sleep latency (time to fall asleep). Also, the simulations suggest that alcohol has a sedative effect in people with insomnia.
Performance Analysis of S-SFR-based OFDMA Cellular Systems

Intercell interference coordination (ICIC) is considered as a promising technique to increase the spectral efficiency of OFDMA cellular systems. The soft frequency reuse (SFR) and fractional frequency reuse (FFR) are representative and efficient management techniques for ICIC. Herein, to enhance the performance of the SFR scheme, we propose a call admission (CAC) scheme. In this CAC scheme, called Spectrum handoff-SFR(S-SFR), the spectrum handoff technique is applied to the user equipment (UE) located near the cell center. We derive the traffic analysis model to describe the S-SFR. In addition, a two-dimensional (2-D) Markov chain and an outage analysis are used in our analytical model. From the traffic analysis, the significant performance measures are the outage probability, call blocking probability, system throughput and resource utilization. Based on those, the outage probability and system throughput are obtained using resource utilization as an interference pattern. The analytical results are verified with computer simulation results. Finally, we compare our proposed scheme with other ICI schemes.
Salt and Pepper Noise Removal with Multi-Class Dictionary Learning and L0 Norm Regularizations

Images may be corrupted by salt and pepper impulse noise during image acquisitions or transmissions. Although promising denoising performances have been recently obtained with sparse representations, how to restore high-quality images remains challenging and open. In this work, image sparsity is enhanced with a fast multiclass dictionary learning, and then both the sparsity regularization and robust data fidelity are formulated as minimizations of L0-L0 norms for salt and pepper impulse noise removal. Additionally, a numerical algorithm of modified alternating direction minimization is derived to solve the proposed denoising model. Experimental results demonstrate that the proposed method outperforms the compared state-of-the-art ones on preserving image details and achieving higher objective evaluation criteria.
CoMP Transmission for Safeguarding Dense Heterogeneous Networks with Imperfect CSI

To ensure reliable and secure communication in heterogeneous cellular network (HCN) with imperfect channel state information (CSI), we proposed a coordinated multipoint (CoMP) transmission scheme based on dual-threshold optimization, in which only base stations (BSs) with good channel conditions are selected for transmission. First, we present a candidate BSs formation policy to increase access efficiency, which provides a candidate region of serving BSs. Then, we design a CoMP networking strategy to select serving BSs from the set of candidate BSs, which degrades the influence of channel estimation errors and guarantees qualities of communication links. Finally, we analyze the performance of the proposed scheme, and present a dual-threshold optimization model to further support the performance. Numerical results are presented to verify our theoretical analysis, which draw a conclusion that the CoMP transmission scheme can ensure reliable and secure communication in dense HCNs with imperfect CSI.
A Multi-Scale Investigation of Global Electrical Heterogeneity: Effects of Body Habitus, Respiration, and Tissue Conductivity

Extracardiac factors such as respiration, fluid overload and body habitus have important effects on the ECG voltage. Vectorcardiographic (VCG) Global Electrical Heterogeneity (GEH) is associated with sudden cardiac death (SCD). Risk of SCD is especially high in end-stage renal disease patients (ESRD) on dialysis. However, extracardiac factors challenge ECG interpretation in ESRD patients. The effects of extracardiac factors on GEH have not been fully studied. To assess effects of extracardiac factors on ECG, we conducted a multi-scale study. An experimental data of ESRD patients and a previously developed biophysically detailed heart-torso model were used to investigate the effects of respiration, fluid overload and body habitus on the VCG and GEH.
Gaze-based Cursor Control Impairs Performance in Divided Attention

In this work we investigate the effects of switching from mouse cursor control to gaze-based control in a computerized divided attention game. We conducted experiments with nine participants performing a task that requires continuous focused concentration and frequent shifts of attention. Despite carefully controlling experimental and design aspects, the performance of subjects was considerably impaired when using gaze-based control. The participants were experienced users of the mouse control version of the task, we adjusted the difficulty to the more demanding conditions and selected the parameters of gaze input based on previous research findings. In contrast to our assumptions, experienced users could not get used to gaze-based control in the amount of experiments we performed. Additionally we consider the strategies of users, i.e. their method of problem solving, and found that it is possible to make progress in our task even during a short amount of practice. The results of this study provide evidence that the adoption of interfaces controlled by human eye-gaze in cognitively demanding environments require careful design, proper testing and sufficient user training.
Occupation times of alternating renewal processes with Lévy applications

Abstract In this paper we present a set of results relating to the occupation time α(t) of a process X(·). The first set of results concerns exact characterizations of α(t), e.g. in terms of its transform up to an exponentially distributed epoch. In addition, we establish a central limit theorem (entailing that a centered and normalized version of α(t)∕t converges to a zero-mean normal random variable as t→∞) and the tail asymptotics of ℙ(α(t)∕t≥q). We apply our findings to spectrally positive Lévy processes reflected at the infimum and establish various new occupation time results for the corresponding model.
Effects on the CMB from magnetic field dissipation before recombination

Magnetic fields present before decoupling are damped due to radiative viscosity. This energy injection affects the thermal and ionization history of the cosmic plasma. The implications for the CMB anisotropies and polarization are investigated for different parameter choices of a non helical stochastic magnetic field. Assuming a Gaussian smoothing scale determined by the magnetic damping wave number at recombination it is found that magnetic fields with present day strength less than 0.1 nG and negative magnetic spectral indices have a sizeable effect on the CMB temperature anisotropies and polarization.