Miracle-dz/UniFAR-FTUs huggingface dataset:
Quantile Treatment Effects in R: The qte Package

When evaluating the effect of a program, policymakers may be interested in more than just the average impact of the program. For example, there may be cases where the mean net impact (benefit minus cost) is negative, but the program is still implemented because of the distributional impacts of the program. One example from economics is the effect of a job training program. Consider two job training programs that both have the same mean net impact which is negative. If the first job training program tends to increase wages at the top of the wage distribution, it is likely to be rejected by policymakers. On the other hand, if the second job training program tends to increase wages at the bottom of the wage distribution, it may be approved by policymakers. Traditional methods, such as linear regression where the effect of participating in a program is constrained to be the same across all individuals are not suitable for distinguishing between these two types of programs. Neither are methods that allow for treatment effect heterogeneity but identify only the average effect of participating in a program. The distributional methods available in the R package QTE are useful in cases such as the above example where the researcher is interested in both allowing for treatment effect heterogeneity and understanding some aspect of the treatment effect heterogeneity.
A Critical View of the Structural Causal Model

In the univariate case, we show that by comparing the individual complexities of univariate cause and effect, one can identify the cause and the effect, without considering their interaction at all. In our framework, complexities are captured by the reconstruction error of an autoencoder that operates on the quantiles of the distribution. Comparing the reconstruction errors of the two autoencoders, one for each variable, is shown to perform surprisingly well on the accepted causality directionality benchmarks. Hence, the decision as to which of the two is the cause and which is the effect may not be based on causality but on complexity. In the multivariate case, where one can ensure that the complexities of the cause and effect are balanced, we propose a new adversarial training method that mimics the disentangled structure of the causal model. We prove that in the multidimensional case, such modeling is likely to fit the data only in the direction of causality. Furthermore, a uniqueness result shows that the learned model is able to identify the underlying causal and residual (noise) components. Our multidimensional method outperforms the literature methods on both synthetic and real world datasets.
Using wavelets to analyze similarities in image datasets

Deep learning image classifiers usually rely on huge training sets and their training process can be described as learning the similarities and differences among training images. But, images in large training sets are not usually studied from this perspective and fine-level similarities and differences among images is usually overlooked. Some studies aim to identify the influential and redundant training images, but such methods require a model that is already trained on the entire training set. Here, we show that analyzing the contents of large training sets can provide valuable insights about the classification task at hand, prior to training a model on them. We use wavelet decomposition of images and other image processing tools to perform such analysis, with no need for a pre-trained model. This makes the analysis of training sets, straightforward and fast. We show that similar images in standard datasets (such as CIFAR) can be identified in a few seconds, a significant speed-up compared to alternative methods in the literature. We also show that similarities between training and testing images may explain the generalization of models and their mistakes. Finally, we investigate the similarities between images in relation to decision boundaries of a trained model.
Non-Abelian Dirac node braiding and near-degeneracy of correlated phases at odd integer filling in magic-angle twisted bilayer graphene

We use the density matrix renormalization group (DMRG) to study the correlated electron states favored by the Coulomb interaction projected onto the narrow bands of twisted bilayer graphene within a spinless one-valley model. The Hilbert space of the narrow bands is constructed from a pair of hybrid Wannier states with opposite Chern numbers, maximally localized in one direction and Bloch extended in another direction. Depending on the parameters in the Bistritzer-Macdonald model, the DMRG in this basis determines the ground state at one particle per unit cell to be either the quantum anomalous Hall (QAH) state or a state with zero Hall conductivity which is nearly a product state. Based on this form, we then apply the variational method to study their competition, thus identifying three states: the QAH, a gapless ${C}_{2}\mathcal{T}$-symmetric nematic, and a gapped ${C}_{2}\mathcal{T}$-symmetric stripe. In the chiral limit, the energies of the two ${C}_{2}\mathcal{T}$-symmetric states are found to be significantly above the energy of the QAH. However, all three states are nearly degenerate at the realistic parameters of the Bistritzer-Macdonald model. The single-particle spectrum of the nematic contains either a quadratic node or two close Dirac nodes near $\mathrm{\ensuremath{\Gamma}}$. Motivated by the Landau level degeneracy found in this state, we propose it to be the state observed at the charge neutrality point once spin and valley degeneracies are restored. The optimal period for the ${C}_{2}\mathcal{T}$ stripe state is found to be two unit cells. In addition, using the fact that the topological charge of the nodes in the ${C}_{2}\mathcal{T}$-nematic phase is no longer described simply by their winding numbers once the translation symmetry is broken, but rather by certain elements of a non-Abelian group that was recently pointed out, we identify the mechanism of the gap opening within the ${C}_{2}\mathcal{T}$ stripe state. Although the nodes at the Fermi energy are locally stable, they can be annihilated after braiding with other nodes connecting them to adjacent (folded) bands. Therefore, if the translation symmetry is broken, the gap at one particle per unit cell can open even if the system preserves the ${C}_{2}\mathcal{T}$ and valley $\text{U}(1)$ symmetries, and the gap to remote bands remains open.
Training Question Answering Models from Synthetic Data

Question and answer generation is a data augmentation method that aims to improve question answering (QA) models given the limited amount of human labeled data. However, a considerable gap remains between synthetic and human-generated question-answer pairs. This work aims to narrow this gap by taking advantage of large language models and explores several factors such as model size, quality of pretrained models, scale of data synthesized, and algorithmic choices. On the SQuAD1.1 question answering task, we achieve higher accuracy using solely synthetic questions and answers than when using the SQuAD1.1 training set questions alone. Removing access to real Wikipedia data, we synthesize questions and answers from a synthetic corpus generated by an 8.3 billion parameter GPT-2 model. With no access to human supervision and only access to other models, we are able to train state of the art question answering networks on entirely model-generated data that achieve 88.4 Exact Match (EM) and 93.9 F1 score on the SQuAD1.1 dev set. We further apply our methodology to SQuAD2.0 and show a 2.8 absolute gain on EM score compared to prior work using synthetic data.
Stochastic Subspace Cubic Newton Method

In this paper, we propose a new randomized second-order optimization algorithm---Stochastic Subspace Cubic Newton (SSCN)---for minimizing a high dimensional convex function $f$. Our method can be seen both as a {\em stochastic} extension of the cubically-regularized Newton method of Nesterov and Polyak (2006), and a {\em second-order} enhancement of stochastic subspace descent of Kozak et al. (2019). We prove that as we vary the minibatch size, the global convergence rate of SSCN interpolates between the rate of stochastic coordinate descent (CD) and the rate of cubic regularized Newton, thus giving new insights into the connection between first and second-order methods. Remarkably, the local convergence rate of SSCN matches the rate of stochastic subspace descent applied to the problem of minimizing the quadratic function $\frac12 (x-x^*)^\top \nabla^2f(x^*)(x-x^*)$, where $x^*$ is the minimizer of $f$, and hence depends on the properties of $f$ at the optimum only. Our numerical experiments show that SSCN outperforms non-accelerated first-order CD algorithms while being competitive to their accelerated variants.
Safe Screening for the Generalized Conditional Gradient Method

The conditional gradient method (CGM) has been widely used for fast sparse approximation, having a low per iteration computational cost for structured sparse regularizers. We explore the sparsity acquiring properties of a generalized CGM (gCGM), where the constraint is replaced by a penalty function based on a gauge penalty; this can be done without significantly increasing the per-iteration computation, and applies to general notions of sparsity. Without assuming bounded iterates, we show $O(1/t)$ convergence of the function values and gap of gCGM. We couple this with a safe screening rule, and show that at a rate $O(1/(t\delta^2))$, the screened support matches the support at the solution, where $\delta \geq 0$ measures how close the problem is to being degenerate. In our experiments, we show that the gCGM for these modified penalties have similar feature selection properties as common penalties, but with potentially more stability over the choice of hyperparameter.
Angle Aware User Cooperation for Secure Massive MIMO in Rician Fading Channel

Massive multiple-input multiple-output communications can achieve high-level security by concentrating radio frequency signals towards the legitimate users. However, this system is vulnerable in a Rician fading environment if the eavesdropper positions itself such that its channel is highly “similar” to the channel of a legitimate user. To address this problem, this paper proposes an angle aware user cooperation (AAUC) scheme, which avoids direct transmission to the attacked user and relies on other users for cooperative relaying. The proposed scheme only requires the eavesdropper’s angle information, and adopts an angular secrecy model to represent the average secrecy rate of the attacked system. With this angular model, the AAUC problem turns out to be nonconvex, and a successive convex optimization algorithm, which converges to a Karush-Kuhn-Tucker solution, is proposed. Furthermore, a closed-form solution and a Bregman first-order method are derived for the cases of large-scale antennas and large-scale users, respectively. Extension to the intelligent reflecting surfaces based scheme is also discussed. Simulation results demonstrate the effectiveness of the proposed successive convex optimization based AAUC scheme, and also validate the low-complexity nature of the proposed large-scale optimization algorithms.
Markov Logic Networks with Complex Weights: Expressivity, Liftability and Fourier Transforms

We study expressivity of Markov logic networks (MLNs). We introduce complex MLNs, which use complex-valued weights, and we show that, unlike standard MLNs with real-valued weights, complex MLNs are fully expressive. We then observe that discrete Fourier transform can be computed using weighted first order model counting (WFOMC) with complex weights and use this observation to design an algorithm for computing relational marginal polytopes which needs substantially less calls to a WFOMC oracle than a recent algorithm.
Estimating Q(s, s') with Deep Deterministic Dynamics Gradients

In this paper, we introduce a novel form of value function, $Q(s, s')$, that expresses the utility of transitioning from a state $s$ to a neighboring state $s'$ and then acting optimally thereafter. In order to derive an optimal policy, we develop a forward dynamics model that learns to make next-state predictions that maximize this value. This formulation decouples actions from values while still learning off-policy. We highlight the benefits of this approach in terms of value function transfer, learning within redundant action spaces, and learning off-policy from state observations generated by sub-optimal or completely random policies. Code and videos are available at this http URL.
Batch Normalization Biases Deep Residual Networks Towards Shallow Paths

Batch normalization has multiple benefits. It improves the conditioning of the loss landscape, and is a surprisingly effective regularizer. However, the most important benefit of batch normalization arises in residual networks, where it dramatically increases the largest trainable depth. We identify the origin of this benefit: At initialization, batch normalization downscales the residual branch relative to the skip connection, by a normalizing factor proportional to the square root of the network depth. This ensures that, early in training, the function computed by deep normalized residual networks is dominated by shallow paths with well-behaved gradients. We use this insight to develop a simple initialization scheme which can train very deep residual networks without normalization. We also clarify that, although batch normalization does enable stable training with larger learning rates, this benefit is only useful when one wishes to parallelize training over large batch sizes. Our results help isolate the distinct benefits of batch normalization in different architectures.
Risk-Aware Optimization of Age of Information in the Internet of Things

Minimization of the expected value of age of information (AoI) is a risk-neutral approach, and it thus cannot capture rare, yet critical, events with potentially large AoI. In order to capture the effect of these events, in this paper, the notion of conditional value-at-risk (CVaR) is proposed as an effective coherent risk measure that is suitable for minimization of AoI for real-time IoT status updates. In the considered monitoring system, an IoT device monitors a physical process and sends the status updates to a remote receiver with an updating cost. The optimal status update process is designed to jointly minimize the AoI at the receiver, the CVaR of the AoI at the receiver, and the energy cost. This stochastic optimization problem is formulated as an infinite horizon discounted risk-aware Markov decision process (MDP), which is computationally intractable due to the time inconsistency of the CVaR. By exploiting the special properties of coherent risk measures, the risk-aware MDP is reduced to a standard MDP with an augmented state space, for which we derive the optimal stationary policy using dynamic programming. In particular, the optimal history-dependent policy of the risk-aware MDP is shown to depend on the history only through the augmented system states and can be readily constructed using the optimal stationary policy of the augmented MDP. The proposed solution is shown to be computationally tractable and able to minimize the AoI in real-time IoT monitoring systems in a risk-aware manner.
Novel insights in the Levy–Levy–Solomon agent-based economic market model

The Levy-Levy-Solomon model (A microscopic model of the stock market: cycles, booms, and crashes, Economic Letters 45 (1))is one of the most influential agent-based economic market models. In several publications this model has been discussed and analyzed. Especially Lux and Zschischang (Some new results on the Levy, Levy and Solomon microscopic stock market model, Physica A, 291(1-4)) have shown that the model exhibits finite-size effects. In this study we extend existing work in several directions. First, we show simulations which reveal finite-size effects of the model. Secondly, we shed light on the origin of these finite-size effects. Furthermore, we demonstrate the sensitivity of the Levy-Levy-Solomon model with respect to random numbers. Especially, we can conclude that a low-quality pseudo random number generator has a huge impact on the simulation results. Finally, we study the impact of the stopping criteria in the market clearance mechanism of the Levy-Levy-Solomon model.
A Model-Based Derivative-Free Approach to Black-Box Adversarial Examples: BOBYQA

We demonstrate that model-based derivative free optimisation algorithms can generate adversarial targeted misclassification of deep networks using fewer network queries than non-model-based methods. Specifically, we consider the black-box setting, and show that the number of networks queries is less impacted by making the task more challenging either through reducing the allowed $\ell^{\infty}$ perturbation energy or training the network with defences against adversarial misclassification. We illustrate this by contrasting the BOBYQA algorithm with the state-of-the-art model-free adversarial targeted misclassification approaches based on genetic, combinatorial, and direct-search algorithms. We observe that for high $\ell^{\infty}$ energy perturbations on networks, the aforementioned simpler model-free methods require the fewest queries. In contrast, the proposed BOBYQA based method achieves state-of-the-art results when the perturbation energy decreases, or if the network is trained against adversarial perturbations.
HRank: Filter Pruning Using High-Rank Feature Map

Neural network pruning offers a promising prospect to facilitate deploying deep neural networks on resource-limited devices. However, existing methods are still challenged by the training inefficiency and labor cost in pruning designs, due to missing theoretical guidance of non-salient network components. In this paper, we propose a novel filter pruning method by exploring the High Rank of feature maps (HRank). Our HRank is inspired by the discovery that the average rank of multiple feature maps generated by a single filter is always the same, regardless of the number of image batches CNNs receive. Based on HRank, we develop a method that is mathematically formulated to prune filters with low-rank feature maps. The principle behind our pruning is that low-rank feature maps contain less information, and thus pruned results can be easily reproduced. Besides, we experimentally show that weights with high-rank feature maps contain more important information, such that even when a portion is not updated, very little damage would be done to the model performance. Without introducing any additional constraints, HRank leads to significant improvements over the state-of-the-arts in terms of FLOPs and parameters reduction, with similar accuracies. For example, with ResNet-110, we achieve a 58.2%-FLOPs reduction by removing 59.2% of the parameters, with only a small loss of $0.14\%$ in top-1 accuracy on CIFAR-10. With Res-50, we achieve a 43.8%-FLOPs reduction by removing 36.7% of the parameters, with only a loss of 1.17% in the top-1 accuracy on ImageNet. The codes can be available at https://github.com/lmbxmu/HRank.
How Transferable are the Representations Learned by Deep Q Agents?

In this paper, we consider the source of Deep Reinforcement Learning (DRL)'s sample complexity, asking how much derives from the requirement of learning useful representations of environment states and how much is due to the sample complexity of learning a policy. While for DRL agents, the distinction between representation and policy may not be clear, we seek new insight through a set of transfer learning experiments. In each experiment, we retain some fraction of layers trained on either the same game or a related game, comparing the benefits of transfer learning to learning a policy from scratch. Interestingly, we find that benefits due to transfer are highly variable in general and non-symmetric across pairs of tasks. Our experiments suggest that perhaps transfer from simpler environments can boost performance on more complex downstream tasks and that the requirements of learning a useful representation can range from negligible to the majority of the sample complexity, based on the environment. Furthermore, we find that fine-tuning generally outperforms training with the transferred layers frozen, confirming an insight first noted in the classification setting.
Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networks

Spiking neural networks (SNNs) are well suited for spatio-temporal learning and implementations on energy-efficient event-driven neuromorphic processors. However, existing SNN error backpropagation (BP) methods lack proper handling of spiking discontinuities and suffer from low performance compared with the BP methods for traditional artificial neural networks. In addition, a large number of time steps are typically required to achieve decent performance, leading to high latency and rendering spike-based computation unscalable to deep architectures. We present a novel Temporal Spike Sequence Learning Backpropagation (TSSL-BP) method for training deep SNNs, which breaks down error backpropagation across two types of inter-neuron and intra-neuron dependencies and leads to improved temporal learning precision. It captures inter-neuron dependencies through presynaptic firing times by considering the all-or-none characteristics of firing activities and captures intra-neuron dependencies by handling the internal evolution of each neuronal state in time. TSSL-BP efficiently trains deep SNNs within a much shortened temporal window of a few steps while improving the accuracy for various image classification datasets including CIFAR10.
Quantitative decompositions of Lipschitz mappings into metric spaces

We study the quantitative properties of Lipschitz mappings from Euclidean spaces into metric spaces. We prove that it is always possible to decompose the domain of such a mapping into pieces on which the mapping "behaves like a projection mapping" along with a "garbage set" that is arbitrarily small in an appropriate sense. Moreover, our control is quantitative, i.e., independent of both the particular mapping and the metric space it maps into. This improves a theorem of Azzam-Schul from the paper "Hard Sard", and answers a question left open in that paper. The proof uses ideas of quantitative differentiation, as well as a detailed study of how to supplement Lipschitz mappings by additional coordinates to form bi-Lipschitz mappings.