GenAI4ELab/papercli-papers huggingface dataset:
DamoFD: Digging into Backbone Design on Face Detection
Authors: Yang Liu; Jiankang Deng; Fei Wang; Lei Shang; Xuansong Xie; Baigui Sun
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: Face detection (FD) has achieved remarkable success over the past few years, yet,
these leaps often arrive when consuming enormous computation costs. Moreover,
when considering a realistic situation, i.e., building a lightweight face detector
under a computation-scarce scenario, such heavy computation cost limits the application
of the face detector. To remedy this, several pioneering works design
tiny face detectors through off-the-shelf neural architecture search (NAS) technologies,
which are usually applied to the classification task. Thus, the searched
architectures are sub-optimal for the face detection task since some design criteria
between detection and classification task are different. As a representative, the
face detection backbone design needs to guarantee the stage-level detection ability
while it is not required for the classification backbone. Furthermore, the detection
backbone consumes a vast body of inference budgets in the whole detection framework.
Considering the intrinsic design requirement and the virtual importance role
of the face detection backbone, we thus ask a critical question: How to employ
NAS to search FD-friendly backbone architecture? To cope with this question,
we propose a distribution-dependent stage-aware ranking score (DDSAR-Score)
to explicitly characterize the stage-level expressivity and identify the individual
importance of each stage, thus satisfying the aforementioned design criterion of
the FD backbone. Based on our proposed DDSAR-Score, we conduct comprehensive
experiments on the challenging Wider Face benchmark dataset and achieve
dominant performance across a wide range of compute regimes. In particular,
compared to the tiniest face detector SCRFD-0.5GF, our method is +2.5 % better
in Average Precision (AP) score when using the same amount of FLOPs. The
code is avaliable at https://github.com/ly19965/FaceMaas/tree/master/face_project/face_detection/DamoFD.
Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning
Authors: Zhen Wang; Rameswar Panda; Leonid Karlinsky; Rogerio Feris; Huan Sun; Yoon Kim
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: Prompt tuning, in which a base pretrained model is adapted to each task via conditioning on learned prompt vectors, has emerged as a promising approach for efficiently adapting large language models to multiple downstream tasks. However, existing methods typically learn soft prompt vectors from scratch, and it has not been clear how to exploit the rich cross-task knowledge with prompt vectors in a multitask learning setting. We propose multitask prompt tuning (MPT), which first learns a single transferable prompt by distilling knowledge from multiple task-specific source prompts. We then learn multiplicative low rank updates to this shared prompt to efficiently adapt it to each downstream target task. Extensive experiments on 23 NLP datasets demonstrate that our proposed approach outperforms the state-of-the-art methods, including the full finetuning baseline in some cases, despite only tuning $0.035\%$ as many task-specific parameters.
Large Language Models Can Self-improve
Authors: Jiaxin Huang; Shixiang Shane Gu; Le Hou; Yuexin Wu; Xuezhi Wang; Hongkun Yu; Jiawei Han
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improve their reasoning abilities by self-thinking without external inputs. In this work, we demonstrate that an LLM is also capable of self-improving with only unlabeled datasets. We use a pre-trained LLM to generate “high-confidence” rationale-augmented answers for unlabeled questions using Chain-of-Thought prompting and self-consistency, and fine-tune the LLM using those self-generated solutions as target outputs. We show that our approach improves the general reasoning ability of a 540B-parameter LLM (74.4%→82.1% on GSM8K, 78.2%→83.0% on DROP, 90.0%→94.4% on OpenBookQA, and 63.4%→67.9% on ANLI-A3) and achieves state-of-the-art-level performance, without any ground truth label. We conduct ablation studies and show that finetuning on reasoning is critical for self-improvement.
Learning What and Where: Disentangling Location and Identity Tracking Without Supervision
Authors: Manuel Traub; Sebastian Otte; Tobias Menge; Matthias Karlbauer; Jannik Thuemmel; Martin V. Butz
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: Our brain can almost effortlessly decompose visual data streams into background and salient objects. Moreover, it can anticipate object motion and interactions, which are crucial abilities for conceptual planning and reasoning. Recent object reasoning datasets, such as CATER, have revealed fundamental shortcomings of current vision-based AI systems, particularly when targeting explicit object representations, object permanence, and object reasoning. Here we introduce a self-supervised LOCation and Identity tracking system (Loci), which excels on the CATER tracking challenge. Inspired by the dorsal and ventral pathways in the brain, Loci tackles the binding problem by processing separate, slot-wise encodings of 'what' and 'where'. Loci's predictive coding-like processing encourages active error minimization, such that individual slots tend to encode individual objects. Interactions between objects and object dynamics are processed in the disentangled latent space. Truncated backpropagation through time combined with forward eligibility accumulation significantly speeds up learning and improves memory efficiency. Besides exhibiting superior performance in current benchmarks, Loci effectively extracts objects from video streams and separates them into location and Gestalt components. We believe that this separation offers a representation that will facilitate effective planning and reasoning on conceptual levels.
Near-optimal Coresets for Robust Clustering
Authors: Lingxiao Huang; Shaofeng H.-C. Jiang; Jianing Lou; Xuan Wu
Venue: ICLR 2023 (ICLR 2023 notable top 5%)

Abstract: We consider robust clustering problems in $\mathbb{R}^d$, specifically $k$-clustering problems (e.g., $k$-Median and $k$-Means) with $m$ \emph{outliers}, where the cost for a given center set $C \subset \mathbb{R}^d$ aggregates the distances from $C$ to all but the furthest $m$ data points, instead of all points as in classical clustering. We focus on the $\epsilon$-coreset for robust clustering, a small proxy of the dataset that preserves the clustering cost within $\epsilon$-relative error for all center sets. Our main result is an $\epsilon$-coreset of size $O(m + \mathrm{poly}(k \epsilon^{-1}))$ that can be constructed in near-linear time. This significantly improves previous results, which either suffers an exponential dependence on $(m + k)$ [Feldman and Schulman, SODA'12], or has a weaker bi-criteria guarantee [Huang et al., FOCS'18]. Furthermore, we show this dependence in $m$ is nearly-optimal, and the fact that it is isolated from other factors may be crucial for dealing with large number of outliers. We construct our coresets by adapting to the outlier setting a recent framework [Braverman et al., FOCS'22] which was designed for capacity-constrained clustering, overcoming a new challenge that the participating terms in the cost, particularly the excluded $m$ outlier points, are dependent on the center set $C$. We validate our coresets on various datasets, and we observe a superior size-accuracy tradeoff compared with popular baselines including uniform sampling and sensitivity sampling. We also achieve a significant speedup of existing approximation algorithms for robust clustering using our coresets.
Diffusion-based Image Translation using disentangled style and content representation
Authors: Gihyun Kwon; Jong Chul Ye
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: Diffusion-based image translation guided by  semantic texts   or a single target image   has enabled flexible style transfer which is not limited to the specific domains. 
Unfortunately, due to the stochastic nature of diffusion models, it is often  difficult to maintain the original content of the image  during the reverse diffusion.
To address this, here we present a novel diffusion-based unsupervised image translation method, dubbed as DiffuseIT, using disentangled style and content representation.
 Specifically, inspired by the  slicing Vision Transformer, we extract intermediate keys of multihead self attention layer  from ViT model and used them as the content preservation loss. Then, an image guided style transfer is performed by matching the [CLS] classification token from the denoised samples and target image, whereas additional CLIP loss is used for the text-driven style transfer.
  To further accelerate the semantic change during the reverse  diffusion, we also propose a novel semantic divergence loss and resampling strategy. 
 Our experimental results show that the proposed method outperforms state-of-the-art baseline models in both text-guided and image-guided translation tasks.
Human-Guided Fair Classification for Natural Language Processing
Authors: Florian E. Dorner; Momchil Peychev; Nikola Konstantinov; Naman Goel; Elliott Ash; Martin Vechev
Venue: ICLR 2023 (ICLR 2023 notable top 25%)

Abstract: Text classifiers have promising applications in high-stake tasks such as resume screening and content moderation. These classifiers must be fair and avoid discriminatory decisions by being invariant to perturbations of sensitive attributes such as gender or ethnicity. However, there is a gap between human intuition about these perturbations and the formal similarity specifications capturing them. While existing research has started to address this gap, current methods are based on hardcoded word replacements, resulting in specifications with limited expressivity or ones that fail to fully align with human intuition (e.g., in cases of asymmetric counterfactuals). This work proposes novel methods for bridging this gap by discovering expressive and intuitive individual fairness specifications. We show how to leverage unsupervised style transfer and GPT-3's zero-shot capabilities to automatically generate expressive candidate pairs of semantically similar sentences that differ along sensitive attributes. We then validate the generated pairs via an extensive crowdsourcing study, which confirms that a lot of these pairs align with human intuition about fairness in the context of toxicity classification. Finally, we show how limited amounts of human feedback can be leveraged to learn a similarity specification that can be used to train downstream fairness-aware models.
Mechanistic Mode Connectivity
Authors: Ekdeep Singh Lubana; Eric J Bigelow; Robert P. Dick; David Krueger; Hidenori Tanaka
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: With the rise of pretrained models, fine-tuning has become of central importance in deep learning. However, unlike retraining from scratch, fine-tuning can fail to qualitatively change the behavior of a pre-trained network. For instance, we find in practice that naive fine-tuning does not eliminate a model’s sensitivity to spurious features. To understand and address this limitation, we study the geometry of neural network loss landscapes through the lens of mode-connectivity. Our work addresses two questions about mode-connectivity: 1) Are models trained on different data distributions mode-connected? 2) Can we fine tune a pre-trained model to switch modes? We define a notion of mechanistic mode-connectivity, and find that only models that already share the same invariances (which we call “mechanistically similar”) are mechanistically mode-connected. We hypothesize this property explains inability of naive fine-tuning methods to induce invariance to spurious features. Based on our analysis, we propose and validate a method of “mechanistic fine-tuning” called connectivity-based fine-tuning (CBFT)
Deep Power Laws for Hyperparameter Optimization
Authors: Arlind Kadra; Maciej Janowski; Martin Wistuba; Josif Grabocka
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Hyperparameter optimization is an important subfield of machine learning that focuses on tuning the hyperparameters of a chosen algorithm to achieve peak performance. Recently, there has been a stream of methods that tackle the issue of hyperparameter optimization, however, most of the methods do not exploit the scaling law property of learning curves. In this work, we propose Deep Power Law (DPL), a neural network model conditioned to yield predictions that follow a power-law scaling pattern. Our model dynamically decides which configurations to pause and train incrementally by making use of multi-fidelity estimation. We compare our method against 7 state-of-the-art competitors on 3 benchmarks related to tabular, image, and NLP datasets covering 59 diverse search spaces. Our method achieves the best results across all benchmarks by obtaining the best any-time results compared to all competitors.
Parameter Averaging for Feature Ranking
Authors: Talip Ucar; Ehsan Hajiramezanali
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Neural Networks are known to be sensitive to initialisation. The methods that rely on neural networks for feature ranking are not robust since they can have variations in their ranking when the model is initialized and trained with different random seeds. In this work, we introduce a novel method based on parameter averaging to estimate accurate and robust feature importance in tabular data setting, referred as XTab. We first initialize and train multiple instances of a shallow network (referred as local masks) with "different random seeds" for a downstream task. We then obtain a global mask model by "averaging the parameters" of local masks. We show that although the parameter averaging might result in a global model with higher loss, it still leads to the discovery of the ground-truth feature importance more consistently than an individual model does. We conduct extensive experiments on a variety of synthetic and real-world data, demonstrating that the XTab can be used to obtain the global feature importance that is not sensitive to sub-optimal model initialisation.
Adversarial Attack Detection Through Network Transport Dynamics
Authors: Skander Karkar; patrick gallinari; Alain Rakotomamonjy
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Adversarial attacks are perturbations to the input that don't change its class for a human observer, but fool a neural network into changing its prediction. In this paper, we propose a detector of such attacks that is based on the view of residual networks as discrete dynamical systems. The detector tells clean inputs from abnormal ones by comparing the discrete vector fields they follow throughout the network's layers before the final classification layer. We compare this detector favorably to other detectors on seen and unseen attacks. We also show that regularizing this vector field during training makes the network more regular on the data distribution's support, thus making the network's activations on clean samples more distinguishable from those of abnormal samples. This regularization of the network's dynamics improves the performance of any detection method that uses the internal embeddings as inputs, while also improving the network's test accuracy.
LilNetX: Lightweight Networks with EXtreme Model Compression and Structured Sparsification
Authors: Sharath Girish; Kamal Gupta; Saurabh Singh; Abhinav Shrivastava
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: We introduce LilNetX, an end-to-end trainable technique for neural networks that enables learning models with specified accuracy-rate-computation trade-off. Prior works approach these problems one at a time and often require post-processing or multistage training which become less practical and do not scale very well for large datasets or architectures. Our method constructs a joint training objective that penalizes the self information of network parameters in a latent representation space to encourage small model size while also introducing priors to increase structured sparsity in the parameter space to reduce computation. When compared with existing state-of-the-art model compression methods, we achieve up to 50% smaller model size and 98% model sparsity on ResNet-20 on the CIFAR-10 dataset as well as 37% smaller model size and 71% structured sparsity on ResNet-50 trained on ImageNet while retaining the same accuracy as those methods. We show that the resulting sparsity can improve the inference time of the models by almost 1.8 times the dense ResNet-50 baseline model. Code is available at https://github.com/Sharath-girish/LilNetX.
Simple Emergent Action Representations from Multi-Task Policy Training
Authors: Pu Hua; Yubei Chen; Huazhe Xu
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: The low-level sensory and motor signals in deep reinforcement learning, which exist in high-dimensional spaces such as image observations or motor torques, are inherently challenging to understand or utilize directly for downstream tasks. While sensory representations have been extensively studied, the representations of motor actions are still an area of active exploration. Our work reveals that a space containing meaningful action representations emerges when a multi-task policy network takes as inputs both states and task embeddings. Moderate constraints are added to improve its representation ability. Therefore, interpolated or composed embeddings can function as a high-level interface within this space, providing instructions to the agent for executing meaningful action sequences. Empirical results demonstrate that the proposed action representations are effective for intra-action interpolation and inter-action composition with limited or no additional learning. Furthermore, our approach exhibits superior task adaptation ability compared to strong baselines in Mujoco locomotion tasks. Our work sheds light on the promising direction of learning action representations for efficient, adaptable, and composable RL, forming the basis of abstract action planning and the understanding of motor signal space. Project page: https://sites.google.com/view/emergent-action-representation/
Consistent Data Distribution Sampling for Large-scale Retrieval
Authors: Hongyu Ou; Jun Yin; Huanqin Wu; ANAN LIU; Lin Zhao; Tao Chen; Yuekui Yang; TAO YANG
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Retrieving candidate items with low latency and computational cost is important for large-scale advertising systems. Negative sampling is a general approach to model million-scale items with rich features in the retrieval. The training-inference inconsistency of data distribution brought from sampling negatives is a key challenge. In this work, we propose a novel negative sampling strategy Consistent Data Distribution Sampling (CDDS) to solve such an issue. Specifically, we employ a relative large-scale of uniform training negatives and batch negatives to adequately train long-tail and hot items respectively, and employ high divergence negatives to improve the learning convergence. To make the above training samples approximate the serving item data distribution, we introduce an auxiliary loss based on an asynchronous item embedding matrix over the entire item pool. Offline experiments on real datasets achieve SOTA performance. Online experiments with multiple advertising scenarios show that our method has achieved significant increases in GMV. The source code will be released in the future.
Determinant regularization for Deep Metric Learning
Authors: Kun Song; Ruben Solozabal; Martin Takáč; Fakhri Karray
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Distance Metric Learning (DML) aims to learn the distance metric that better reflects the semantical similarities in the data. Current \textit{pair-based} and \textit{proxy-based} methods on DML focus on reducing the distance between similar samples while expanding the distance of dissimilar ones. However, we reveal that shrinking the distance between similar samples may distort the feature space, increasing the distance between points of the same class region and, therefore, harming the generalization of the model. Traditional regularization terms (such as $L_2$-norm on weights) cannot be adopted to solve this issue as they are based on linear projection. To alleviate this issue, we adopt the structure of normalizing flow as the deep metric layer and calculate the determinant of the Jacobi Matrix as the regularization term. At last, we conduct experiments on several \textit{pair-based} and \textit{proxy-based} algorithms that demonstrate the benefits of our method.
Substructured Graph Convolution for Non-overlapping Graph Decomposition
Authors: Youngkyu Lee; Chang-Ock Lee
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Graph convolutional networks have been widely used to solve the graph problems such as node classification, link prediction, and recommender systems. It is well known that large graphs require large amount of memory and time to train graph convolutional networks. To deal with large graphs, many methods are being done, such as graph sampling or decomposition. In particular, graph decomposition has the advantage of parallel computation, but information loss occurs in the interface part. In this paper, we propose a novel substructured graph convolution that reinforces the interface part lost by graph decomposition. Numerical results indicate that the proposed method is robust in the number of subgraphs compared to other methods.
Co-Evolution As More Than a Scalable Alternative for Multi-Agent Reinforcement Learning
Authors: Patrick Grzywok
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: In recent years, gradient based multi-agent reinforcement learning is growing in success. One contributing factor is the use of shared parameters for learning policy networks. While this approach scales well with the number of agents during execution it lacks this ambiguity for training as the number of produced samples grows linearly with the number of agents. For a very large number of agents, this could lead to an inefficient use of the circumstantial amount of produced samples. Moreover in single-agent reinforcement learning policy search with evolutionary algorithms showed viable success when sampling can be parallelized on a larger scale. The here proposed method does not only consider sampling in concurrent environments but further investigates sampling diverse parameters from the population in co-evolution in joint environments during training. This co-evolutionary policy search has shown to be capable of training a large number of agents. Beyond that, it has been shown to produce competitive results in smaller environments in comparison to gradient descent based methods. This surprising result make evolutionary algorithms a promising candidate for further research in the context of multi-agent reinforcement learning.
Offline Reinforcement Learning with Differential Privacy
Authors: Dan Qiao; Yu-Xiang Wang
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: The offline reinforcement learning (RL) problem is often motivated by the need to learn data-driven decision policies in financial, legal and healthcare applications.  However, the learned policy could retain sensitive information of individuals in the training data (e.g., treatment and outcome of patients), thus susceptible to various privacy risks. We design offline RL algorithms with differential privacy guarantees which provably prevent such risks. These algorithms also enjoy strong instance-dependent learning bounds under both tabular and linear Markov Decision Process (MDP) settings. Our theory and simulation suggest that the privacy guarantee comes at (almost) no drop in utility comparing to the non-private counterpart for a medium-size dataset.