GenAI4ELab/papercli-papers huggingface dataset:
AudioGen: Textually Guided Audio Generation
Authors: Felix Kreuk; Gabriel Synnaeve; Adam Polyak; Uriel Singer; Alexandre Défossez; Jade Copet; Devi Parikh; Yaniv Taigman; Yossi Adi
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: In this work, we tackle the problem of generating audio samples conditioned on descriptive text captions. We propose AudioGen, an auto-regressive generative model, operating on a learnt discrete audio representation, that generates audio samples conditioned on text inputs. The task of text-to-audio generation poses multiple challenges. Due to the way audio travels through a medium, differentiating ``objects'' can be a difficult task (e.g., separating multiple people simultaneously speaking). This is further complicated by real-world recording conditions (e.g., background noise, reverberation, etc.). Scarce text annotations impose another constraint, limiting the ability to scale models. Finally, modeling high fidelity audio requires one to operate over extremely long sequences. To alleviate the aforementioned challenges we propose an augmentation technique that mixes different audio samples, driving the model to internally learn to separate multiple sources. We curated 10 datasets containing different types of audio and text annotations to handle the scarcity of text-audio data points. For faster inference, we explore the use of multi-stream modeling, allowing the use of shorter sequences while maintaining a similar bitrate and perceptual quality. Finally, we apply classifier-free guidance to improve adherence to text. Comparing to the evaluated baselines, AudioGen outperforms over both objective and subjective metrics. We further conduct an ablation study to gauge the effects of pre-trained text and audio components.
MPCFORMER: FAST, PERFORMANT AND PRIVATE TRANSFORMER INFERENCE WITH MPC
Authors: Dacheng Li; Hongyi Wang; Rulin Shao; Han Guo; Eric Xing; Hao Zhang
Venue: ICLR 2023 (ICLR 2023 notable top 25%)

Abstract: Enabling private inference is crucial for many cloud inference services that are based on Transformer models. However, existing private inference solutions can increase the inference latency by more than 60$\times$ or significantly compromise the inference quality. In this paper, we design the framework MPCFORMER as a practical solution, using Secure Multi-Party Computation (MPC) and Knowledge Distillation (KD). Through extensive evaluations, we show that MPCFORMER significantly speeds up Transformer inference in MPC settings while achieving similar ML performance to the input model. On the IMDb dataset, it achieves similar performance to $\text{BERT}_\text{BASE}$, while being 5.3$\times$ faster. On the GLUE benchmark, it achieves 97% performance of $\text{BERT}_\text{BASE}$ with a 2.2$\times$ speedup. MPCFORMER remains effective with different trained Transformer weights such as $\text{ROBERTA}_\text{BASE}$ and larger models including $\text{BERT}_\text{LARGE}$. Code is available at https://github.com/MccRee177/MPCFormer.
DAVA: Disentangling Adversarial Variational Autoencoder
Authors: Benjamin Estermann; Roger Wattenhofer
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: The use of well-disentangled representations offers many advantages for downstream tasks, e.g. an increased sample efficiency, or better interpretability.
However, the quality of disentangled interpretations is often highly dependent on the choice of dataset-specific hyperparameters, in particular the regularization strength.
To address this issue, we introduce DAVA, a novel training procedure for variational auto-encoders. DAVA completely alleviates the problem of hyperparameter selection.
We compare DAVA to models with optimal hyperparameters.
Without any hyperparameter tuning, DAVA is competitive on a diverse range of commonly used datasets.
Underlying DAVA, we discover a necessary condition for unsupervised disentanglement, which we call PIPE.
We demonstrate the ability of PIPE to positively predict the performance of downstream models in abstract reasoning.
We also thoroughly investigate correlations with existing supervised and unsupervised metrics. The code is available at https://github.com/besterma/dava.
Progressive Mixup Augmented Teacher-Student Learning for Unsupervised Domain Adaptation
Authors: Aotian Zheng; Jie Mei; Farron Wallace; Craig Rose; Rania Hussein; Jenq-Neng Hwang
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Unsupervised Domain Adaptation (UDA) aims to transfer knowledge learned from a labeled source domain to an unlabeled target domain, mostly through learning a domain invariant feature representation.  Currently, the best performing UDA methods use category level domain alignment to capture fine-grained information, resulting in significantly improved performance over global alignment.  While successful, category level UDA methods suffer from the unreliable pseudo-labels for target data.  Additionally, most UDA methods directly adapt from source to target domain without regard for the large domain discrepancy.  In this paper, we propose an UDA approach with teacher-student learning where the teacher network is used to provide more reliable target pseudo-labels for the student network to train with. Furthermore, we use a progressive mixup augmentation strategy which generates intermediate samples that become increasingly target-dominant as training progresses.  Aligning the source and intermediate domains allows the model to gradually transfer fine-grained domain knowledge from the source to the target domain while minimizing the negative impact of noisy target pseudo-labels.  This progressive mixup augmented teacher-student (PMATS) training strategy along with class subset sampling and clustering based pseudo-label refinement achieves state-of-the-art performance on two public UDA benchmark datasets: Office-31, and Office-Home.
Hierarchical Sliced Wasserstein Distance
Authors: Khai Nguyen; Tongzheng Ren; Huy Nguyen; Litu Rout; Tan Minh Nguyen; Nhat Ho
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: Sliced Wasserstein (SW) distance has been widely used in different application scenarios since it can be scaled to a large number of supports without suffering from the curse of dimensionality. The value of sliced Wasserstein distance is the average of transportation cost between one-dimensional representations (projections) of original measures that are obtained by Radon Transform (RT). Despite its efficiency in the number of supports, estimating the sliced Wasserstein requires a relatively large number of projections in high-dimensional settings. Therefore, for applications where the number of supports is relatively small compared with the dimension, e.g., several deep learning applications where the mini-batch approaches are utilized, the complexities from matrix multiplication of Radon Transform become the main computational bottleneck. To address this issue, we propose to derive projections by linearly and randomly combining a smaller number of projections which are named bottleneck projections. We explain the usage of these projections by introducing Hierarchical Radon Transform (HRT) which is constructed by applying  Radon Transform variants recursively. We then formulate the approach into a new metric between measures, named Hierarchical Sliced Wasserstein (HSW) distance. By proving the injectivity of HRT, we derive the metricity of HSW. Moreover, we investigate the theoretical properties of HSW including its connection to SW variants and its computational and sample complexities. Finally, we compare the computational cost and generative quality of HSW with the conventional SW on the task of deep generative modeling using various benchmark datasets including CIFAR10, CelebA, and Tiny ImageNet.
Prefer to Classify: Improving Text Classifier via Pair-wise Preference Learning
Authors: Jaehyung Kim; Jinwoo Shin; Dongyeop Kang
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: The development of largely human-annotated benchmarks has driven the success of deep neural networks in various NLP tasks. These benchmarks are collected by aggregating decisions made by different annotators on the target task. Aggregating the annotated decisions via majority is still used as a common practice, despite its inevitable limitation from simple aggregation. In this paper, we establish a novel classification framework, based on task-specific human preference between a pair of samples, which provides an informative training signal to capture fine-grained and complementary task information through pair-wise comparison. Hence, it improves the existing instance-wise annotation system by enabling better task modeling from learning the relation between samples. Specifically, we propose a new multi-task learning framework, called prefer-to-classify (P2C), to effectively learn human preferences in addition to the given classification task.
We collect human preference signals in two ways: (1) extracting relative preferences implicitly from annotation records (for free) or (2) collecting subjective preferences explicitly from (paid) crowd workers. In various text classification tasks, we demonstrate that both extractive and subjective preferences are effective in improving the classifier with our preference learning framework. Interestingly, we found that subjective preference shows more significant improvements than extractive preference, revealing the effectiveness of explicit modeling of human preferences. Our code and preference dataset will be publicly available upon acceptance.
Federated Learning for Inference at Anytime and Anywhere
Authors: Zicheng Liu; Da Li; Javier Fernandez-Marques; Stefanos Laskaridis; Yan Gao; Łukasz Dudziak; Stan Z. Li; Shell Xu Hu; Timothy Hospedales
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Federated learning has been predominantly concerned with collaborative training of deep networks from scratch, and especially the many challenges that arise, such as communication cost, robustness to heterogeneous data, and support for diverse device capabilities. However, there is no unified framework that addresses all these problems together. This paper studies the challenges and opportunities of exploiting pre-trained Transformer models in FL. In particular, we propose to efficiently adapt such pre-trained models by injecting a novel attention-based adapter module at each transformer block that both modulates the forward pass and makes an early prediction. Training only the lightweight adapter by FL leads to fast and communication-efficient learning even in the presence of heterogeneous data and devices. Extensive experiments on standard FL benchmarks, including CIFAR- 100, FEMNIST and SpeechCommandsv2 demonstrate that this simple framework provides fast and accurate FL while supporting heterogenous device capabilities, efficient personalization, and scalable-cost anytime inference.
Copy is All You Need
Authors: Tian Lan; Deng Cai; Yan Wang; Heyan Huang; Xian-Ling Mao
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: The dominant text generation models compose the output by sequentially selecting words from a fixed vocabulary. In this paper, we formulate text generation as progressively copying text segments (e.g., words or phrases) from an existing text collection. We compute the contextualized representations of meaningful text segments and index them using efficient vector search toolkits. The task of text generation is then decomposed into a series of copy-and-paste operations: at each time step, we seek suitable text spans from the text collection rather than selecting from a standalone vocabulary. Experiments on the standard language modeling benchmark (WikiText-103) show that our approach achieves better generation quality according to both automatic and human evaluations. Besides, its inference efficiency is comparable to token-level autoregressive models thanks to the reduction of decoding steps. We also show that our approach allows for effective domain adaptation by simply switching to domain-specific text collection without extra training. Finally, we observe that our approach attains additional performance gains by simply scaling up to larger text collections, again without further training.\footnote{Our source codes are publicly available at \url{https://github.com/gmftbyGMFTBY/Copyisallyouneed}.}
NORM: Knowledge Distillation via N-to-One Representation Matching
Authors: Xiaolong Liu; LUKING LI; Chao Li; Anbang Yao
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: Existing feature distillation methods commonly adopt the One-to-one Representation Matching between any pre-selected teacher-student layer pair. In this paper, we present $N$-to-$O$ne $R$epresentation $M$atching (NORM), a new two-stage knowledge distillation method, which relies on a simpleFeature Transform (FT) module consisting of two linear layers. In view of preserving the intact information learnt by the teacher network, during training, our FT module is merely inserted after the last convolutional layer of the student network. The first linear layer projects the student representation to a feature space having $N$ times feature channels than the teacher representation from the last convolutional layer, and the second linear layer contracts the expanded output back to the original feature space. By sequentially splitting the expanded student representation into $N$ non-overlapping feature segments having the same number of feature channels as the teacher's, they can be readily forced to approximate the intact teacher representation simultaneously, formulating a novel many-to-one representation matching mechanism conditioned on a single teacher-student layer pair. After training, such an FT module will be naturally merged into the subsequent fully connected layer thanks to its linear property, introducing no extra parameters or architectural modifications to the student network at inference. Extensive experiments on different visual recognition benchmarks demonstrate the leading performance of our method. For instance, the ResNet18|MobileNet|ResNet50-1/4 model trained by NORM reaches 72.14%|74.26%|68.03% top-1 accuracy on the ImageNet dataset when using a pre-trained ResNet34|ResNet50|ResNet50 model as the teacher, achieving an absolute improvement of 2.01%|4.63%|3.03% against the individually trained counterpart. Code is available at https://github.com/OSVAI/NORM.
Git Re-Basin: Merging Models modulo Permutation Symmetries
Authors: Samuel Ainsworth; Jonathan Hayase; Siddhartha Srinivasa
Venue: ICLR 2023 (ICLR 2023 notable top 5%)

Abstract: The success of deep learning is due in large part to our ability to solve certain massive non-convex optimization problems with relative ease. Though non-convex optimization is NP-hard, simple algorithms -- often variants of stochastic gradient descent -- exhibit surprising effectiveness in fitting large neural networks in practice. We argue that neural network loss landscapes often contain (nearly) a single basin after accounting for all possible permutation symmetries of hidden units a la Entezari et al. 2021. We introduce three algorithms to permute the units of one model to bring them into alignment with a reference model in order to merge the two models in weight space. This transformation produces a functionally equivalent set of weights that lie in an approximately convex basin near the reference model. Experimentally, we demonstrate the single basin phenomenon across a variety of model architectures and datasets, including the first (to our knowledge) demonstration of zero-barrier linear mode connectivity between independently trained ResNet models on CIFAR-10. Additionally, we identify intriguing phenomena relating model width and training time to mode connectivity. Finally, we discuss shortcomings of the linear mode connectivity hypothesis, including a counterexample to the single basin theory.
Efficient Large-scale Transformer Training via Random and Layerwise Token Dropping
Authors: Zhewei Yao; Xiaoxia Wu; Conglong Li; Connor Holmes; Minjia Zhang; Cheng Li; Yuxiong He
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Large-scale transformer models have become the de-facto architectures for various machine learning applications, e.g., CV and NLP. 
However, those large models also introduce prohibitive training costs. 
To mitigate this issue, we propose a novel random and layerwise token dropping method (\OURS), which skips the computation of a subset of the input tokens at all middle layers.
Particularly, \OURS achieves considerable speedups and comparable accuracy as the standard training baseline. 
Compared to other token dropping methods, \OURS does not require (1) any importance score-based metrics, (2) any   special token treatment (e.g., \texttt{[CLS]}), and (3) many layers in full sequence length training except the first and the last layers. 
Besides, a new \layertoken learning rate schedule is proposed for pretraining problems that resolve the heavy tuning requirement for our proposed training mechanism. 
Finally, we demonstrate that \OURS can be applied to broader applications, including \gpt and \bert pretraining as well as ViT and \gpt finetuning tasks. 
Our results show that \OURS can save about 33.3\% theoretical compute cost and 25.6\% wall-clock training time while achieving similar zero-shot evaluations on \gptb as compared to baseline.
Win: Weight-Decay-Integrated Nesterov Acceleration for Adaptive Gradient Algorithms
Authors: Pan Zhou; Xingyu Xie; Shuicheng YAN
Venue: ICLR 2023 (ICLR 2023 notable top 5%)

Abstract: Training deep  networks on  large-scale datasets is computationally challenging.  In this work, we explore the problem of ``\textit{how to  accelerate  adaptive gradient algorithms in a general manner}", and aim to provide practical efficiency-boosting insights.   To this end, we   propose an effective and general   {Weight-decay-Integrated Nesterov acceleration} (Win) to accelerate adaptive  algorithms. Taking AdamW and Adam as examples,  we minimize a dynamical   loss per iteration which combines the vanilla training loss and a dynamic regularizer inspired by proximal point method (PPM) to improve the convexity of the problem. To introduce Nesterov-alike-acceleration into AdamW and Adam,  we respectively  use the  first- and second-order Taylor approximations of vanilla loss  to  update the variable  twice. In this way,  we arrive at  our Win acceleration  for AdamW and Adam that uses  a conservative step  and a  reckless step to update twice and then linearly combines these two updates for acceleration. Next,  we  extend  Win acceleration to LAMB and SGD. Our transparent acceleration derivation  could  provide insights for  other accelerated methods and their integration into  adaptive algorithms.  Besides, we prove the convergence of Win-accelerated adaptive  algorithms and  justify their convergence superiority over their non-accelerated counterparts by taking AdamW and Adam as examples.  Experimental results testify to the faster convergence speed and superior performance of our Win-accelerated AdamW, Adam, LAMB and SGD over their non-accelerated counterparts on vision classification tasks and  language modeling tasks with both CNN and Transformer backbones.  We hope Win  shall be a default acceleration option for  popular optimizers in deep learning community to improve the training efficiency. Code will be released at \url{https://github.com/sail-sg/win}.
Proactive Multi-Camera Collaboration for 3D Human Pose Estimation
Authors: Hai Ci; Mickel Liu; Xuehai Pan; fangwei zhong; Yizhou Wang
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: This paper presents a multi-agent reinforcement learning (MARL) scheme for proactive Multi-Camera Collaboration in 3D Human Pose Estimation in dynamic human crowds. Traditional fixed-viewpoint multi-camera solutions for human motion capture (MoCap) are limited in capture space and susceptible to dynamic occlusions. Active camera approaches proactively control camera poses to find optimal viewpoints for 3D reconstruction. However, current methods still face challenges with credit assignment and environment dynamics. To address these issues, our proposed method introduces a novel Collaborative Triangulation Contribution Reward (CTCR) that improves convergence and alleviates multi-agent credit assignment issues resulting from using 3D reconstruction accuracy as the shared reward. Additionally, we jointly train our model with multiple world dynamics learning tasks to better capture environment dynamics and encourage anticipatory behaviors for occlusion avoidance. We evaluate our proposed method in four photo-realistic UE4 environments to ensure validity and generalizability. Empirical results show that our method outperforms fixed and active baselines in various scenarios with different numbers of cameras and humans.
On Representation Learning Under Class Imbalance
Authors: Ravid Shwartz-Ziv; Micah Goldblum; Yucen Lily Li; C. Bayan Bruss; Andrew Gordon Wilson
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Unlike carefully curated academic benchmarks, real-world datasets are often highly class-imbalanced, involving training and test sets which contain few examples from certain minority classes. While there is a common understanding that neural network generalization is negatively impacted by imbalance, the source of this problem and its resolution are unclear.  Through extensive empirical investigation, we study foundational learning behaviors for various models such as neural networks, gradient-boosted decision trees, and SVMs across a range of domains and find that (1) contrary to conventional wisdom, re-balancing the training set to include a higher proportion of minority samples degrades performance on imbalanced test sets; (2) minority samples are hard to fit, yet algorithms which fit them, such as oversampling, do not improve generalization.  Motivated by the observation that re-balancing class-imbalanced training data is ineffective, we show that several existing techniques for improving representation learning are effective in this setting: (3) self-supervised pre-training is insensitive to imbalance and can be used for feature learning before fine-tuning on labels; (4) Bayesian inference is effective because neural networks are especially underspecified under class imbalance; (5) flatness-seeking regularization pulls decision boundaries away from minority samples, especially when we seek minima that are particularly flat on the minority samples’ loss.