GenAI4ELab/papercli-papers huggingface dataset:
DIGEST: FAST AND COMMUNICATION EFFICIENT DECENTRALIZED LEARNING WITH LOCAL UPDATES
Authors: Peyman Gholami; Hulya Seferoglu
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Decentralized learning advocates the elimination of centralized parameter servers
(aggregation points) for potentially better utilization of underlying resources, de-
lay reduction, and resiliency against parameter server unavailability and catas-
trophic failures. Gossip based decentralized algorithms, where each node in a net-
work has its own locally kept model on which it effectuates the learning by talking
to its neighbors, received a lot of attention recently. Despite their potential, Gossip
algorithms introduce huge communication costs. In this work, we show that nodes
do not need to communicate as frequently as in Gossip for fast convergence; in
fact, a sporadic exchange of a digest of a trained model is sufficient. Thus, we
design a fast and communication-efficient decentralized learning mechanism; DI-
GEST by particularly focusing on stochastic gradient descent (SGD). DIGEST is
a decentralized algorithm building on local-SGD algorithms, which are originally
designed for communication efficient centralized learning. We show through anal-
ysis and experiments that DIGEST significantly reduces the communication cost
without hurting convergence time for both iid and non-iid data.
Differentiable Gaussianization Layers for Inverse Problems Regularized by Deep Generative Models
Authors: Dongzhuo Li
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: Deep generative models such as GANs, normalizing flows, and diffusion models are powerful regularizers for inverse problems. They exhibit great potential for helping reduce ill-posedness and attain high-quality results. However, the latent tensors of such deep generative models can fall out of the desired high-dimensional standard Gaussian distribution during inversion, particularly in the presence of data noise and inaccurate forward models, leading to low-fidelity solutions. To address this issue, we propose to reparameterize and Gaussianize the latent tensors using novel differentiable data-dependent layers wherein custom operators are defined by solving optimization problems. These proposed layers constrain inverse problems to obtain high-fidelity in-distribution solutions. We validate our technique on three inversion tasks: compressive-sensing MRI, image deblurring, and eikonal tomography (a nonlinear PDE-constrained inverse problem) using two representative deep generative models: StyleGAN2 and Glow. Our approach achieves state-of-the-art performance in terms of accuracy and consistency.
SAGE: Semantic-Aware Global Explanations for Named Entity Recognition
Authors: Andrea Zugarini; Leonardo Rigutini
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: In the last decades, deep learning approaches achieved impressive results in many research fields, 
such as Computer Vision and Natural Language Processing (NLP). 
NLP in particular has greatly benefit from unsupervised methods that allow to learn distributed representation of language.
On the race for better performances Language Models have reached hundred of billions parameters nowadays. 
Despite the remarkable results, deep models are still far from being fully exploited in real world applications. 
Indeed, these approaches are black-boxes, i.e. they are not interpretable by design nor explainable, which is often crucial to make decisions in business. 
Several task-agnostic methods have been proposed in literature to explain models' decisions. 
Most techniques rely on the "local" assumption, i.e. explanations are made example-wise.
In this paper instead, we present a post-hoc method to produce highly interpretable global rules to explain NLP classifiers. 
Rules are extracted with a data mining approach on a semantically enriched input representation, instead of using words/wordpieces solely. 
Semantic information yields more abstract and general rules that are both more explanatory and less complex, while being also better at reflecting the model behaviour.
In the experiments we focus on Named Entity Recognition, an NLP task where explainability is under-investigated. 
We explain the predictions of BERT NER classifiers trained on two popular benchmarks, CoNLL03 and Ontonotes, and compare our model against LIME.
Learning Implicit Scale Conditioned Memory Compensation for Talking Head Generation
Authors: Fa-Ting Hong; Dan Xu
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Talking head video generation aims to animate the pose and expression of a person in a target driving video using motion information contained in the video, while maintaining a person's identity in a given still source image. Highly dynamic and complex motions in the driving video cause ambiguous generation from the source image, because the still source image cannot provide sufficient appearance information for occluded regions or delicate expressions, which severely produces artifacts and significantly degrades the generation quality. However, existing works mainly focus on learning more accurate motion estimation and representation in 2D and 3D, and they ignore the facial structural prior in addressing the facial ambiguities. Therefore, effective handling of the ambiguities in the dramatic appearance changes of the source to largely improve facial details and completeness in generation still remains barely explored. To this end, we propose a novel implicit scale conditioned memory compensation network (MCNet) for high-fidelity talking head generation. Specifically, considering human faces are symmetric and structured, we aim to automatically learn a representative global facial memory bank from all training data as a prior to compensate for the facial generation features. Each face in the source image contains a scale that can be reflected in detected facial keypoints. To better query the learned global memory, we further propose to learn implicit scale representations from the discrete keypoints, which can be used to condition on the query of the global memory, to obtain scale-aware memory for the feature compensation. Extensive experiments from quantitative and qualitative perspectives demonstrate that MCNet can learn representative and complementary facial memory, and can clearly outperform previous state-of-the-art methods on VoxCeleb1 and CelebV datasets.
Temporal Relevance Analysis for Video Action Models
Authors: Quanfu Fan; Donghyun Kim; Chun-Fu Chen; Stan Sclaroff; Kate Saenko; Sarah Adel Bargal
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: In this paper, we provide a deep analysis of temporal modeling for action recognition, an important but underexplored problem in
the literature. We first propose a new approach to quantify the temporal relationships between frames captured by CNN-based action models based on layer-wise relevance propagation. We then conduct comprehensive experiments and in-depth analysis to provide a better understanding of how temporal modeling is affected by various factors such as dataset, network architecture, and input frames. With this, we further study some important questions for action recognition that lead to interesting findings. Our analysis shows that there is no strong correlation between temporal relevance and model performance; and action models tend to capture local temporal information, but less long-range dependencies.
Global Explainability of GNNs via Logic Combination of Learned Concepts
Authors: Steve Azzolin; Antonio Longa; Pietro Barbiero; Pietro Lio; Andrea Passerini
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: While instance-level explanation of GNN is a well-studied problem with plenty of approaches being developed, providing a global explanation for the behaviour of a GNN is much less explored, despite its potential in interpretability and debugging. Existing solutions either simply list local explanations for a given class, or generate a synthetic prototypical graph with maximal score for a given class, completely missing any combinatorial aspect that the GNN could have learned.
In this work, we propose GLGExplainer (Global Logic-based GNN Explainer), the first Global Explainer capable of generating explanations as arbitrary Boolean combinations of learned graphical concepts. GLGExplainer is a fully differentiable architecture that takes local explanations as inputs and combines them into a logic formula over graphical concepts, represented as clusters of local explanations. 
Contrary to existing solutions, GLGExplainer provides accurate and human-interpretable global explanations that are perfectly aligned with ground-truth explanations (on synthetic data) or match existing domain knowledge (on real-world data). Extracted formulas are faithful to the model predictions, to the point of providing insights into some occasionally incorrect rules learned by the model, making GLGExplainer a promising diagnostic tool for learned GNNs.
Generative Multi-Flow Networks: Centralized, Independent and Conservation
Authors: Yinchuan Li; Haozhi Wang; Shuang Luo; yunfeng shao; Jianye HAO
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Generative flow networks utilize the flow matching loss to learn a stochastic policy for generating objects from a sequence of actions, such that the probability of generating a pattern can be proportional to the corresponding given reward. However, existing works can only handle single flow model tasks and cannot directly generalize to multi-agent flow networks due to limitations such as flow estimation complexity and independent sampling. In this paper, we propose the framework of generative multi-flow networks (GMFlowNets) that can be applied to multiple agents to generate objects collaboratively through a series of joint actions. Then, the centralized flow network algorithm is proposed for centralized training GMFlowNets, while the independent flow network algorithm is proposed to achieve decentralized execution of GMFlowNets. Based on the independent global conservation condition, the flow conservation network algorithm is then proposed to realize centralized training with decentralized execution paradigm. Theoretical analysis proves that using the multi-flow matching loss function can train a unique Markovian flow, and the flow conservation network can ensure independent policies can generate samples with probability proportional to the reward function. Experimental results demonstrate the performance superiority of the proposed algorithms compared to reinforcement learning and MCMC-based methods.
Rethinking Backdoor Data Poisoning Attacks in the Context of Semi-Supervised Learning
Authors: Marissa Catherine Connor; Vincent Emanuele
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Semi-supervised learning methods can train high-accuracy machine learning models with a fraction of the labeled training samples required for traditional supervised learning. Such methods do not typically involve close review of the unlabeled training samples, making them tempting targets for data poisoning attacks. In this paper we investigate the vulnerabilities of semi-supervised learning methods to backdoor data poisoning attacks on the unlabeled samples. We show that a simple poisoning attack using adversarially perturbed samples is highly effective - achieving an average attack success rate of 93.6%. We introduce a generalized attack framework targeting semi-supervised learning methods to better understand and exploit their limitations and to motivate future defense strategies.
HRBP: Hardware-friendly Regrouping towards Block-wise Pruning for Sparse Training
Authors: Haoyu Ma; Chengming Zhang; lizhi xiang; Xiaolong Ma; Geng Yuan; Wenkai Zhang; Shiwei Liu; Tianlong Chen; Dingwen Tao; Yanzhi Wang; Zhangyang Wang; Xiaohui Xie
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Recently, pruning at initialization and training a sparse network from scratch (sparse training) become increasingly popular. However, most sparse training literature addresses only the unstructured sparsity, which in practice brings little benefit to the training acceleration on GPU due to the irregularity of non-zero weights. In this paper, we work on sparse training with fine-grained structured sparsity, by extracting a few dense blocks from unstructured sparse weights. For Convolutional Neural networks (CNN), however, the extracted dense blocks will be broken in backpropagation due to the shape transformation of convolution filters implemented by GEMM. Thus, previous block-wise pruning methods can only be used to accelerate the forward pass of sparse CNN training. To this end, we propose the Hardware-friendly Regrouping towards Block-based Pruning (HRBP), where the grouping is conducted on the kernel-wise mask. With HRBP, extracted dense blocks are preserved in backpropagation. We further propose HRBP++ to reduce zero kernels by extracting common sparse kernel patterns on all kernels within one block. Extensive experiments on CIFAR-10, CIFAR-100, and ImageNet demonstrate that HRBP (HRBP++) can almost match the accuracy of unstructured sparse training methods while achieving a huge acceleration on hardware.
Leveraging Unlabeled Data to Track Memorization
Authors: Mahsa Forouzesh; Hanie Sedghi; Patrick Thiran
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: Deep neural networks may easily memorize noisy labels present in real-world data, which degrades their ability to generalize. It is therefore important to track and evaluate the robustness of models against noisy label memorization. We propose a metric, called $\textit{susceptibility}$, to gauge such memorization for neural networks. Susceptibility is simple and easy to compute during training. Moreover, it does not require access to ground-truth labels and it only uses unlabeled data. We empirically show the effectiveness of our metric in tracking memorization on various architectures and datasets and provide theoretical insights into the design of the susceptibility metric. Finally, we show through extensive experiments on datasets with synthetic and real-world label noise that one can utilize susceptibility and the overall training accuracy to distinguish models that maintain a low memorization on the training set and generalize well to unseen clean data.
Rethinking Symbolic Regression: Morphology and Adaptability in the Context of Evolutionary Algorithms
Authors: Kei Sen Fong; Shelvia Wongso; Mehul Motani
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: Symbolic Regression (SR) is the well-studied problem of finding closed-form analytical expressions that describe the relationship between variables in a measurement dataset. In this paper, we rethink SR from two perspectives: morphology and adaptability. Morphology: Current SR algorithms typically use several man-made heuristics to influence the morphology (or structure) of the expressions in the search space. These man-made heuristics may introduce unintentional bias and data leakage, especially with the relatively few equation-recovery benchmark problems available for evaluating SR approaches. To address this, we formulate a novel minimalistic approach, based on constructing a depth-aware mathematical language model trained on terminal walks of expression trees, as a replacement to these heuristics. Adaptability: Current SR algorithms tend to select expressions based on only a single fitness function (e.g., MSE on the training set). We promote the use of an adaptability framework in evolutionary SR which uses fitness functions that alternate across generations. This leads to robust expressions that perform well on the training set and are close to the true functional form. We demonstrate this by alternating fitness functions that quantify faithfulness to values (via MSE) and empirical derivatives (via a novel theoretically justified fitness metric coined MSEDI). Proof-of-concept: We combine these ideas into a minimalistic evolutionary SR algorithm that outperforms all benchmark and state of-the-art SR algorithms in problems with unknown constants added, which we claim are more reflective of SR performance for real-world applications. Our claim is then strengthened by reproducing the superior performance on real-world regression datasets from SRBench. For researchers interested in equation-recovery problems, we also propose a set of conventions that can be used to promote fairness in comparison across SR methods and to reduce unintentional bias.
Quantized Compressed Sensing with Score-Based Generative Models
Authors: Xiangming Meng; Yoshiyuki Kabashima
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: We consider the general problem of recovering a high-dimensional signal from noisy quantized measurements. Quantization, especially coarse quantization such as 1-bit sign measurements, leads to severe information loss and thus a good prior knowledge of the unknown signal is helpful for accurate recovery. Motivated by the power of score-based generative models (SGM, also known as diffusion models) in capturing the rich structure of natural signals beyond simple sparsity, we propose an unsupervised data-driven approach called quantized compressed sensing with SGM (QCS-SGM), where the prior distribution is modeled by a pre-trained SGM. To perform posterior sampling, an annealed pseudo-likelihood score called ${\textit{noise perturbed pseudo-likelihood score}}$ is introduced and combined with the prior score of SGM. The proposed QCS-SGM applies to an arbitrary number of quantization bits. Experiments on a variety of baseline datasets demonstrate that the proposed QCS-SGM significantly outperforms existing state-of-the-art algorithms by a large margin for both in-distribution and out-of-distribution samples. Moreover, as a posterior sampling method, QCS-SGM can be easily used to obtain confidence intervals or uncertainty estimates of the reconstructed results. $\textit{The code  is available at}$ https://github.com/mengxiangming/QCS-SGM.
Towards Federated Learning of Deep Graph Neural Networks
Authors: Zhihua Tian; Yuan Ding; Rui Zhang; Jian Liu; Kui Ren
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Graph neural networks (GNNs) learn node representations by recursively aggregating neighborhood information on graph data. However, in the federated setting, data samples (nodes) located in different clients may be connected to each other, leading to huge information loss to the training method.
Existing federated graph learning frameworks solve such a problem by generating missing neighbors or sending information across clients directly. None are suitable for training deep GNNs, which require a more expansive receptive field and higher communication costs.
In this work, we introduce a novel framework named  $Fed^2GNN$ for federated graph learning of deep GNNs via reconstructing neighborhood information of nodes. Specifically, we design a graph structure named rooted tree. The node embedding obtained by encoding on the rooted tree is the same as that obtained by encoding on the induced subgraph surrounding the node, which allows us to reconstruct the neighborhood information by building the rooted tree of the node. An encoder-decoder framework is then proposed, wherein we first encode missing neighbor information and then decode it to build the rooted tree.
Extensive experiments on real-world network datasets show the effectiveness of our framework for training deep GNNs while also achieving better performance for training shadow GNN models
ILA-DA: Improving Transferability of Intermediate Level Attack with Data Augmentation
Authors: Chiu Wai Yan; Tsz-Him Cheung; Dit-Yan Yeung
Venue: ICLR 2023 (ICLR 2023 poster)

Abstract: Adversarial attack aims to generate deceptive inputs to fool a machine learning model. In deep learning, an adversarial input created for a specific neural network can also trick other neural networks. This intriguing property is known as black-box transferability of adversarial examples. To improve black-box transferability, a previously proposed method called Intermediate Level Attack (ILA) fine-tunes an adversarial example by maximizing its perturbation on an intermediate layer of the source model. Meanwhile, it has been shown that simple image transformations can also enhance attack transferability. Based on these two observations, we propose ILA-DA, which employs three novel augmentation techniques to enhance ILA. Specifically, we propose (1) an automated way to apply effective image transformations, (2) an efficient reverse adversarial update technique, and (3) an attack interpolation method to create more transferable adversarial examples. Shown by extensive experiments, ILA-DA greatly outperforms ILA and other state-of-the-art attacks by a large margin. On ImageNet, we attain an average attack success rate of 84.5%, which is 19.5% better than ILA and 4.7% better than the previous state-of-the-art across nine undefended models. For defended models, ILA-DA also leads existing attacks and provides further gains when incorporated into more advanced attack methods.
Evaluating Fairness Without Sensitive Attributes: A Framework Using Only Auxiliary Models
Authors: Zhaowei Zhu; Yuanshun Yao; Jiankai Sun; Yang Liu; Hang Li
Venue: ICLR 2023 (Submitted to ICLR 2023)

Abstract: Although the volume of literature and public attention on machine learning fairness has been growing significantly in recent years, in practice some tasks as basic as measuring fairness, which is the first step in studying and promoting fairness, can be challenging. This is because the sensitive attributes are often unavailable in a machine learning system due to privacy regulations. The straightforward solution is to use auxiliary models to predict the missing sensitive attributes. However, our theoretical analyses show that the estimation error of the directly measured fairness metrics is proportional to the error rates of auxiliary models' predictions. Existing works that attempt to reduce the estimation error often require strong assumptions, e.g. access to the ground-truth sensitive attributes in a subset of samples, auxiliary models' training data and the target data are i.i.d, or some form of conditional independence. In this paper, we drop those assumptions and propose a framework that uses only off-the-shelf auxiliary models. The main challenge is how to reduce the negative impact of imperfectly predicted sensitive attributes on the fairness metrics without knowing the ground-truth sensitive attribute values. Inspired by the noisy label learning literature, we first derive a closed-form relationship between the directly measured fairness metrics and their corresponding ground-truth metrics. And then we estimate some key statistics (most importantly transition matrix in the noisy label literature), which we use, together with the derived relationship, to calibrate the fairness metrics. Our framework can be applied to all popular group fairness definitions as well as multi-class classifiers and multi-category sensitive attributes. In addition, we theoretically prove the upper bound of the estimation error in our calibrated metrics and show our method can substantially decrease the estimation error especially when auxiliary models are inaccurate or the target model is highly biased. Experiments on COMPAS and CelebA validate our theoretical analyses and show our method can measure fairness significantly more accurately than baselines under favorable circumstances.