GenAI4ELab/papercli-papers huggingface dataset:
Particle-based Variational Inference with Generalized Wasserstein Gradient Flow
Authors: Ziheng Cheng; Shiyue Zhang; Longlin Yu; Cheng Zhang
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Particle-based variational inference methods (ParVIs) such as Stein variational gradient descent (SVGD) update the particles based on the kernelized Wasserstein gradient flow for the Kullback-Leibler (KL) divergence. However, the design of kernels is often non-trivial and can be restrictive for the flexibility of the method. Recent works show that functional gradient flow approximations with quadratic form regularization terms can improve performance. In this paper, we propose a ParVI framework, called generalized Wasserstein gradient descent (GWG), based on a generalized Wasserstein gradient flow of the KL divergence, which can be viewed as a functional gradient method with a broader class of regularizers induced by convex functions. We show that GWG exhibits strong convergence guarantees. We also provide an adaptive version that automatically chooses Wasserstein metric to accelerate convergence. In experiments, we demonstrate the effectiveness and efficiency of the proposed framework on both simulated and real data problems.
Soft-Unification in Deep Probabilistic Logic
Authors: Jaron Maene; Luc De Raedt
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: A fundamental challenge in neuro-symbolic AI is to devise primitives that fuse the logical and neural concepts. The Neural Theorem Prover has proposed the notion of soft-unification to turn the symbolic comparison between terms (i.e. unification) into a comparison in embedding space. It has been shown that soft-unification is a powerful mechanism that can be used to learn logic rules in an end-to-end differentiable manner. We study soft-unification from a conceptual point and outline several desirable properties of this operation. These include non-redundancy in the proof, well-defined proof scores, and non-sparse gradients. Unfortunately, these properties are not satisfied by previous systems such as the Neural Theorem Prover. Therefore, we introduce a more principled framework called DeepSoftLog based on probabilistic rather than fuzzy semantics. Our experiments demonstrate that DeepSoftLog can outperform the state-of-the-art on neuro-symbolic benchmarks, highlighting the benefits of these properties.
Robust covariance estimation with missing values and cell-wise contamination
Authors: gregoire pacreau; Karim Lounici
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Large datasets are often affected by cell-wise outliers in the form of missing or erroneous data. However, discarding any samples containing outliers may result in a dataset that is too small to accurately estimate the covariance matrix. Moreover, the robust procedures designed to address this problem require the invertibility of the covariance operator and thus are not effective on high-dimensional data. In this paper, we propose an unbiased estimator for the covariance in the presence of missing values that does not require any imputation step and still achieves near minimax statistical accuracy with the operator norm. We also advocate for its use in combination with cell-wise outlier detection methods to tackle cell-wise contamination in a high-dimensional and low-rank setting, where state-of-the-art methods may suffer from numerical instability and long computation times. To complement our theoretical findings, we conducted an experimental study which demonstrates the superiority of our approach over the state of the art both in low and high dimension settings.
Tree-Based Diffusion Schrödinger Bridge with Applications to Wasserstein Barycenters
Authors: Maxence Noble; Valentin De Bortoli; Arnaud Doucet; Alain Durmus
Venue: NeurIPS 2023 (NeurIPS 2023 spotlight)

Abstract: Multi-marginal Optimal Transport (mOT), a generalization of OT, aims at minimizing the integral of a cost function with respect to a distribution with some prescribed marginals. In this paper, we consider an entropic version of mOT
  with a tree-structured quadratic cost, i.e., a function that can be written as
  a sum of pairwise cost functions between the nodes of a tree. To address this
  problem, we develop Tree-based Diffusion Schr\"odinger Bridge (TreeDSB), an
  extension of the Diffusion Schr\"odinger Bridge (DSB) algorithm. TreeDSB
  corresponds to a dynamic and continuous state-space counterpart of the
  multimarginal Sinkhorn algorithm. A notable use case of our methodology is to
  compute Wasserstein barycenters which can be recast as the solution of a mOT
  problem on a star-shaped tree. We demonstrate that our methodology can be applied in high-dimensional settings such as image interpolation and
  Bayesian fusion.
Investigating how ReLU-networks encode symmetries
Authors: Georg Bökman; Fredrik Kahl
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Many data symmetries can be described in terms of group equivariance and
 the most common way of encoding group equivariances in neural networks is by building linear layers that are group equivariant.
In this work we investigate whether equivariance of a network implies that all layers are equivariant.
On the theoretical side we find cases where equivariance implies layerwise equivariance, but also
demonstrate that this is not the case generally.
Nevertheless, we conjecture that CNNs that are trained to be equivariant will exhibit layerwise equivariance and explain how this conjecture is a weaker version of the recent permutation conjecture by Entezari et al.\ [2022].
We perform quantitative experiments with VGG-nets on CIFAR10 and qualitative experiments with ResNets on ImageNet to illustrate and support our theoretical findings. 
These experiments are not only of interest for understanding how group equivariance is encoded in ReLU-networks, but they also give a new perspective on Entezari et al.'s permutation conjecture as we find that it
is typically easier to merge a network with a group-transformed version of itself than merging two different networks.
Enhancing Minority Classes by Mixing: An Adaptative Optimal Transport Approach for Long-tailed Classification
Authors: Jintong Gao; He Zhao; Zhuo Li; Dan dan Guo
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Real-world data usually confronts severe class-imbalance problems, where several majority classes have a significantly larger presence in the training set than minority classes. One effective solution is using mixup-based methods to generate synthetic samples to enhance the presence of minority classes. Previous approaches mix the background images from the majority classes and foreground images from the
minority classes in a random manner, which ignores the sample-level semantic similarity, possibly resulting in less reasonable or less useful images. In this work, we propose an adaptive image-mixing method based on optimal transport (OT) to incorporate both class-level and sample-level information, which is able to generate semantically reasonable and meaningful mixed images for minority classes. Due to
its flexibility, our method can be combined with existing long-tailed classification methods to enhance their performance and it can also serve as a general data augmentation method for balanced datasets. Extensive experiments indicate that our method achieves effective performance for long-tailed classification tasks. The code is available at https://github.com/JintongGao/Enhancing-Minority-Classes-by-Mixing.
Activity Grammars for Temporal Action Segmentation
Authors: Dayoung Gong; Joonseok Lee; Deunsol Jung; Suha Kwak; Minsu Cho
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Sequence prediction on temporal data requires the ability to understand compositional structures of multi-level semantics beyond individual and contextual properties of parts. The task of temporal action segmentation remains challenging for the reason, aiming at translating an untrimmed activity video into a sequence of action segments. 
This paper addresses the problem by introducing an effective activity grammar to guide neural predictions for temporal action segmentation.  
We propose a novel grammar induction algorithm, dubbed KARI, that extracts a powerful context-free grammar from action sequence data. We also develop an efficient generalized parser, dubbed BEP, that transforms frame-level probability distributions into a reliable sequence of actions according to the induced grammar with recursive rules. 
Our approach can be combined with any neural network for temporal action segmentation to enhance the sequence prediction and discover its compositional structure. 
Experimental results demonstrate that our method significantly improves temporal action segmentation in terms of both performance and interpretability on two standard benchmarks, Breakfast and 50 Salads.
GLOBER: Coherent Non-autoregressive Video Generation via GLOBal Guided Video DecodER
Authors: Mingzhen Sun; Weining Wang; Zihan Qin; Jiahui Sun; Sihan Chen; Jing Liu
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Video generation necessitates both global coherence and local realism. This work presents a novel non-autoregressive method GLOBER, which first generates global features to obtain comprehensive global guidance and then synthesizes video frames based on the global features to generate coherent videos. Specifically, we propose a video auto-encoder, where a video encoder encodes videos into global features, and a video decoder, built on a diffusion model, decodes the global features and synthesizes video frames in a non-autoregressive manner. To achieve maximum flexibility, our video decoder perceives temporal information through normalized frame indexes, which enables it to synthesize arbitrary sub video clips with predetermined starting and ending frame indexes. Moreover, a novel adversarial loss is introduced to improve the global coherence and local realism between the synthesized video frames. Finally, we employ a diffusion-based video generator to fit the global features outputted by the video encoder for video generation. Extensive experimental results demonstrate the effectiveness and efficiency of our proposed method, and new state-of-the-art results have been achieved on multiple benchmarks.
Generate What You Prefer: Reshaping Sequential Recommendation via Guided Diffusion
Authors: Zhengyi Yang; Jiancan Wu; Zhicai Wang; Xiang Wang; Yancheng Yuan; Xiangnan He
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Sequential recommendation aims to recommend the next item that matches a user’s
interest, based on the sequence of items he/she interacted with before. Scrutinizing
previous studies, we can summarize a common learning-to-classify paradigm—
given a positive item, a recommender model performs negative sampling to add
negative items and learns to classify whether the user prefers them or not, based on
his/her historical interaction sequence. Although effective, we reveal two inherent
limitations: (1) it may differ from human behavior in that a user could imagine
an oracle item in mind and select potential items matching the oracle; and (2)
the classification is limited in the candidate pool with noisy or easy supervision
from negative samples, which dilutes the preference signals towards the oracle
item. Yet, generating the oracle item from the historical interaction sequence is
mostly unexplored. To bridge the gap, we reshape sequential recommendation
as a learning-to-generate paradigm, which is achieved via a guided diffusion
model, termed DreamRec. Specifically, for a sequence of historical items, it
applies a Transformer encoder to create guidance representations. Noising target
items explores the underlying distribution of item space; then, with the guidance of
historical interactions, the denoising process generates an oracle item to recover
the positive item, so as to cast off negative sampling and depict the true preference
of the user directly. We evaluate the effectiveness of DreamRec through extensive
experiments and comparisons with existing methods. Codes and data are open-sourced
at https://github.com/YangZhengyi98/DreamRec.
Learning Adaptive Tensorial Density Fields for Clean Cryo-ET Reconstruction
Authors: YUANHAO WANG; Ramzi Idoughi; Wolfgang Heidrich
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: We present a novel learning-based framework for reconstructing 3D structures from tilt-series cryo-Electron Tomography (cryo-ET) data. Cryo-ET is a powerful imaging technique that can achieve near-atomic resolutions. Still, it suffers from challenges such as missing-wedge acquisition, large data size, and high noise levels. Our framework addresses these challenges by using an adaptive tensorial-based representation for the 3D density field of the scanned sample. First, we optimize a quadtree structure to partition the volume of interest. Then, we learn a vector-matrix factorization of the tensor representing the density field in each node. Moreover, we use a loss function that combines a differentiable tomographic formation model with three regularization terms: total variation, boundary consistency constraint, and an isotropic Fourier prior. Our framework allows us to query the density at any location using the learned representation and obtain a high-quality 3D tomogram. We demonstrate the superiority of our framework over existing methods using synthetic and real data. Thus, our framework boosts the quality of the reconstruction while reducing the computation time and the memory footprint. The code is available at https://github.com/yuanhaowang1213/adaptivetensordf.
Fine-Grained Cross-View Geo-Localization Using a Correlation-Aware Homography Estimator
Authors: Xiaolong Wang; Runsen Xu; Zhuofan Cui; Zeyu Wan; Yu Zhang
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: In this paper, we introduce a novel approach to fine-grained cross-view geo-localization. Our method aligns a warped ground image with a corresponding GPS-tagged satellite image covering the same area using homography estimation. We first employ a differentiable spherical transform, adhering to geometric principles, to accurately align the perspective of the ground image with the satellite map. This transformation effectively places ground and aerial images in the same view and on the same plane, reducing the task to an image alignment problem. To address challenges such as occlusion, small overlapping range, and seasonal variations, we propose a robust correlation-aware homography estimator to align similar parts of the transformed ground image with the satellite image. Our method achieves sub-pixel resolution and meter-level GPS accuracy by mapping the center point of the transformed ground image to the satellite image using a homography matrix and determining the orientation of the ground camera using a point above the central axis. Operating at a speed of 30 FPS, our method outperforms state-of-the-art techniques, reducing the mean metric localization error by 21.3\% and 32.4\% in same-area and cross-area generalization tasks on the VIGOR benchmark, respectively, and by 34.4\% on the KITTI benchmark in same-area evaluation.
Adaptive Normalization for Non-stationary Time Series Forecasting: A Temporal Slice Perspective
Authors: Zhiding Liu; Mingyue Cheng; Zhi Li; Zhenya Huang; Qi Liu; Yanhu Xie; Enhong Chen
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Deep learning models have progressively advanced time series forecasting due to their powerful capacity in capturing sequence dependence. Nevertheless, it is still challenging to make accurate predictions due to the existence of non-stationarity in real-world data, denoting the data distribution rapidly changes over time. To mitigate such a dilemma, several efforts have been conducted by reducing the non-stationarity with normalization operation. However, these methods typically overlook the distribution discrepancy between the input series and the horizon series, and assume that all time points within the same instance share the same statistical properties, which is too ideal and may lead to suboptimal relative improvements. To this end, we propose a novel slice-level adaptive normalization, referred to \textbf{SAN}, which is a novel scheme for empowering time series forecasting with more flexible normalization and denormalization. SAN includes two crucial designs. First, SAN tries to eliminate the non-stationarity of time series in units of a local temporal slice (i.e., sub-series) rather than a global instance. Second, SAN employs a slight network module to independently model the evolving trends of statistical properties of raw time series. Consequently, SAN could serve as a general model-agnostic plugin and better alleviate the impact of the non-stationary nature of time series data. We instantiate the proposed SAN on four widely used forecasting models and test their prediction results on benchmark datasets to evaluate its effectiveness. Also, we report some insightful findings to deeply analyze and understand our proposed SAN. We make our codes publicly available.
DiffTraj: Generating GPS Trajectory with Diffusion Probabilistic Model
Authors: Yuanshao Zhu; Yongchao Ye; Shiyao Zhang; Xiangyu Zhao; James Yu
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Pervasive integration of GPS-enabled devices and data acquisition technologies has led to an exponential increase in GPS trajectory data, fostering advancements in spatial-temporal data mining research. Nonetheless, GPS trajectories contain personal geolocation information, rendering serious privacy concerns when working with raw data. A promising approach to address this issue is trajectory generation, which involves replacing original data with generated, privacy-free alternatives. Despite the potential of trajectory generation, the complex nature of human behavior and its inherent stochastic characteristics pose challenges in generating high-quality trajectories. 
In this work, we propose a spatial-temporal diffusion probabilistic model for trajectory generation (DiffTraj). This model effectively combines the generative abilities of diffusion models with the spatial-temporal features derived from real trajectories. The core idea is to reconstruct and synthesize geographic trajectories from white noise through a reverse trajectory denoising process. Furthermore, we propose a Trajectory UNet (Traj-UNet) deep neural network to embed conditional information and accurately estimate noise levels during the reverse process. Experiments on two real-world datasets show that DiffTraj can be intuitively applied to generate high-fidelity trajectories while retaining the original distributions. Moreover, the generated results can support downstream trajectory analysis tasks and significantly outperform other methods in terms of geo-distribution evaluations.
SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process
Authors: Mengyu Wang; Henghui Ding; Jun Hao Liew; Jiajun Liu; Yao Zhao; Yunchao Wei
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: In this paper, we explore a principal way to enhance the quality of object masks produced by different segmentation models. We propose a model-agnostic solution called SegRefiner, which offers a novel perspective on this problem by interpreting segmentation refinement as a data generation process. As a result, the refinement process can be smoothly implemented through a series of denoising diffusion steps. Specifically, SegRefiner takes coarse masks as inputs and refines them using a discrete diffusion process. By predicting the label and corresponding states-transition probabilities for each pixel, SegRefiner progressively refines the noisy masks in a conditional denoising manner. To assess the effectiveness of SegRefiner, we conduct comprehensive experiments on various segmentation tasks, including semantic segmentation, instance segmentation, and dichotomous image segmentation. The results demonstrate the superiority of our SegRefiner from multiple aspects. Firstly, it consistently improves both the segmentation metrics and boundary metrics across different types of coarse masks. Secondly, it outperforms previous model-agnostic refinement methods by a significant margin. Lastly, it exhibits a strong capability to capture extremely fine details when refining high-resolution images. The source code and trained models are available at [SegRefiner.git](https://github.com/MengyuWang826/SegRefiner)
CSOT: Curriculum and Structure-Aware Optimal Transport for Learning with Noisy Labels
Authors: Wanxing Chang; Ye Shi; Jingya Wang
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: Learning with noisy labels (LNL) poses a significant challenge in training a well-generalized model while avoiding overfitting to corrupted labels.
Recent advances have achieved impressive performance by identifying clean labels and correcting corrupted labels for training.
However, the current approaches rely heavily on the model’s predictions and evaluate each sample independently without considering either the global or local structure of the sample distribution.
These limitations typically result in a suboptimal solution for the identification and correction processes, which eventually leads to models overfitting to incorrect labels.
In this paper, we propose a novel optimal transport (OT) formulation, called Curriculum and Structure-aware Optimal Transport (CSOT). CSOT concurrently considers the inter- and intra-distribution structure of the samples to construct a robust denoising and relabeling allocator.
During the training process, the allocator incrementally assigns reliable labels to a fraction of the samples with the highest confidence. 
These labels have both global discriminability and local coherence.
Notably, CSOT is a new OT formulation with a nonconvex objective function and curriculum constraints, so it is not directly compatible with classical OT solvers. 
Here, we develop a lightspeed computational method that involves a scaling iteration within a generalized conditional gradient framework to solve CSOT efficiently.
Extensive experiments demonstrate the superiority of our method over the current state-of-the-arts in LNL.
Lift Yourself Up: Retrieval-augmented Text Generation with Self-Memory
Authors: Xin Cheng; Di Luo; Xiuying Chen; Lemao Liu; Dongyan Zhao; Rui Yan
Venue: NeurIPS 2023 (NeurIPS 2023 poster)

Abstract: With direct access to human-written reference as memory, retrieval-augmented generation has achieved much progress in a wide range of text generation tasks. Since better memory would typically prompt better generation (we define this as primal problem). The traditional approach for memory retrieval involves selecting memory that exhibits the highest similarity to the input. However, this method is constrained by the quality of the fixed corpus from which memory is retrieved. In this paper, by exploring the duality of the primal problem: better generation also prompts better memory, we propose a novel framework, selfmem, which addresses this limitation by iteratively employing a retrieval-augmented generator to create an unbounded memory pool and using a memory selector to choose one output as memory for the subsequent generation round. This enables the model to leverage its own output, referred to as self-memory, for improved generation. We evaluate the effectiveness of selfmem on three distinct text generation tasks: neural machine translation, abstractive text summarization, and dialogue generation, under two generation paradigms: fine-tuned small model and few-shot LLM. Our approach achieves state-of-the-art results in four directions in JRC-Acquis translation dataset, 50.3 ROUGE-1 in XSum, and 62.9 ROUGE-1 in BigPatent, demonstrating the potential of self-memory in enhancing retrieval-augmented generation models. Furthermore, we conduct thorough analyses of each component in the selfmem framework to identify current system bottlenecks and provide insights for future research.