OpenRaiser/Intern-Atlas huggingface dataset:
Deep variational bayesian modeling of haze degradation process
Authors: Eun Woo Im, Junsung Shin, Sungyong Baik, Tae Hyun Kim
Venue: Conference CIKM 2023
Year: 2023

Relying on the representation power of neural networks, most recent works have often neglected several factors involved in haze degradation, such as transmission (the amount of light reaching an observer from a scene over distance) and atmospheric light. These factors are generally unknown, making dehazing problems ill-posed and creating inherent uncertainties. To account for such uncertainties and factors involved in haze degradation, we introduce a variational Bayesian framework for single image dehazing. We propose to take not only a clean image and but also transmission map as latent variables, the posterior distributions of which are parameterized by corresponding neural networks: dehazing and transmission networks, respectively. Based on a physical model for haze degradation, our variational Bayesian framework leads to a new objective function that encourages the cooperation between them, facilitating the joint training of and thereby boosting the performance of each other. In our framework, a dehazing network can estimate a clean image independently of a transmission map estimation during inference, introducing no overhead. Furthermore, our model-agnostic framework can be seamlessly incorporated with other existing dehazing networks, greatly enhancing the performance consistently across datasets and models.
Fast 3D Scanning with Automatic Motion Compensation
Authors: Thibaut Weise, Bastian Leibe, Luc Van Gool
Venue: Conference CVPR 2007
Year: 2007

We present a novel 3D scanning system combining stereo and active illumination based on phase-shift for robust and accurate scene reconstruction. Stereo overcomes the traditional phase discontinuity problem and allows for the reconstruction of complex scenes containing multiple objects. Due to the sequential recording of three patterns, motion will introduce artifacts in the reconstruction. We develop a closed-form expression for the motion error in order to apply motion compensation on a pixel level. The resulting scanning system can capture accurate depth maps of complex dynamic scenes at 17 fps and can cope with both rigid and deformable objects.
Global minimum for active contour models: A minimal path approach
Authors: Laurent D. Cohen, Ron Kimmel
Venue: Conference CVPR 1996
Year: 1996

A new boundary detection approach for shape modeling is presented. It detects the global minimum of an active contour model’s energy between two end points. Initialization is made easier and the curve is not trapped at a local minimum by spurious edges. We modify the “snake” energy by including the internal regularization term in the external potential term. Our method is based on finding a path of minimal length in a Riemannian metric. We then make use of a new efficient numerical method to find this shortest path.It is shown that the proposed energy, though based only on a potential integrated along the curve, imposes a regularization effect like snakes. We explore the relation between the maximum curvature along the resulting contour and the potential generated from the image.The method is capable to close contours, given only one point on the objects' boundary by using a topology-based saddle search routine.We show examples of our method applied to real aerial and medical images.
Flash: fast bayesian optimization for data analytic pipelines
Authors: Yuyu Zhang, Mohammad Taha Bahadori, Hang Su, Jimeng Sun
Venue: Conference KDD 2016
Year: 2016

Modern data science relies on data analytic pipelines to organize interdependent computational steps. Such analytic pipelines often involve different algorithms across multiple steps, each with its own hyperparameters. To achieve the best performance, it is often critical to select optimal algorithms and to set appropriate hyperparameters, which requires large computational efforts. Bayesian optimization provides a principled way for searching optimal hyperparameters for a single algorithm. However, many challenges remain in solving pipeline optimization problems with high-dimensional and highly conditional search space. In this work, we propose Fast LineAr SearcH (FLASH), an efficient method for tuning analytic pipelines. FLASH is a two-layer Bayesian optimization framework, which firstly uses a parametric model to select promising algorithms, then computes a nonparametric model to fine-tune hyperparameters of the promising algorithms. FLASH also includes an effective caching algorithm which can further accelerate the search process. Extensive experiments on a number of benchmark datasets have demonstrated that FLASH significantly outperforms previous state-of-the-art methods in both search speed and accuracy. Using 50% of the time budget, FLASH achieves up to 20% improvement on test error rate compared to the baselines. FLASH also yields state-of-the-art performance on a real-world application for healthcare predictive modeling.
Define: Deep factorized input token embeddings for neural sequence modeling
Authors: Sachin Mehta, Rik Koncel-Kedziorski, Mohammad Rastegari, Hannaneh Hajishirzi
Venue: Conference ICLR 2020
Year: 2020

For sequence models with large vocabularies, a majority of network parameters lie in the input and output layers. In this work, we describe a new method, DeFINE, for learning deep token representations efficiently. Our architecture uses a hierarchical structure with novel skip-connections which allows for the use of low dimensional input and output layers, reducing total parameters and training time while delivering similar or better performance versus existing methods. DeFINE can be incorporated easily in new or existing sequence models. Compared to state-of-the-art methods including adaptive input representations, this technique results in a 6% to 20% drop in perplexity. On WikiText-103, DeFINE reduces the total parameters of Transformer-XL by half with minimal impact on performance. On the Penn Treebank, DeFINE improves AWD-LSTM by 4 points with a 17% reduction in parameters, achieving comparable performance to state-of-the-art methods with fewer parameters. For machine translation, DeFINE improves the efficiency of the Transformer model by about 1.4 times while delivering similar performance.
Learning invariances for policy generalization
Authors: Remi Tachet des Combes, Philip Bachman, Harm van Seijen
Venue: Conference ICLR 2018
Year: 2018

While recent progress has spawned very powerful machine learning systems, those agents remain extremely specialized and fail to transfer the knowledge they gain to similar yet unseen tasks. In this paper, we study a simple reinforcement learning problem and focus on learning policies that encode the proper invariances for generalization to different settings. We evaluate three potential methods for policy generalization: data augmentation, meta-learning and adversarial training. We find our data augmentation method to be effective, and study the potential of meta-learning and adversarial learning as alternative task-agnostic approaches.
Gradual AutoML using Lale
Authors: Martin Hirzel, Kiran Kate, Parikshit Ram, Avraham Shinnar, Jason Tsay
Venue: Conference KDD 2022
Year: 2022

Lale is a sklearn-compatible library for automated machine learning (AutoML). It is open-source (https://github.com/ibm/lale) and addresses the need for gradual automation of machine learning as opposed to offering a black-box AutoML tool. Black-box AutoML tools are difficult to customize and thus restrict data scientists in leveraging their knowledge and intuition in the automation process. Lale is built on three principles: progressive disclosure, orthogonality, and least surprise. These enable a gradual approach offering a spectrum of usage patterns starting from total automation to controlling almost every aspect of AutoML. Lale provides compositional constructs that let data scientists control some aspects of their pipelines while leaving other aspects free to be searched automatically. This tutorial demonstrates the use of Lale for various machine-learning tasks, showing how to progressively exercise more customization. It also covers AutoML for advanced scenarios such as class imbalance correction, bias detection and mitigation, multi-objective optimization, and working with multi-table datasets. While Lale comes with hyperparameter specifications for 216 operators out-of-the-box, users can also add more operators of their own, and this tutorial covers how to do that. Overall, this tutorial teaches you how you can exercise fine-grained control over AutoML without having to be an AutoML expert.
Physics-infused machine learning for crowd simulation
Authors: Guozhen Zhang, Zihan Yu, Depeng Jin, Yong Li 0008
Venue: Conference KDD 2022
Year: 2022

Crowd simulation acts as the basic component in traffic management, urban planning, and emergency management. Most existing approaches use physics-based models due to their robustness and strong generalizability, yet they fall short in fidelity since human behaviors are too complex and heterogeneous for a universal physical model to describe. Recent research tries to solve this problem by deep learning methods. However, they are still unable to generalize well beyond training distributions. In this work, we propose to jointly leverage the strength of the physical and neural network models for crowd simulation by a Physics-Infused Machine Learning (PIML) framework. The key idea is to let the two models learn from each other by iteratively going through a physics-informed machine learning process and a machine-learning-aided physics discovery process. We present our realization of the framework with a novel neural network model, Physics-informed Crowd Simulator (PCS), and tailored interaction mechanisms enabling the two models to facilitate each other. Specifically, our designs enable the neural network model to identify generalizable signals from real-world data better and yield physically consistent simulations with the physical model's form and simulation results as a prior. Further, by performing symbolic regression on the well-trained neural network, we obtain improved physical models that better describe crowd dynamics. Extensive experiments on two publicly available large-scale real-world datasets show that, with the framework, we successfully obtain a neural network model with strong generalizability and a new physical model with valid physical meanings at the same time. Both models outperform existing state-of-the-art simulation methods in accuracy, fidelity, and generalizability, which demonstrates the effectiveness of the PIML framework for improving simulation performance and its capability for facilitating scientific discovery and deepening our understandings of crowd dynamics. We release the codes at https://github.com/tsinghua-fib-lab/PIML.
Globally optimal training of neural networks with threshold activation functions
Authors: Tolga Ergen, Halil Ibrahim Gulluk, Jonathan Lacotte, Mert Pilanci
Venue: Conference ICLR 2023
Year: 2023

Threshold activation functions are highly preferable in neural networks due to their efficiency in hardware implementations. Moreover, their mode of operation is more interpretable and resembles that of biological neurons. However, traditional gradient based algorithms such as Gradient Descent cannot be used to train the parameters of neural networks with threshold activations since the activation function has zero gradient except at a single non-differentiable point. To this end, we study weight decay regularized training problems of deep neural networks with threshold activations. We first show that regularized deep threshold network training problems can be equivalently formulated as a standard convex optimization problem, which parallels the LASSO method, provided that the last hidden layer width exceeds a certain threshold. We also derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network. We corroborate our theoretical results with various numerical experiments.
Training discrete deep generative models via gapped straight-through estimator
Authors: Ting-Han Fan, Ta-Chung Chi, Alexander I. Rudnicky, Peter J. Ramadge
Venue: Conference ICML 2022
Year: 2022

While deep generative models have succeeded in image processing, natural language processing, and reinforcement learning, training that involves discrete random variables remains challenging due to the high variance of its gradient estimation process. Monte Carlo is a common solution used in most variance reduction approaches. However, this involves time-consuming resampling and multiple function evaluations. We propose a Gapped Straight-Through (GST) estimator to reduce the variance without incurring resampling overhead. This estimator is inspired by the essential properties of Straight-Through Gumbel-Softmax. We determine these properties and show via an ablation study that they are essential. Experiments demonstrate that the proposed GST estimator enjoys better performance compared to strong baselines on two discrete deep generative modeling tasks, MNIST-VAE and ListOps.
Towards feature selection for ranking and classification exploiting quantum annealers
Authors: Maurizio Ferrari Dacrema, Fabio Moroni, Riccardo Nembrini, Nicola Ferro, Guglielmo Faggioli, Paolo Cremonesi
Venue: Conference SIGIR 2022
Year: 2022

Feature selection is a common step in many ranking, classification, or prediction tasks and serves many purposes. By removing redundant or noisy features, the accuracy of ranking or classification can be improved and the computational cost of the subsequent learning steps can be reduced. However, feature selection can be itself a computationally expensive process. While for decades confined to theoretical algorithmic papers, quantum computing is now becoming a viable tool to tackle realistic problems, in particular special-purpose solvers based on the Quantum Annealing paradigm. This paper aims to explore the feasibility of using currently available quantum computing architectures to solve some quadratic feature selection algorithms for both ranking and classification. The experimental analysis includes 15 state-of-the-art datasets. The effectiveness obtained with quantum computing hardware is comparable to that of classical solvers, indicating that quantum computers are now reliable enough to tackle interesting problems. In terms of scalability, current generation quantum computers are able to provide a limited speedup over certain classical algorithms and hybrid quantum-classical strategies show lower computational cost for problems of more than a thousand features.
Adavi: Automatic dual amortized variational inference applied to pyramidal bayesian models
Authors: Louis Rouillard, Demian Wassermann
Venue: Conference ICLR 2022
Year: 2022

Frequently, population studies feature pyramidally-organized data represented using Hierarchical Bayesian Models (HBM) enriched with plates. These models can become prohibitively large in settings such as neuroimaging, where a sample is composed of a functional MRI signal measured on 300 brain locations, across 4 measurement sessions, and 30 subjects, resulting in around 1 million latent parameters.Such high dimensionality hampers the usage of modern, expressive flow-based techniques.To infer parameter posterior distributions in this challenging class of problems, we designed a novel methodology that automatically produces a variational family dual to a target HBM. This variational family, represented as a neural network, consists in the combination of an attention-based hierarchical encoder feeding summary statistics to a set of normalizing flows. Our automatically-derived neural network exploits exchangeability in the plate-enriched HBM and factorizes its parameter space. The resulting architecture reduces by orders of magnitude its parameterization with respect to that of a typical flow-based representation, while maintaining expressivity.Our method performs inference on the specified HBM in an amortized setup: once trained, it can readily be applied to a new data sample to compute the parameters' full posterior.We demonstrate the capability and scalability of our method on simulated data, as well as a challenging high-dimensional brain parcellation experiment. We also open up several questions that lie at the intersection between normalizing flows, SBI, structured Variational Inference, and inference amortization.
Diffeomorphic matching of distributions: A new approach for unlabelled point-sets and sub-manifolds matching
Authors: Joan Alexis Glaunès, Alain Trouvé, Laurent Younes
Venue: Conference CVPR 2004
Year: 2004

In the paper, we study the problem of optimal matching of two generalized functions (distributions) via a diffeomorphic transformation of the ambient space. In the particular case of discrete distributions (weighted sums of Dirac measures), we provide a new algorithm to compare two arbitrary unlabelled sets of points, and show that it behaves properly in limit of continuous distributions on sub-manifolds. As a consequence, the algorithm may apply to various matching problems, such as curve or surface matching (via a sub-sampling), or mixings of landmark and curve data. As the solution forbids high energy solutions, it is also robust towards addition of noise and the technique can be used for nonlinear projection of datasets. We present 2D and 3D experiments.
Avoiding biases due to similarity assumptions in node embeddings
Authors: Deepayan Chakrabarti
Venue: Conference KDD 2022
Year: 2022

Node embeddings are vectors, one per node, that capture a graph's structure. The basic structure is the adjacency matrix of the graph. Recent methods also make assumptions about the similarity of unlinked nodes. However, such assumptions can lead to unintentional but systematic biases against groups of nodes. Calculating similarities between far-off nodes is also difficult under privacy constraints and in dynamic graphs. Our proposed embedding, called NEWS, makes no similarity assumptions, avoiding potential risks to privacy and fairness. NEWS is parameter-free, enables fast link prediction, and has linear complexity. These gains from avoiding assumptions do not significantly affect accuracy, as we show via comparisons against several existing methods on $21$ real-world networks. Code is available at https://github.com/deepayan12/news.
From closing triangles to closing higher-order motifs
Authors: Ryan A. Rossi, Anup Rao, Sungchul Kim, Eunyee Koh, Nesreen K. Ahmed
Venue: Conference WWW 2020
Year: 2020

This work introduces higher-order ranking and link prediction methods based on closing higher-order network motifs. In particular, we propose the general notion of a motif closure that goes beyond simple triangle closures and demonstrate that these new motif closures often outperform triangle-based methods. This result implies that one should consider other motif closures beyond simple triangles. We also find that the “best” motif closure depends highly on the underlying network and its structural properties. Furthermore, the methods are fast and efficient for real-time applications such as online visitor stitching, web search, and recommendation. The experimental results indicate the importance of these new motif closures. Finally, the new motif closures can serve as a basis for developing better (un)supervised ranking/link prediction methods.
Contextualizing citations for scientific summarization using word embeddings and domain knowledge
Authors: Arman Cohan, Nazli Goharian
Venue: Conference SIGIR 2017
Year: 2017

Citation texts are sometimes not very informative or in some cases inaccurate by themselves; they need the appropriate context from the referenced paper to reflect its exact contributions. To address this problem, we propose an unsupervised model that uses distributed representation of words as well as domain knowledge to extract the appropriate context from the reference paper. Evaluation results show the effectiveness of our model by significantly outperforming the state-of-the-art. We furthermore demonstrate how an effective contextualization method results in improving citation-based summarization of the scientific articles.
Learning to pack: A data-driven tree search algorithm for large-scale 3d bin packing problem
Authors: Qianwen Zhu, Xihan Li, Zihan Zhang, Zhixing Luo, Xialiang Tong, Mingxuan Yuan, Jia Zeng
Venue: Conference CIKM 2021
Year: 2021

The 3-dimensional bin packing problem (3D-BPP) is not only fundamental in combinatorial optimization but also widely applied in real world logistics. In the modern logistics industry, the complexity of constraints, heterogeneity of cargoes and scale of orders are dramatically increased, leading to great challenges to devise packing plans up to standard. While the tree search algorithm is proved to be a successful paradigm to solve the 3D-BPP, it is too time-consuming to be applied in the aforementioned large-scale scenarios. To overcome the limitation, we propose a data-driven tree search algorithm (DDTS) to tackle the 3D-BPP. The solution space with complicated constraints is explored by a tree search algorithm, and a convolutional neural network trained with historical data guides pruning the tree so as to accelerate the search process. Computational experiments on real-world datasets show that our algorithm outperforms the state-of-the-art approach with a loading rate improvement of 2.47%. Moreover, the deep learning technique increases searching efficiency by 37.14% with only 0.04% performance loss. The algorithm has been deployed in Huawei Logistics System, which increases the loading rate by 3% and could reduce the logistics cost by millions of dollars per year. To the best of our knowledge, we are the first to embed pruning networks into tree search for the large-scale 3D-BPP.
Counterfactual generative models for time-varying treatments
Authors: Shenghao Wu, Wenbin Zhou, Minshuo Chen, Shixiang Zhu
Venue: Conference KDD 2024
Year: 2024

Estimating the counterfactual outcome of treatment is essential for decision-making in public health and clinical science, among others. Often, treatments are administered in a sequential, time-varying manner, leading to an exponentially increased number of possible counterfactual outcomes. Furthermore, in modern applications, the outcomes are high-dimensional and conventional average treatment effect estimation fails to capture disparities in individuals. To tackle these challenges, we propose a novel conditional generative framework capable of producing counterfactual samples under time-varying treatment, without the need for explicit density estimation. Our method carefully addresses the distribution mismatch between the observed and counterfactual distributions via a loss function based on inverse probability re-weighting, and supports integration with state-of-the-art conditional generative models such as the guided diffusion and conditional variational autoencoder. We present a thorough evaluation of our method using both synthetic and real-world data. Our results demonstrate that our method is capable of generating high-quality counterfactual samples and outperforms the state-of-the-art baselines.