OpenRaiser/Intern-Atlas huggingface dataset:
Grammatically recognizing images with tree convolution
Authors: Guangrun Wang, Guangcong Wang, Keze Wang, Xiaodan Liang, Liang Lin
Venue: Conference KDD 2020
Year: 2020

Similar to language, understanding an image can be considered as a hierarchical decomposition process from scenes to objects, parts, pixels, and the corresponding spatial/contextual relations. However, the existing convolutional networks concentrate on stacking redundant convolutional layers with a large number of kernels in a hierarchical organization to implicitly approximate this decomposition. This may limit the network to learn the semantic information conveyed in the internal feature maps that may reveal minor yet crucial differences for visual understanding. Attempting to tackle this problem, this paper proposes a simple yet effective tree convolution (TreeConv) operation for deep neural networks. Specifically, inspired by the image grammar techniques[73] that serve as a unified framework of object representation, learning, and recognition, our TreeConv designs a generative image grammar, i.e., tree generation rule, to parse the hierarchy of internal feature maps by generating tree structures and implicitly learning the specific visual grammars for each object category. Extensive experiments on a variety of benchmarks, i.e., classification (ImageNet / CIFAR), detection & segmentation (COCO 2017), and person re-identification (CUHK03), demonstrate the superiority of our TreeConv in both boosting the accuracy and reducing the computational cost. The source code will be available at: https://github.com/wanggrun/TreeConv.
Secrets of Optical Flow Estimation and Their Principles
Authors: Deqing Sun, Stefan Roth 0001, Michael J. Black
Venue: Conference CVPR 2010
Year: 2010

The accuracy of optical flow estimation algorithms has been improving steadily as evidenced by results on the Middlebury optical flow benchmark. The typical formulation, however, has changed little since the work of Horn and Schunck. We attempt to uncover what has made recent advances possible through a thorough analysis of how the objective function, the optimization method, and modern implementation practices influence accuracy. We discover that “classical” flow formulations perform surprisingly well when combined with modern optimization and implementation techniques. Moreover, we find that while median filtering of intermediate flow fields during optimization is a key to recent performance gains, it leads to higher energy solutions. To understand the principles behind this phenomenon, we derive a new objective that formalizes the median filtering heuristic. This objective includes a nonlocal term that robustly integrates flow estimates over large spatial neighborhoods. By modifying this new term to include information about flow and image boundaries we develop a method that ranks at the top of the Middlebury benchmark.
Curgraph: Curriculum learning for graph classification
Authors: Yiwei Wang, Wei Wang, Yuxuan Liang, Yujun Cai, Bryan Hooi
Venue: Conference WWW 2021
Year: 2021

Graph neural networks (GNNs) have achieved state-of-the-art performance on graph classification tasks. Existing work usually feeds graphs to GNNs in random order for training. However, graphs can vary greatly in their difficulty for classification, and we argue that GNNs can benefit from an easy-to-difficult curriculum, similar to the learning process of humans. Evaluating the difficulty of graphs is challenging due to the high irregularity of graph data. To address this issue, we present the CurGraph (Curriculum Learning for Graph Classification) framework, that analyzes the graph difficulty in the high-level semantic feature space. Specifically, we use the infomax method to obtain graph-level embeddings and a neural density estimator to model the embedding distributions. Then we calculate the difficulty scores of graphs based on the intra-class and inter-class distributions of their embeddings. Given the difficulty scores, CurGraph first exposes a GNN to easy graphs, before gradually moving on to hard ones. To provide a soft transition from easy to hard, we propose a smooth-step method, which utilizes a time-variant smooth function to filter out hard graphs. Thanks to CurGraph, a GNN learns from the graphs at the border of its capability, neither too easy or too hard, to gradually expand its border at each training step. Empirically, CurGraph yields significant gains for popular GNN models on graph classification and enables them to achieve superior performance on miscellaneous graphs.
A data-driven approach for real-time full body pose reconstruction from a depth camera
Authors: Andreas Baak, Meinard Müller, Gaurav Bharaj, Hans-Peter Seidel, Christian Theobalt
Venue: Conference ICCV 2011
Year: 2011

In recent years, depth cameras have become a widely available sensor type that captures depth images at real-time frame rates. Even though recent approaches have shown that 3D pose estimation from monocular 2.5D depth images has become feasible, there are still challenging problems due to strong noise in the depth data and self-occlusions in the motions being captured. In this paper, we present an efficient and robust pose estimation framework for tracking full-body motions from a single depth image stream. Following a data-driven hybrid strategy that combines local optimization with global retrieval techniques, we contribute several technical improvements that lead to speed-ups of an order of magnitude compared to previous approaches. In particular, we introduce a variant of Dijkstra's algorithm to efficiently extract pose features from the depth data and describe a novel late-fusion scheme based on an efficiently computable sparse Hausdorff distance to combine local and global pose estimates. Our experiments show that the combination of these techniques facilitates real-time tracking with stable results even for fast and complex motions, making it applicable to a wide range of inter-active scenarios.
N-best maximal decoders for part models
Authors: Dennis Park, Deva Ramanan
Venue: Conference ICCV 2011
Year: 2011

We describe a method for generating N-best configurations from part-based models, ensuring that they do not overlap according to some user-provided definition of overlap. We extend previous N-best algorithms from the speech community to incorporate non-maximal suppression cues, such that pixel-shifted copies of a single configuration are not returned. We use approximate algorithms that perform nearly identical to their exact counterparts, but are orders of magnitude faster. Our approach outperforms standard methods for generating multiple object configurations in an image. We use our method to generate multiple pose hypotheses for the problem of human pose estimation from video sequences. We present quantitative results that demonstrate that our framework significantly improves the accuracy of a state-of-the-art pose estimation algorithm.
Articulated part-based model for joint object detection and pose estimation
Authors: Min Sun 0001, Silvio Savarese
Venue: Conference ICCV 2011
Year: 2011

Despite recent successes, pose estimators are still somewhat fragile, and they frequently rely on a precise knowledge of the location of the object. Unfortunately, articulated objects are also very difficult to detect. Knowledge about the articulated nature of these objects, however, can substantially contribute to the task of finding them in an image. It is somewhat surprising, that these two tasks are usually treated entirely separately. In this paper, we propose an Articulated Part-based Model (APM) for jointly detecting objects and estimating their poses. APM recursively represents an object as a collection of parts at multiple levels of detail, from coarse-to-fine, where parts at every level are connected to a coarser level through a parent-child relationship (Fig. 1(b)-Horizontal). Parts are further grouped into part-types (e.g., left-facing head, long stretching arm, etc) so as to model appearance variations (Fig. 1(b)-Vertical). By having the ability to share appearance models of part types and by decomposing complex poses into parent-child pairwise relationships, APM strikes a good balance between model complexity and model richness. Extensive quantitative and qualitative experiment results on public datasets show that APM outperforms state-of-the-art methods. We also show results on PASCAL 2007 - cats and dogs - two highly challenging articulated object categories.
A three-point minimal solution for panoramic stitching with lens distortion
Authors: Hailin Jin
Venue: Conference CVPR 2008
Year: 2008

We present a minimal solution for aligning two images taken by a rotating camera from point correspondences. The solution particularly addresses the case where there is lens distortion in the images. We assume to know the two camera centers but not the focal lengths and allow the latter to vary. Our solution uses a minimal number (three) of point correspondences and is well suited to be used in a hypothesis testing framework. It does not suffer from numerical instabilities observed in other algebraic minimal solvers and is also efficient. We validate our solution in multi-image panoramic stitching on real images with lens distortion.
Removing rolling shutter wobble
Authors: Simon Baker, Eric P. Bennett, Sing Bing Kang, Richard Szeliski
Venue: Conference CVPR 2010
Year: 2010

We present an algorithm to remove wobble artifacts from a video captured with a rolling shutter camera undergoing large accelerations or jitter. We show how estimating the rapid motion of the camera can be posed as a temporal super-resolution problem. The low-frequency measurements are the motions of pixels from one frame to the next. These measurements are modeled as temporal integrals of the underlying high-frequency jitter of the camera. The estimated high-frequency motion of the camera is then used to re-render the sequence as though all the pixels in each frame were imaged at the same time. We also present an auto-calibration algorithm that can estimate the time between the capture of subsequent rows in the camera.
Maximum weight cliques with mutex constraints for video object segmentation
Authors: Tianyang Ma, Longin Jan Latecki
Venue: Conference CVPR 2012
Year: 2012

In this paper, we address the problem of video object segmentation, which is to automatically identify the primary object and segment the object out in every frame. We propose a novel formulation of selecting object region candidates simultaneously in all frames as finding a maximum weight clique in a weighted region graph. The selected regions are expected to have high objectness score (unary potential) as well as share similar appearance (binary potential). Since both unary and binary potentials are unreliable, we introduce two types of mutex (mutual exclusion) constraints on regions in the same clique: intra-frame and inter-frame constraints. Both types of constraints are expressed in a single quadratic form. We propose a novel algorithm to compute the maximal weight cliques that satisfy the constraints. We apply our method to challenging benchmark videos and obtain very competitive results that outperform state-of-the-art methods.
A learning based deformable template matching method for automatic rib centerline extraction and labeling in ct images
Authors: Dijia Wu, David Liu 0001, Zoltan Puskas, Chao Lu 0011, Andreas Wimmer, Christian Tietjen, Grzegorz Soza, Shaohua Kevin Zhou
Venue: Conference CVPR 2012
Year: 2012

The automatic extraction and labeling of the rib centerlines is a useful yet challenging task in many clinical applications. In this paper, we propose a new approach integrating rib seed point detection and template matching to detect and identify each rib in chest CT scans. The bottom-up learning based detection exploits local image cues and top-down deformable template matching imposes global shape constraints. To adapt to the shape deformation of different rib cages whereas maintain high computational efficiency, we employ a Markov Random Field (MRF) based articulated rigid transformation method followed by Active Contour Model (ACM) deformation. Compared with traditional methods that each rib is individually detected, traced and labeled, the new approach is not only much more robust due to prior shape constraints of the whole rib cage, but removes tedious post-processing such as rib pairing and ordering steps because each rib is automatically labeled during the template matching. For experimental validation, we create an annotated database of 112 challenging volumes with ribs of various sizes, shapes, and pathologies such as metastases and fractures. The proposed approach shows orders of magnitude higher detection and labeling accuracy than state-of-the-art solutions and runs about 40 seconds for a complete rib cage on the average.
Learning to recognize shadows in monochromatic natural images
Authors: Jiejie Zhu, Kegan G. G. Samuel, Syed Zain Masood, Marshall F. Tappen
Venue: Conference CVPR 2010
Year: 2010

This paper addresses the problem of recognizing shadows from monochromatic natural images. Without chromatic information, shadow classification is very challenging because the invariant color cues are unavailable. Natural scenes make this problem even harder because of ambiguity from many near black objects. We propose to use both shadow-variant and shadow-invariant cues from illumination, textural and odd order derivative characteristics. Such features are used to train a classifier from boosting a decision tree and integrated into a Conditional random Field, which can enforce local consistency over pixel labels. The proposed approach is evaluated using both qualitative and quantitative results based on a novel database of hand-labeled shadows. Our results show shadowed areas of an image can be identified using proposed monochromatic cues.
Prototype-guided attribute-wise interpretable scheme for clothing matching
Authors: Xianjing Han, Xuemeng Song, Jianhua Yin, Yinglong Wang, Liqiang Nie
Venue: Conference SIGIR 2019
Year: 2019

Recently, as an essential part of people's daily life, clothing matching has gained increasing research attention. Most existing efforts focus on the numerical compatibility modeling between fashion items with advanced neural networks, and hence suffer from the poor interpretation, which makes them less applicable in real world applications. In fact, people prefer to know not only whether the given fashion items are compatible, but also the reasonable interpretations as well as suggestions regarding how to make the incompatible outfit harmonious. Considering that the research line of the comprehensively interpretable clothing matching is largely untapped, in this work, we propose a prototype-guided attribute-wise interpretable compatibility modeling (PAICM) scheme, which seamlessly integrates the latent compatible/incompatible prototype learning and compatibility modeling with the Bayesian personalized ranking (BPR) framework. In particular, the latent attribute interaction prototypes, learned by the non-negative matrix factorization (NMF), are treated as templates to interpret the discordant attribute and suggest the alternative item for each fashion item pair. Extensive experiments on the real-world dataset have demonstrated the effectiveness of our scheme.
Interpretable fashion matching with rich attributes
Authors: Xun Yang, Xiangnan He, Xiang Wang, Yunshan Ma, Fuli Feng, Meng Wang, Tat-Seng Chua
Venue: Conference SIGIR 2019
Year: 2019

Understanding the mix-and-match relationships of fashion items receives increasing attention in fashion industry. Existing methods have primarily utilized the visual content to learn the visual compatibility and performed matching in a latent space. Despite their effectiveness, these methods work like a black box and cannot reveal the reasons that two items match well. The rich attributes associated with fashion items, e.g.,off-shoulder dress and black skinny jean, which describe the semantics of items in a human-interpretable way, have largely been ignored. This work tackles the interpretable fashion matching task, aiming to inject interpretability into the compatibility modeling of items. Specifically, given a corpus of matched pairs of items, we not only can predict the compatibility score of unseen pairs, but also learn the interpretable patterns that lead to a good match, e.g., white T-shirt matches with black trouser. We propose a new solution named A ttribute-based I nterpretable C ompatibility (AIC) method, which consists of three modules: 1) a tree-based module that extracts decision rules on matching prediction; 2) an embedding module that learns vector representation for a rule by accounting for the attribute semantics; and 3) a joint modeling module that unifies the visual embedding and rule embedding to predict the matching score. To justify our proposal, we contribute a new Lookastic dataset with fashion attributes available. Extensive experiments show that AIC not only outperforms several state-of-the-art methods, but also provides good interpretability on matching decisions.
Generative Attribute Manipulation Scheme for Flexible Fashion Search
Authors: Xin Yang 0008, Xuemeng Song, Xianjing Han, Haokun Wen, Jie Nie, Liqiang Nie
Venue: Conference SIGIR 2020
Year: 2020

In this work, we aim to investigate the practical task of flexible fashion search with attribute manipulation, where users can retrieve the target fashion items by replacing the unwanted attributes of an available query image with the desired ones (e.g., changing the collar attribute from v-neck to round). Although several pioneer efforts have been dedicated to fulfilling the task, they mainly ignore the potential of generative models in enhancing the visual understanding of target fashion items. To this end, we propose an end-to-end generative attribute manipulation scheme, which consists of a generator and a discriminator. The generator works on producing the prototype image that meets the user's requirement of attribute manipulation over the query image with the regularization of visual-semantic consistency and pixel-wise consistency. Besides, the discriminator aims to jointly fulfill the semantic learning towards correct attribute manipulation and adversarial metric learning for fashion search. Pertaining to the adversarial metric learning, we provide two general paradigms: the pair-based scheme and the triplet-based scheme, where the fake generated prototype images that closely resemble the ground truth images of target items are incorporated as hard negative samples to boost the model performance. Extensive experiments on two real-world datasets verify the effectiveness of our scheme.
Generalization and robustness implications in object-centric learning
Authors: Andrea Dittadi, Samuele S. Papa, Michele De Vita, Bernhard Schölkopf, Ole Winther, Francesco Locatello
Venue: Conference ICML 2022
Year: 2022

The idea behind object-centric representation learning is that natural scenes can better be modeled as compositions of objects and their relations as opposed to distributed representations. This inductive bias can be injected into neural networks to potentially improve systematic generalization and performance of downstream tasks in scenes with multiple objects. In this paper, we train state-of-the-art unsupervised models on five common multi-object datasets and evaluate segmentation metrics and downstream object property prediction. In addition, we study generalization and robustness by investigating the settings where either a single object is out of distribution -- e.g., having an unseen color, texture, or shape -- or global properties of the scene are altered -- e.g., by occlusions, cropping, or increasing the number of objects. From our experimental study, we find object-centric representations to be useful for downstream tasks and generally robust to most distribution shifts affecting objects. However, when the distribution shift affects the input in a less structured manner, robustness in terms of segmentation and downstream task performance may vary significantly across models and distribution shifts.
Lagrangian Fluid Simulation with Continuous Convolutions
Authors: Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, Vladlen Koltun
Venue: Conference ICLR 2020
Year: 2020

We present an approach to Lagrangian fluid simulation with a new type of convolutional network. Our networks process sets of moving particles, which describe fluids in space and time. Unlike previous approaches, we do not build an explicit graph structure to connect the particles but use spatial convolutions as the main differentiable operation that relates particles to their neighbors. To this end we present a simple, novel, and effective extension of N-D convolutions to the continuous domain. We show that our network architecture can simulate different materials, generalizes to arbitrary collision geometries, and can be used for inverse problems. In addition, we demonstrate that our continuous convolutions outperform prior formulations in terms of accuracy and speed.
Towards Physics-informed Deep Learning for Turbulent Flow Prediction
Authors: Rui Wang 0086, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, Rose Yu
Venue: Conference KDD 2020
Year: 2020

While deep learning has shown tremendous success in a wide range of domains, it remains a grand challenge to incorporate physical principles in a systematic manner to the design, training, and inference of such models. In this paper, we aim to predict turbulent flow by learning its highly nonlinear dynamics from spatiotemporal velocity fields of large-scale fluid flow simulations of relevance to turbulence modeling and climate modeling. We adopt a hybrid approach by marrying two well-established turbulent flow simulation techniques with deep learning. Specifically, we introduce trainable spectral filters in a coupled model of Reynolds-averaged Navier-Stokes (RANS) and Large Eddy Simulation (LES), followed by a specialized U-net for prediction. Our approach, which we call Turbulent-Flow Net, is grounded in a principled physics model, yet offers the flexibility of learned representations. We compare our model with state-of-the-art baselines and observe significant reductions in error for predictions 60 frames ahead. Most importantly, our method predicts physical fields that obey desirable physical characteristics, such as conservation of mass, whilst faithfully emulating the turbulent kinetic energy field and spectrum, which are critical for accurate prediction of turbulent flows.
Algorithms for Implicit Deformable Models
Authors: Ross T. Whitaker
Venue: Conference ICCV 1995
Year: 1995

This paper presents a framework for implicit deformable models and a pair of new algorithms for solving the nonlinear partial differential equations that result from this framework. Implicit models offer a useful alternative to parametric models, particularly when dealing with the deformation of higher-dimensional objects. The basic expressions for the evolution of implicit models are relatively straightforward; they follow as a direct consequence of the chain rule for differentiation. More challenging, however, is the development of algorithms that are stable and efficient. The first algorithm is a viscosity approximation which gives solutions over a dense set in the range, providing a means of calculating the solutions of embedded families of contours simultaneously. The second algorithm incorporates sparse solutions for a discrete set of contours. This sparse-field method requires a fraction of the computation compared to the first but offers solutions only for a finite number of contours. Results from 3d medical data as well as video images are shown.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">></ETX>