OpenRaiser/Intern-Atlas huggingface dataset:
Unsupervised learning of hierarchical spatial structures in images
Authors: Devi Parikh, C. Lawrence Zitnick, Tsuhan Chen
Venue: Conference CVPR 2009
Year: 2009

The visual world demonstrates organized spatial patterns, among objects or regions in a scene, object-parts in an object, and low-level features in object-parts. These classes of spatial structures are inherently hierarchical in nature. Although seemingly quite different these spatial patterns are simply manifestations of different levels in a hierarchy. In this work, we present a unified approach to unsupervised learning of hierarchical spatial structures from a collection of images. Ours is a hierarchical rule-based model capturing spatial patterns, where each rule is represented by a star-graph. We propose an unsupervised EM-style algorithm to learn our model from a collection of images. We show that the inference problem of determining the set of learnt rules instantiated in an image is equivalent to finding the minimum-cost Steiner tree in a directed acyclic graph. We evaluate our approach on a diverse set of data sets of object categories, natural outdoor scenes and images from complex street scenes with multiple objects.
Wdrass: A web-scale dataset for document retrieval and answer sentence selection
Authors: Zeyu Zhang, Thuy Vu, Sunil Gandhi, Ankit Chadha, Alessandro Moschitti
Venue: Conference CIKM 2022
Year: 2022

Open-Domain Question Answering (ODQA) systems generate answers from relevant text returned by search engines, e.g., lexical features-based such as BM25, or embeddings-based such as dense passage retrieval (DPR). Few datasets are available for this task: they mainly focus on QA systems based on machine reading (MR) approach, and show problematic evaluation, mostly based on uncontextualized short answer matching. In this paper, we present WDRASS, a dataset for ODQA based on answer sentence selection (AS2) models, which consider sentences as candidate answers for QA systems. WDRASS consists of ∼64k questions and 800k+ labeled passages and sentences extracted from 30M documents. We evaluate the dataset by training models on it and comparing with the same models trained on Google NQ. Our experiments show that WDRASS significantly improves the performance of retrieval and reranking models, thus boosting the accuracy of downstream QA tasks. We believe our dataset can produce significant impact in advancing IR research.
Efficient matching of pictorial structures
Authors: Pedro F. Felzenszwalb, Daniel P. Huttenlocher
Venue: Conference CVPR 2000
Year: 2000

A pictorial structure is a collection of parts arranged in a deformable configuration. Each part is represented using a simple appearance model and the deformable configuration is represented by spring-like connections between pairs of parts. While pictorial structures were introduced a number of years ago, they have not been broadly applied to matching and recognition problems. This has been due in part to the computational difficulty of matching pictorial structures to images. In this paper we present an efficient algorithm for finding the best global match of a pictorial stucture to an image. With this improved algorithm, pictorial structures provide a practical and powerful framework for quantitative descriptions of objects and scenes, and are suitable for many generic image recognition problems. We illustrate the approach using simple models of a person and a car.
A Bi-illuminant Dichromatic Reflection Model for Understanding Images
Authors: Bruce A. Maxwell, Richard M. Friedhoff, Casey A. Smith
Venue: Conference CVPR 2008
Year: 2008

This paper presents a new model for understanding the appearance of objects that exhibit both body and surface reflection under realistic illumination. Specifically, the model represents the appearance of surfaces that interact with a dominant illuminant and a non-negligible ambient illuminant that may have different spectral power distributions. Real illumination environments usually have an ambient illuminant, and the current dynamic range of consumer cameras is sufficient to capture significant information in shadows. The bi-illuminant dichromatic reflection model explains numerous empirical findings in the literature and has implications for commonly used chromaticity spaces that claim to be illumination invariant but are not in many natural situations. One outcome of the model is the first 2-D chromaticity space for an RGB image that is robust to illumination change given dominant and ambient illuminants with different spectral power distributions.
Deformable surface tracking ambiguities
Authors: Mathieu Salzmann, Vincent Lepetit, Pascal Fua
Venue: Conference CVPR 2007
Year: 2007

We study from a theoretical standpoint the ambiguities that occur when tracking a generic deformable surface under monocular perspective projection given 3D to 2D correspondences. We show that, additionally to the known scale ambiguity, a set of potential ambiguities can be clearly identified. From this, we deduce a minimal set of constraints required to disambiguate the problem and incorporate them into a working algorithm that runs on real noisy data.
PEBL: Positive example based learning for Web page classification using SVM
Authors: Hwanjo Yu, Jiawei Han, Kevin Chen-Chuan Chang
Venue: Conference KDD 2002
Year: 2002

Web page classification is one of the essential techniques for Web mining. Specifically, classifying Web pages of a user-interesting class is the first step of mining interesting information from the Web. However, constructing a classifier for an interesting class requires laborious pre-processing such as collecting positive and negative training examples. For instance, in order to construct a homepage classifier, one needs to collect a sample of homepages (positive examples) and a sample of non-homepages (negative examples). In particular, collecting negative training examples requires arduous work and special caution to avoid biasing them. We introduce in this paper the Positive Example Based Learning (PEBL) framework for Web page classification which eliminates the need for manually collecting negative training examples in pre-processing. We present an algorithm called Mapping-Convergence (M-C) that achieves classification accuracy (with positive and unlabeled data) as high as that of traditional SVM (with positive and negative data). Our experiments show that when the M-C algorithm uses the same amount of positive examples as that of traditional SVM, the M-C algorithm performs as well as traditional SVM.
Efficient discriminative learning of parts-based models
Authors: M. Pawan Kumar, Andrew Zisserman, Philip H. S. Torr
Venue: Conference ICCV 2009
Year: 2009

Supervised learning of a parts-based model can be formulated as an optimization problem with a large (exponential in the number of parts) set of constraints. We show how this seemingly difficult problem can be solved by (i) reducing it to an equivalent convex problem with a small, polynomial number of constraints (taking advantage of the fact that the model is tree-structured and the potentials have a special form); and (ii) obtaining the globally optimal model using an efficient dual decomposition strategy. Each component of the dual decomposition is solved by a modified version of the highly optimized SVM-Light algorithm. To demonstrate the effectiveness of our approach, we learn human upper body models using two challenging, publicly available datasets. Our model accounts for the articulation of humans as well as the occlusion of parts. We compare our method with a baseline iterative strategy as well as a state of the art algorithm and show significant efficiency improvements.
Text-to-motion retrieval: Towards joint understanding of human motion data and natural language
Authors: Nicola Messina, Jan Sedmidubský, Fabrizio Falchi, Tomás Rebok
Venue: Conference SIGIR 2023
Year: 2023

Due to recent advances in pose-estimation methods, human motion can be extracted from a common video in the form of 3D skeleton sequences. Despite wonderful application opportunities, effective and efficient content-based access to large volumes of such spatio-temporal skeleton data still remains a challenging problem. In this paper, we propose a novel content-based text-to-motion retrieval task, which aims at retrieving relevant motions based on a specified natural-language textual description. To define baselines for this uncharted task, we employ the BERT and CLIP language representations to encode the text modality and successful spatio-temporal models to encode the motion modality. We additionally introduce our transformer-based approach, called Motion Transformer (MoT), which employs divided space-time attention to effectively aggregate the different skeleton joints in space and time. Inspired by the recent progress in text-to-image/video matching, we experiment with two widely-adopted metric-learning loss functions. Finally, we set up a common evaluation protocol by defining qualitative metrics for assessing the quality of the retrieved motions, targeting the two recently-introduced KIT Motion-Language and HumanML3D datasets. The code for reproducing our results is available here: https://github.com/mesnico/text-to-motion-retrieval.
Enhancing on-device llm inference with historical cloud-based llm interactions
Authors: Yucheng Ding, Chaoyue Niu, Fan Wu 0006, Shaojie Tang 0001, Chengfei Lyu, Guihai Chen
Venue: Conference KDD 2024
Year: 2024

Many billion-scale large language models (LLMs) have been released for resource-constraint mobile devices to provide local LLM inference service when cloud-based powerful LLMs are not available. However, the capabilities of current on-device LLMs still lag behind those of cloud-based LLMs, and how to effectively and efficiently enhance on-device LLM inference becomes a practical requirement. We thus propose to collect the user's historical interactions with the cloud-based LLM and build an external datastore on the mobile device for enhancement using nearest neighbors search. Nevertheless, the full datastore improves the quality of token generation at the unacceptable expense of much slower generation speed. To balance performance and efficiency, we propose to select an optimal subset of the full datastore within the given size limit, the optimization objective of which is proven to be submodular. We further design an offline algorithm, which selects the subset after the construction of the full datastore, as well as an online algorithm, which performs selection over the stream and can be flexibly scheduled. We theoretically analyze the performance guarantee and the time complexity of the offline and the online designs to demonstrate effectiveness and scalability. We finally take three ChatGPT related dialogue datasets and four different on-device LLMs for evaluation. Evaluation results show that the proposed designs significantly enhance LLM performance in terms of perplexity while maintaining fast token generation speed. Practical overhead testing on the smartphone reveal the efficiency of on-device datastore subset selection from memory usage and computation overhead.
Breaking certified defenses: Semantic adversarial examples with spoofed robustness certificates
Authors: Amin Ghiasi, Ali Shafahi, Tom Goldstein
Venue: Conference ICLR 2020
Year: 2020

To deflect adversarial attacks, a range of "certified" classifiers have been proposed. In addition to labeling an image, certified classifiers produce (when possible) a certificate guaranteeing that the input image is not an $\ell_p$-bounded adversarial example. We present a new attack that exploits not only the labelling function of a classifier, but also the certificate generator. The proposed method applies large perturbations that place images far from a class boundary while maintaining the imperceptibility property of adversarial examples. The proposed "Shadow Attack" causes certifiably robust networks to mislabel an image and simultaneously produce a "spoofed" certificate of robustness.
Zero-shot image classification with logic adapter and rule prompt
Authors: Dongran Yu, Xueyan Liu, Bo Yang
Venue: Conference WWW 2024
Year: 2024

Zero-shot image classification, which aims to predict unseen classes whose samples have never appeared during the training phase, is crucial in the Web domain because many new web images appear on various websites. Attributes, as annotations for class-level characteristics, are widely used semantic information for this task. However, most current methods often fail to capture discriminative image features between similar images from different classes, leading to unsatisfactory zero-shot image classification results. This is because they solely focus on limited visual-attribute feature alignment. Therefore, we propose a Zero-Shot image Classification with Logic adapter and Rule prompt method called ZSCLR, which utilizes logic adapter and rule prompts to encourage the model to capture discriminative image features and achieve reasoning. Specifically, ZSCLR consists of a visual perception module and a logic adapter. The visual perception module extracts image features from training data. At the same time, the logic adapter utilizes the Markov logic network to encode the extracted image features and rule prompts for refining the discriminative image features. Due to predicates of rule prompts representing symbolic discriminative features, the proposed model can focus more on these discriminative features and achieve more precise image classification. Additionally, the logic adapter enables the model to adapt from recognizing images in seen classes to those in unseen classes through the reasoning of the Markov logic networks. We implement experiments on three standard zero-shot image classification benchmarks, and ZSCLR achieves competitive performance. Furthermore, ZSCLR can provide explanations for its predictions through rule prompts.
A Population-to-individual Tuning Framework for Adapting Pretrained LM to On-device User Intent Prediction
Authors: Jiahui Gong, Jingtao Ding, Fanjin Meng, Guilong Chen, Hong Chen, Shen Zhao, Haisheng Lu, Yong Li 0008
Venue: Conference KDD 2024
Year: 2024

Mobile devices, especially smartphones, can support rich functions and have developed into indispensable tools in daily life. With the rise of generative AI services, smartphones can potentially transform into personalized assistants, anticipating user needs and scheduling services accordingly. Predicting user intents on smartphones, and reflecting anticipated activities based on past interactions and context, remains a pivotal step towards this vision. Existing research predominantly focuses on specific domains, neglecting the challenge of modeling diverse event sequences across dynamic contexts. Leveraging pre-trained language models (PLMs) offers a promising avenue, yet adapting PLMs to on-device user intent prediction presents significant challenges. To address these challenges, we propose PITuning, a Population-to-Individual Tuning framework. PITuning enhances common pattern extraction through dynamic event-to-intent transition modeling and addresses long-tailed preferences via adaptive unlearning strategies. Experimental results on real-world datasets demonstrate PITuning's superior intent prediction performance, highlighting its ability to capture long-tailed preferences and its practicality for on-device prediction scenarios.
Ginver: Generative model inversion attacks against collaborative inference
Authors: Yupeng Yin, Xianglong Zhang, Huanle Zhang, Feng Li, Yue Yu, Xiuzhen Cheng, Pengfei Hu
Venue: Conference WWW 2023
Year: 2023

Deep Learning (DL) has been widely adopted in almost all domains, from threat recognition to medical diagnosis. Albeit its supreme model accuracy, DL imposes a heavy burden on devices as it incurs overwhelming system overhead to execute DL models, especially on Internet-of-Things (IoT) and edge devices. Collaborative inference is a promising approach to supporting DL models, by which the data owner (the victim) runs the first layers of the model on her local device and then a cloud provider (the adversary) runs the remaining layers of the model. Compared to offloading the entire model to the cloud, the collaborative inference approach is more data privacy-preserving as the owner’s model input is not exposed to outsiders. However, we show in this paper that the adversary can restore the victim’s model input by exploiting the output of the victim’s local model. Our attack is dubbed Ginver 1: Generative model inversion attacks against collaborative inference. Once trained, Ginver can infer the victim’s unseen model inputs without remaking the inversion attack model and thus has the generative capability. We extensively evaluate Ginver under different settings (e.g., white-box and black-box of the victim’s local model) and applications (e.g., CIFAR10 and FaceScrub datasets). The experimental results show that Ginver recovers high-quality images from the victims.
2.5D building modeling by discovering global regularities
Authors: Qian-Yi Zhou, Ulrich Neumann
Venue: Conference CVPR 2012
Year: 2012

We introduce global regularities in the 2.5D building modeling problem, to reflect the orientation and placement similarities between planar elements in building structures. Given a 2.5D point cloud scan, we present an automatic approach that simultaneously detects locally fitted plane primitives and global regularities. While global regularities are extracted by analyzing the plane primitives, they adjust the planes in return and effectively correct local fitting errors. We explore a broad variety of global regularities between 2.5D planar elements including both planer roof patches and planar facade patches. By aligning planar elements to global regularities, our method significantly improves the model quality in terms of both geometry and human judgement.
Diffuse Shading, Visibility Fields, and the Geometry of Ambient Light
Authors: Michael S. Langer, Steven W. Zucker
Venue: Conference ICCV 1993
Year: 1993

A dominating sky principle is introduced. It states that under diffuse lighting conditions, ambient light has a bimodal structure: the sky is typically much brighter than the ground. The dominating sky principle suggests a general model of image formation under diffuse lighting conditions. The idea behind this model is that the luminous intensity of a surface is determined by the visible region of the diffuse source, or surface visibility function. The model is presented. A novel concept in shading analysis, surface aperture, is discussed. Surface aperture is the solid angle of the visible diffuse source. Although the unit surface normal is a factor under diffuse lighting conditions, in most situations surface aperture dominates. The surface visibility and aperture functions are embedded into visibility and aperture fields defined in the free space above the surfaces. The visibility field is useful because it satisfies simple local constraints. A fast, parallel computation is described for the forward problem of rendering a surface which is illuminated by a diffuse source and for the inverse problem, namely, shape from shading on a cloudy day. The forward and inverse solutions are quite similar. Experimental results are presented.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">></ETX>
Robust and efficient image alignment with spatially varying illumination models
Authors: Shang-Hong Lai, Ming Fang
Venue: Conference CVPR 1999
Year: 1999

Image alignment is one of the most important task in computer vision. In this paper, we explicitly model spatial illumination variations by low-order polynomial functions in an energy minimization framework. Data constraints for the alignment and illumination parameters are derived from the first-order Taylor approximation of a generalized brightness assumption. We formulate the parameter estimation problem in a weighted least-square framework by using the influence function from robust estimation to derive an iterative re-weighted least-square algorithm. A dynamic weighting scheme, which combines the factors from influence function, consistency of image gradients and nonlinear image intensity sensing, is used to improve the robustness of the image matching. In addition, a constraint sampling scheme and an estimation-warping alternative strategy are used in the proposed algorithm to improve its efficiency and accuracy. Experimental results are shown to demonstrate the robustness, efficiency and accuracy of the algorithm.
A parametric template method and its application to robust matching
Authors: Koichi Tanaka, Mutsuo Sano, Shuichi Ohara, Masashi Okudaira
Venue: Conference CVPR 2000
Year: 2000

We propose a template-matching method in which a parametric template space is constructed from a given set of template images then quickly matched to a reference image. In this method geometrical changes in an image object due to translation, rotation or scaling, and non-geometrical changes such as illumination variations or individual variation between objects are represented by template images in the parametric template space. The method also provides a subpixel matching scheme that yields a high-precision estimation of the object position. Experiments using real images have confirmed the effectiveness of the method.
Discriminative Generalized Hough Transform for Object Dectection
Authors: Ryuzo Okada
Venue: Conference ICCV 2009
Year: 2009

This paper present a part-based approach for detecting objects with large variation of appearance. We extract local image patches as local features both from the object and from the background in training images to learn an object part model discriminatively. Our object part model discriminates the local features whether they are an object part or not. Based on the discrimination results, each local feature casts probabilistic votes for the object location and size which are learned from the training images. Our object part model also requires regression performance for predicting the object location and size through the voting procedure. We build such an object part model with an ensemble of randomized trees trained by splitting each tree node so as to reduce the entropy of class label distribution and the variance of object location and size. Experimental results on hand detection with large pose variation show that our approach outperforms conventional generalized Hough transform. We verified the performance on a public dataset of side-view cars.
Non-homogeneous content-driven video-retargeting
Authors: Lior Wolf, Moshe Guttmann, Daniel Cohen-Or
Venue: Conference ICCV 2007
Year: 2007

Video retargeting is the process of transforming an existing video to fit the dimensions of an arbitrary display. A compelling retargeting aims at preserving the viewers' experience by maintaining the information content of important regions in the frame, whilst keeping their aspect ratio. An efficient algorithm for video retargeting is introduced. It consists of two stages. First, the frame is analyzed to detect the importance of each region in the frame. Then, a transformation that respects the analysis shrinks less important regions more than important ones. Our analysis is fully automatic and based on local saliency, motion detection and object detectors. The performance of the proposed algorithm is demonstrated on a variety of video sequences, and compared to the state of the art in image retargeting.