OpenRaiser/Intern-Atlas huggingface dataset:
The cortical activity of graded relevance
Authors: Zuzana Pinkosova, William J. McGeown, Yashar Moshfeghi
Venue: Conference SIGIR 2020
Year: 2020

Relevance is an essential concept in Information Retrieval (IR). Recent studies using brain imaging have significantly contributed towards the understanding of this concept, but only as a binary notion, i.e. a document being judged as relevant or non-relevant. While such a binary division is prevalent in IR, seminal theories have proposed relevance as a graded variable; i.e. having different degrees. In this paper, we aim to investigate the brain activity associated with relevance when it is treated as a graded concept. Twenty-five participants provided graded relevance judgements in the context of a Question Answering (Q/A) Task, during assessment with an electroencephalogram (EEG). Our findings show that significant differences in event-related potentials (ERPs) were observed in response to information segments processed in the context of high-relevance, low-relevance and no-relevance, supporting the concept of graded relevance. We speculate that differences in attentional engagement, semantic mismatch (between the question and answer) and memory processing underpin the electrophysiological responses to the graded relevance judgements. We believe our conclusions constitute an important step in unravelling the nature of graded relevance and knowledge of the electrophysiological modulation to each grade of relevance will help to improve the design and evaluation of IR systems.
Beyond sliding windows: Object localization by efficient subwindow search
Authors: Christoph H. Lampert, Matthew B. Blaschko, Thomas Hofmann 0001
Venue: Conference CVPR 2008
Year: 2008

Most successful object recognition systems rely on binary classification, deciding only if an object is present or not, but not providing information on the actual object location. To perform localization, one can take a sliding window approach, but this strongly increases the computational cost, because the classifier function has to be evaluated over a large set of candidate subwindows. In this paper, we propose a simple yet powerful branch-and-bound scheme that allows efficient maximization of a large class of classifier functions over all possible subimages. It converges to a globally optimal solution typically in sublinear time. We show how our method is applicable to different object detection and retrieval scenarios. The achieved speedup allows the use of classifiers for localization that formerly were considered too slow for this task, such as SVMs with a spatial pyramid kernel or nearest neighbor classifiers based on the chi <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -distance. We demonstrate state-of-the-art performance of the resulting systems on the UIUC Cars dataset, the PASCAL VOC 2006 dataset and in the PASCAL VOC 2007 competition.
Translation symmetry detection in a fronto-parallel view
Authors: Peng Zhao, Long Quan
Venue: Conference CVPR 2011
Year: 2011

In this paper, we present a method of detecting translation symmetries from a fronto-parallel image. The proposed method automatically detects unknown multiple repetitive patterns of arbitrary shapes, which are characterized by translation symmetries on a plane. The central idea of our approach is to take advantage of the interesting properties of translation symmetries in both image space and the space of transformation group. We first detect feature points in input image as sampling points. Then for each sampling point, we search for the most probable corresponding lattice structures in the image and transform spaces using scale-space similarity maps. Finally, using a MRF formulation, we optimally partition the graph of all sampling points associated with the estimated lattices into subgraphs of sampling points and lattices belonging to the same symmetry pattern. Our method is robust because of the joint analysis in image and transform spaces, and the MRF optimization. We demonstrate the robustness and effectiveness of our method on a large variety of images.
Kinecting the dots: Particle based scene flow from depth sensors
Authors: Simon Hadfield, Richard Bowden
Venue: Conference ICCV 2011
Year: 2011

The motion field of a scene can be used for object segmentation and to provide features for classification tasks like action recognition. Scene flow is the full 3D motion field of the scene, and is more difficult to estimate than it's 2D counterpart, optical flow. Current approaches use a smoothness cost for regularisation, which tends to over-smooth at object boundaries. This paper presents a novel formulation for scene flow estimation, a collection of moving points in 3D space, modelled using a particle filter that supports multiple hypotheses and does not oversmooth the motion field. In addition, this paper is the first to address scene flow estimation, while making use of modern depth sensors and monocular appearance images, rather than traditional multi-viewpoint rigs. The algorithm is applied to an existing scene flow dataset, where it achieves comparable results to approaches utilising multiple views, while taking a fraction of the time.
Learning complex motions by sequencing simpler motion templates
Authors: Gerhard Neumann, Wolfgang Maass 0001, Jan Peters 0001
Venue: Conference ICML 2009
Year: 2009

Abstraction of complex, longer motor tasks into simpler elemental movements enables humans and animals to exhibit motor skills which have not yet been matched by robots. Humans intuitively decompose complex motions into smaller, simpler segments. For example when describing simple movements like drawing a triangle with a pen, we can easily name the basic steps of this movement.
Enhancing photographs with near infra-red images
Authors: Xiaopeng Zhang 0001, Terence Sim, Xiaoping Miao
Venue: Conference CVPR 2008
Year: 2008

Near Infra-Red (NIR) images of natural scenes usually have better contrast and contain rich texture details that may not be perceived in visible light photographs (VIS). In this paper, we propose a novel method to enhance a photograph by using the contrast and texture information of its corresponding NIR image. More precisely, we first decompose the NIR/VIS pair into average and detail wavelet subbands. We then transfer the contrast in the average subband and transfer texture in the detail subbands. We built a special camera mount that optically aligns two consumer-grade digital cameras, one of which was modified to capture NIR. Our results exhibit higher visual quality than tone-mapped HDR images, showing that NIR imaging is useful for computational photography.
Color lines: image specific color representation
Authors: Ido Omer, Michael Werman
Venue: Conference CVPR 2004
Year: 2004

The problem of deciding whether two pixels in an image have the same real world color is a fundamental problem in computer vision. Many color spaces are used in different applications for discriminating color from intensity to create an informative representation of color. The major drawback of all of these representations is that they assume no color distortion. In practice the colors of real world images are distorted both in the scene itself and in the image capturing process. In this work we introduce color lines, an image specific color representation that is robust to color distortion and provides a compact and useful representation of the colors in a scene.
Fancy: human-centered, deep learning-based framework for fashion style analysis
Authors: Youngseung Jeon, Seungwan Jin, Kyungsik Han
Venue: Conference WWW 2021
Year: 2021

Fashion style analysis is of the utmost importance for fashion professionals. However, it has an issue of having different style classification criteria that rely heavily on professionals’ subjective experiences with no quantitative criteria. We present FANCY (Fashion Attributes detectioN for Clustering stYle), a human-centered, deep learning-based framework to support fashion professionals’ analytic tasks using a computational method integrated with their insights. We work closely with fashion professionals in the whole study process to reflect their domain knowledge and experience as much as possible. We redefine fashion attributes, demonstrate a strong association with fashion attributes and styles, and develop a deep learning model that detects attributes in a given fashion image and reflects fashion professionals’ insight. Based on attribute-annotated 302,772 runway fashion images, we developed 25 new fashion styles (FANCY dataset 1). We summarize quantitative standards of the fashion style groups and present fashion trends based on time, location, and brand.
A new convolution kernel for atmospheric point spread function applied to computer vision
Authors: Samy Metari, François Deschênes
Venue: Conference ICCV 2007
Year: 2007

In this paper we introduce a new filter to approximate multiple scattering of light rays within a participating media. This filter is derived from the generalized Gaussian distribution GGD. It characterizes the Atmospheric Point Spread Function (APSF) and thus makes it possible to introduce three new approaches. First, it allows us to accurately simulate various weather conditions that induce multiple scattering including fog, haze, rain, etc. Second, it allows us to propose a new method for a cooperative and simultaneous estimation of visual cues, i.e., the identification of weather degradations and the estimation of optical thickness between two images of the same scene acquired under unknown weather conditions. Third, by combining this filter with two new sets of invariant features we recently developed, we obtain invariant features that can be used for the matching of atmospheric degraded images. The first set leads to atmospheric invariant features while the second one simultaneously provides atmospheric and geometric invariance.
Demystifying illegal mobile gambling apps
Authors: Yuhao Gao, Haoyu Wang, Li Li, Xiapu Luo, Guoai Xu, Xuanzhe Liu
Venue: Conference WWW 2021
Year: 2021

Mobile gambling app, as a new type of online gambling service emerging in the mobile era, has become one of the most popular and lucrative underground businesses in the mobile app ecosystem. Since its born, mobile gambling app has received strict regulations from both government authorities and app markets. However, to the best of our knowledge, mobile gambling apps have not been investigated by our research community. In this paper, we take the first step to fill the void. Specifically, we first perform a 5-month dataset collection process to harvest illegal gambling apps in China, where mobile gambling apps are outlawed. We have collected 3,366 unique gambling apps with 5,344 different versions. We then characterize the gambling apps from various perspectives including app distribution channels, network infrastructure, malicious behaviors, abused third-party and payment services. Our work has revealed a number of covert distribution channels, the unique characteristics of gambling apps, and the abused fourth-party payment services. At last, we further propose a “guilt-by-association” expansion method to identify new suspicious gambling services, which help us further identify over 140K suspicious gambling domains and over 57K gambling app candidates. Our study demonstrates the urgency for detecting and regulating illegal gambling apps.
Bayesian imitation learning for end-to-end mobile manipulation
Authors: Yuqing Du, Daniel Ho, Alex Alemi, Eric Jang, Mohi Khansari
Venue: Conference ICML 2022
Year: 2022

In this work we investigate and demonstrate benefits of a Bayesian approach to imitation learning from multiple sensor inputs, as applied to the task of opening office doors with a mobile manipulator. Augmenting policies with additional sensor inputs, such as RGB + depth cameras, is a straightforward approach to improving robot perception capabilities, especially for tasks that may favor different sensors in different situations. As we scale multi-sensor robotic learning to unstructured real-world settings (e.g. offices, homes) and more complex robot behaviors, we also increase reliance on simulators for cost, efficiency, and safety. Consequently, the sim-to-real gap across multiple sensor modalities also increases, making simulated validation more difficult. We show that using the Variational Information Bottleneck (Alemi et al., 2016) to regularize convolutional neural networks improves generalization to held-out domains and reduces the sim-to-real gap in a sensor-agnostic manner. As a side effect, the learned embeddings also provide useful estimates of model uncertainty for each sensor. We demonstrate that our method is able to help close the sim-to-real gap and successfully fuse RGB and depth modalities based on understanding of the situational uncertainty of each sensor. In a real-world office environment, we achieve 96% task success, improving upon the baseline by +16%.
3d building detection and modeling from aerial lidar data
Authors: Vivek Verma, Rakesh Kumar 0001, Stephen C. Hsu
Venue: Conference CVPR 2006
Year: 2006

This paper presents a method to detect and construct a 3D geometric model of an urban area with complex buildings using aerial LIDAR (Light Detection and Ranging) data. The LIDAR data collected from a nadir direction is a point cloud containing surface samples of not only the building roofs and terrain but also undesirable clutter from trees, cars, etc. The main contribution of this work is the automatic recognition and estimation of simple parametric shapes that can be combined to model very complex buildings from aerial LIDAR data. The main components of the detection and modeling algorithms are (i) Segmentation of roof and terrain points. (ii) Roof topology Inference. We introduce the concept of a roof-topology graph to represent the relationships between the various planar patches of a complex roof structure. (iii) Parametric roof composition. Simple parametric roof shapes that can be combined to create a complex roof structure of a building are recognized by searching for sub-graphs in its roof-topology graph. (iv) Terrain Modeling. The terrain is identified and modeled as a triangulated mesh. Finally, we provide experimental results that demonstrate the validity of our approach for rapid and automatic building detection and geometric modeling with real LIDAR data. We are able to model cities and other urban areas at the rate of about 10 minutes per sq. mile on a low-end PC.
Neural collapse inspired debiased representation learning for min-max fairness
Authors: Shenyu Lu, Junyi Chai 0004, Xiaoqian Wang 0001
Venue: Conference KDD 2024
Year: 2024

Although machine learning algorithms demonstrate impressive performance, their trustworthiness remains a critical issue, particularly concerning fairness when implemented in real-world applications. Many notions of group fairness aim to minimize disparities in performance across protected groups. However, it can inadvertently reduce performance in certain groups, leading to sub-optimal outcomes. In contrast, Min-max group fairness notion prioritizes the improvement for the worst-performing group, thereby advocating a utility-promoting approach to fairness. However, it has been proven that existing efforts to achieve Min-max fairness exhibit limited effectiveness. In response to this challenge, we leverage the recently proposed "Neural Collapse'' framework to re-examine Empirical Risk Minimization (ERM) training, specifically investigating the root causes of poor performance in minority groups. The layer-peeled model is employed to decompose a network into two parts: an encoder to learn latent representation, and a subsequent classifier, with a systematic characterization of their training behaviors being conducted. Our analysis reveals that while classifiers achieve maximum separation, the separability of representations is insufficient, particularly for minority groups. This indicates the sub-optimal performance in minority groups stems from less separable representations, rather than classifiers. To tackle this issue, we introduce a novel strategy that incorporates a frozen classifier to directly enhance representation. Furthermore, we introduce two easily implemented loss functions to guide the learning process. The experimental assessments carried out on real-world benchmark datasets spanning the domains of Computer Vision, Natural Language Processing, and Tabular data demonstrate that our approach outperforms existing state-of-the-art methods in promoting the Min-max fairness notion.
Ensemble pretrained models for multimodal sentiment analysis using textual and video data fusion
Authors: Zhicheng Liu, Ali Braytee, Ali Anaissi, Guifu Zhang, Lingyun Qin, Junaid Akram
Venue: Conference WWW 2024
Year: 2024

We introduce an ensemble model approach for multimodal sentiment analysis, focusing on the fusion of textual and video data to enhance the accuracy and depth of emotion interpretation. By integrating three foundational models-IFFSA, BFSA, and TBJE-using advanced ensemble techniques, we achieve a significant improvement in sentiment analysis performance across diverse datasets, including MOSI and MOSEI. Specifically, we propose two novel models-IFFSA and BFSA, which utilise the large language models BERT and GPT-2 to extract the features from text modality and ResNet and VGG for video modality. Our work uniquely contributes to the field by demonstrating the synergistic potential of combining different modal analytical strengths, thereby addressing the intricate challenge of nuanced emotion detection in multimodal contexts. Through comprehensive experiments and an extensive ablation study, we not only validate the superior performance of our ensemble model against current state-of-the-art benchmarks but also reveal critical insights into the model's capability to discern complex emotional states. Our findings underscore the strategic advantage of ensemble methods in multimodal sentiment analysis and set a new precedent for future research in effectively integrating multimodal data sources.
Automatic photo-to-terrain alignment for the annotation of mountain pictures
Authors: Lionel Baboud, Martin Cadík, Elmar Eisemann, Hans-Peter Seidel
Venue: Conference CVPR 2011
Year: 2011

We present a system for the annotation and augmentation of mountain photographs. The key issue resides in the registration of a given photograph with a 3D geo-referenced terrain model. Typical outdoor images contain little structural information, particularly mountain scenes whose aspect changes drastically across seasons and varying weather conditions. Existing approaches usually fail on such difficult scenarios. To avoid the burden of manual registration, we propose a novel automatic technique. Given only a viewpoint and FOV estimates, the technique is able to automatically derive the pose of the camera relative to the geometric terrain model. We make use of silhouette edges, which are among most reliable features that can be detected in the targeted situations. Using an edge detection algorithm, our technique then searches for the best match with silhouette edges rendered using the synthetic model. We develop a robust matching metric allowing us to cope with the inevitable noise affecting detected edges (e.g. due to clouds, snow, rocks, forests, or any phenomenon not encoded in the digital model). Once registered against the model, photographs can easily be augmented with annotations (e.g. topographic data, peak names, paths), which would otherwise imply a tedious fusion process. We further illustrate various other applications, such as 3D model-assisted image enhancement, or, inversely, texturing of digital models.
Measuring Image Flow by Tracking Edge-Lines
Authors: James L. Crowley, Patrick Stelmaszyk, Christophe Discours
Venue: Conference ICCV 1988
Year: 1988

This paper describes a technique for measuring the movement of edge-lines in a sequence of images by maintalning an image plane model. Edge-lines are expressed as a set of parameter vectors representing the center-point, orientation and length of a segment. Each parameter vector is composed of an estimate, a temporal derivative, and their covariance matrix. Line segment parameters in the flow model are updated using a Kalman filter. The eorrespondance of observed edge-lines segments to segments predicted from the flow model is determined by a linear complexity algorithm using distance normalized by covariance. The existence of segments in the flow model is controlled using a confidence factor. This technique is in everyday use as part of a larger system for building 3-D scene descriptions using a camera mounted on a robot arm. A near video-rate hardware implementation is currently under development
A Frequency-based and a Poisson-based Definition of the Probability of Being Informative
Authors: Thomas Rölleke
Venue: Conference SIGIR 2003
Year: 2003

This paper reports on theoretical investigations about the assumptions underlying the inverse document frequency (idf). We show that an intuitive idf-based probability function for the probability of a term being informative assumes disjoint document events. By assuming documents to be independent rather than disjoint, we arrive at a Poisson-based probability of being informative. The framework is useful for understanding and deciding the parameter estimation and combination in probabilistic retrieval models.
On Arabic search: improving the retrieval effectiveness via a light stemming approach
Authors: Mohammed Aljlayl, Ophir Frieder
Venue: Conference CIKM 2002
Year: 2002

The inflectional structure of a word impacts the retrieval accuracy of information retrieval systems of Latin-based languages. We present two stemming algorithms for Arabic information retrieval systems. We empirically investigate the effectiveness of surface-based retrieval. This approach degrades retrieval precision since Arabic is a highly inflected language. Accordingly, we propose root-based retrieval. We notice a statistically significant improvement over the surface-based approach. Many variant word senses are based on an identical root; thus, the root-based algorithm creates invalid conflation classes that result in an ambiguous query which degrades the performance by adding extraneous terms. To resolve ambiguity, we propose a novel light-stemming algorithm for Arabic texts. This automatic rule-based stemming algorithm is not as aggressive as the root extraction algorithm. We show that the light stemming algorithm significantly outperforms the root-based algorithm. We also show that a significant improvement in retrieval precision can be achieved with light inflectional analysis of Arabic words.
Navigating the intranet with high precision
Authors: Huaiyu Zhu, Sriram Raghavan, Shivakumar Vaithyanathan, Alexander Löser
Venue: Conference WWW 2007
Year: 2007

Despite the success of web search engines, search over large enterprise intranets still suffers from poor result quality. Earlier work [6] that compared intranets and the Internet from the view point of keyword search has pointed to several reasons why the search problem is quite different in these two domains. In this paper, we address the problem of providing high quality answers to navigational queries in the intranet (e.g., queries intended to find product or personal home pages, service pages, etc.). Our approach is based on offline identification of navigational pages, intelligent generation of term-variants to associate with each page, and the construction of separate indices exclusively devoted to answering navigational queries. Using a testbed of 5.5M pages from the IBM intranet, we present evaluation results that demonstrate that for navigational queries, our approach of using custom indices produces results of significantly higher precision than those produced by a general purpose search algorithm.