OpenRaiser/Intern-Atlas huggingface dataset:
Understanding human preferences: Towards more personalized video to text generation
Authors: Yihan Wu, Ruihua Song, Xu Chen, Hao Jiang, Zhao Cao, Jin Yu
Venue: Conference WWW 2024
Year: 2024

While previous video to text models have achieved remarkable successes, they mostly focus on how to understand the video contents in a general sense, but fail to capture the human personalized preferences, which is highly demanded for an engaging multimodal chatbots. Different from user modeling in collaborative filtering, there is no other user behaviors in inference as a real-time video stream is coming. In this paper, we formally define the task of personalized video commenting task and design an end-to-end personalized framework for solving this task. In specific, we argue that the personalization for video comment generation can be reflected in two aspects, that is, (1) for the same video, different users may comment on different clips, and (2) for the same clip, different people may also express various opinions with diverse commentary styles. Motivated by these considerations, we design our framework based on two components. The first one is a clip selector, which is responsible for predicting the clips that the user may comment in the video. The second one is a text generator, which aims to produce the comment based on the above predicted clips and the user's preference. In our framework, these two components are optimized in an end-to-end manner to mutually enhance each other, where we design confidence-aware scheduled sampling and iterative inference strategies to solve the problem that the ground truth clips are absent in the inference phase. As the absence of personalized video to text dataset, we collect and release a new dataset for studying this problem. We conduct extensive experiments to demonstrate the effectiveness of our model.
MathWriting: A dataset for handwritten mathematical expression recognition
Authors: Philippe Gervais, Anastasiia Fadeeva, Andrii Maksai
Venue: Conference KDD 2025
Year: 2025

Recognition of handwritten mathematical expressions allows to transfer scientific notes into their digital form. It facilitates the sharing, searching, and preservation of scientific information. We introduce MathWriting, the largest online handwritten mathematical expression dataset to date. It consists of 230k human-written samples and an additional 400k synthetic ones}. This dataset can also be used in its rendered form for offline HME recognition. One MathWriting sample consists of a formula written on a touch screen and a corresponding LaTeX expression. We also provide a normalized version of LaTeX expression to simplify the recognition task and enhance the result quality. We provide baseline performance of standard models like OCR and CTC Transformer as well as Vision-Language Models like PaLI on the dataset. The dataset together with an example colab is accessible on Github.
Learning Flexible Sprites in Video Layers
Authors: Nebojsa Jojic, Brendan J. Frey
Venue: Conference CVPR 2001
Year: 2001

We propose a technique for automatically learning layers of "flexible sprites" (probabilistic 2-dimensional appearance maps and masks of moving, occluding objects). The model explains each input image as a layered composition of flexible sprites. A variational expectation maximization algorithm is used to learn a mixture of sprites from a video sequence. For each input image, probabilistic inference is used to infer the sprite class, translation, mask values and pixel intensities (including obstructed pixels) in each layer. Exact inference is intractable, but we show how a variational inference technique can be used to process 320/spl times/240 images at 1 frame/second. The only inputs to the learning algorithm are the video sequence, the number of layers and the number of flexible sprites. We give results on several tasks, including summarizing a video sequence with sprites, point-and-click video stabilization, and point-and-click object removal.
Unist: a prompt-empowered universal model for urban spatio-temporal prediction
Authors: Yuan Yuan 0032, Jingtao Ding, Jie Feng 0002, Depeng Jin, Yong Li 0008
Venue: Conference KDD 2024
Year: 2024

Urban spatio-temporal prediction is crucial for informed decision-making, such as traffic management, resource optimization, and emergence response. Despite remarkable breakthroughs in pretrained natural language models that enable one model to handle diverse tasks, a universal solution for spatio-temporal prediction remains challenging Existing prediction approaches are typically tailored for specific spatio-temporal scenarios, requiring task-specific model designs and extensive domain-specific training data. In this study, we introduce UniST, a universal model designed for general urban spatio-temporal prediction across a wide range of scenarios. Inspired by large language models, UniST achieves success through: (i) utilizing diverse spatio-temporal data from different scenarios, (ii) effective pre-training to capture complex spatio-temporal dynamics, (iii) knowledge-guided prompts to enhance generalization capabilities. These designs together unlock the potential of building a universal model for various scenarios Extensive experiments on more than 20 spatio-temporal scenarios demonstrate UniST's efficacy in advancing state-of-the-art performance, especially in few-shot and zero-shot prediction. The datasets and code implementation are released on https://github.com/tsinghua-fib-lab/UniST.
Combining scene and auto-calibration constraints
Authors: David Liebowitz, Andrew Zisserman
Venue: Conference ICCV 1999
Year: 1999

We present a simple approach to combining scene and auto-calibration constraints for the calibration of cameras from single views and stereo pairs. Calibration constraints are provided by imaged scene structure, such as vanishing points of orthogonal directions, or rectified planes. In addition, constraints are available from the nature of the cameras and the motion between views. We formulate these constraints in terms of the geometry of the imaged absolute conic and its relationship to pole-polar pairs and the imaged circular points of planes. Three significant advantages result: first, constraints from scene features, camera characteristics and auto-calibration constraints provide linear equations in the elements of the image of the absolute conic. This means that constraints may easily be combined, and their solution is straightforward. Second, the degeneracies that occur when constraints are not independent may be easily identified. Lastly, the constraints from scene planes and image planes may be treated uniformly. Examples of various cases of constraint combination and degeneracy as well as computational techniques are presented.
Detecting rotational symmetries
Authors: V. Shiv Naga Prasad, Larry S. Davis
Venue: Conference ICCV 2005
Year: 2005

We present an algorithm for detecting multiple rotational symmetries in natural images. Given an image, its gradient magnitude field is computed, and information from the gradients is spread using a diffusion process in the form of a gradient vector flow (GVF) field. We construct a graph whose nodes correspond to pixels in tire image, connecting points that are likely to be rotated versions of one another The n-cycles present in tire graph are made to vote for C/sub n/ symmetries, their votes being weighted by the errors in transformation between GVF in the neighborhood of the voting points, and the irregularity of the n-sided polygons formed by the voters. The votes are accumulated at tire centroids of possible rotational symmetries, generating a confidence map for each order of symmetry. We tested the method with several natural images.
Doclaynet: A large human-annotated dataset for document-layout segmentation
Authors: Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S. Nassar, Peter W. J. Staar
Venue: Conference KDD 2022
Year: 2022

Accurate document layout analysis is a key requirement for high-quality PDF document conversion. With the recent availability of public, large ground-truth datasets such as PubLayNet and DocBank, deep-learning models have proven to be very effective at layout detection and segmentation. While these datasets are of adequate size to train such models, they severely lack in layout variability since they are sourced from scientific article repositories such as PubMed and arXiv only. Consequently, the accuracy of the layout segmentation drops significantly when these models are applied on more challenging and diverse layouts. In this paper, we presentDocLayNet, a new, publicly available, document-layout annotation dataset in COCO format. It contains 80863 manually annotated pages from diverse data sources to represent a wide variability in layouts. For each PDF page, the layout annotations provide labelled bounding-boxes with a choice of 11 distinct classes. DocLayNet also provides a subset of double- and triple-annotated pages to determine the inter-annotator agreement. In multiple experiments, we provide baseline accuracy scores (in mAP) for a set of popular object detection models. We also demonstrate that these models fall approximately 10% behind the inter-annotator agreement. Furthermore, we provide evidence that DocLayNet is of sufficient size. Lastly, we compare models trained on PubLayNet, DocBank and DocLayNet, showing that layout predictions of the DocLayNet-trained models are more robust and thus the preferred choice for general-purpose document-layout analysis.
Neuropose: 3d hand pose tracking using emg wearables
Authors: Yilin Liu, Shijia Zhang, Mahanth Gowda
Venue: Conference WWW 2021
Year: 2021

Ubiquitous finger motion tracking enables a number of exciting applications in augmented reality, sports analytics, rehabilitation-healthcare, haptics etc. This paper presents NeuroPose, a system that shows the feasibility of 3D finger motion tracking using a platform of wearable ElectroMyoGraphy (EMG) sensors. EMG sensors can sense electrical potential from muscles due to finger activation, thus offering rich information for fine-grained finger motion sensing. However converting the sensor information to 3D finger poses is non trivial since signals from multiple fingers superimpose at the sensor in complex patterns. Towards solving this problem, NeuroPose fuses information from anatomical constraints of finger motion with machine learning architectures on Recurrent Neural Networks (RNN), Encoder-Decoder Networks, and ResNets to extract 3D finger motion from noisy EMG data. The generated motion pattern is temporally smooth as well as anatomically consistent. Furthermore, a transfer learning algorithm is leveraged to adapt a pretrained model on one user to a new user with minimal training overhead. A systematic study with 12 users demonstrates a median error of 6.24° and a 90%-ile error of 18.33° in tracking 3D finger joint angles. The accuracy is robust to natural variation in sensor mounting positions as well as changes in wrist positions of the user. NeuroPose is implemented on a smartphone with a processing latency of 0.101s, and a low energy overhead.
Bringing multimodality to amazon visual search system
Authors: Xinliang Zhu, Sheng-Wei Huang, Han Ding 0004, Jinyu Yang, Kelvin Chen, Tao Zhou, Tal Neiman, Ouye Xie, Son Tran, Benjamin Z. Yao, Douglas Gray 0001, Anuj Bindal, Arnab Dhua
Venue: Conference KDD 2024
Year: 2024

Image to image matching has been well studied in the computer vision community. Previous studies mainly focus on training a deep metric learning model matching visual patterns between the query image and gallery images. In this study, we show that pure image-to- image matching suffers from false positives caused by matching to local visual patterns. To alleviate this issue, we propose to leverage recent advances in vision-language pretraining research. Specifically, we introduce additional image-text alignment losses into deep metric learning, which serve as constraints to the image-to-image matching loss. With additional alignments between the text (e.g., product title) and image pairs, the model can learn concepts from both modalities explicitly, which avoids matching low-level visual features. We progressively develop two variants, a 3-tower and a 4-tower model, where the latter takes one more short text query input. Through extensive experiments, we show that this change leads to a substantial improvement to the image to image matching problem. We further leveraged this model for multimodal search, which takes both image and reformulation text queries to improve search quality. Both offline and online experiments show strong improvements on the main metrics. Specifically, we see 4.95% relative improvement on image matching click through rate with the 3-tower model and 1.13% further improvement from the 4-tower model.
Class-based grouping in perspective images
Authors: Andrew Zisserman, Joseph L. Mundy, David A. Forsyth, Jane Liu, Nic Pillow, Charlie Rothwell, Sven Utcke
Venue: Conference ICCV 1995
Year: 1995

In any object recognition system a major and primary task is to associate those image features, within an image of a complex scene, that arise from an individual object. The key idea here is that a geometric class defined in 3D induces relationships in the image which must hold between points on the image outline (the perspective projection of the object). The resulting image constraints enable both identification and grouping of image features belonging to objects of that class. The classes include surfaces of revolution, canal surfaces (pipes) and polyhedra. Recognition proceeds by first recognising an object as belonging to one of the classes (for example a surface of revolution) and subsequently identifying the object (for example as a particular vase). This differs from conventional object recognition systems where recognition is generally targetted at particular objects. These classes also support the computation of 3D invariant descriptions including symmetry axes, canonical coordinate frames and projective signatures. The constraints and grouping methods are viewpoint invariant, and proceed with no information on object pose. We demonstrate the effectiveness of this class-based grouping on real, cluttered scenes using grouping algorithms developed for rotationally symmetric surfaces, canal-surfaces and polyhedra.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">></ETX>
Color constancy in diagonal chromaticity space
Authors: Graham D. Finlayson
Venue: Conference ICCV 1995
Year: 1995

Simple constraints on the sets of possible surface reflectances and illuminants are exploited in a new color constancy algorithm that builds upon Forsyth's (1990) theory of color constancy. The goal defined for a color constancy algorithm is to discount variations in the color and intensity of the incident illumination and thereby extract illumination-independent descriptors of surface colors from images. Forsyth's method is based on two constraints: first, the surface colors under a canonical illuminant all fall within an established maximal convex gamut of possible colors and second that a diagonal matrix accurately maps colors between illuminants. These constraints taken together turn out to be very effective in solving for color constancy; however, other strong assumptions about the scenes are required for the method to work-the illumination must be uniform, the surfaces must be planar, and there can be no specularities. We show that these restrictions are necessary only because Forsyth sets out to recover the intensity of descriptors. At the outset we abandon 3-dimensional descriptor recovery in favor of recovering only orientation (i.e. 2 dimensions). Intensity information is factored out of the problem by mapping 3-dimensional (r, g, b) camera responses onto 2-dimensional chromaticities; specifically (r/b, g/b). We show that this "diagonal chromaticity space" has two important properties: first, gamut convexity is preserved and second illumination change is still described by a diagonal matrix. It follows that Forsyth's algorithm can be directly applied to the recover chromaticity descriptors and from these the 3D descriptor orientations can be derived. The basic algorithm is then extended to include a maximal gamut constraint on the set of illuminants that is analogous to the gamut constraint on surface colors. The diagonal chromaticity space facilitates the expression of the illumination constraint in the algorithm. Tests on real images show that the algorithm provides good color constancy.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">></ETX>
Automatic indexing based on bayesian inference networks
Authors: Kostas Tzeras, Stephan Hartmann 0004
Venue: Conference SIGIR 1993
Year: 1993

In this paper, a Bayesian inference network model for automatic indexing with index terms (descriptors) from a prescribed vocabulary is presented. It requires an indexing dictionary with rules mapping terms of the respective subject field onto descriptors and inverted lists for terms occuring in a set of documents of the subject field and descriptors manually assigned to these documents. The indexing dictionary can be derived automatically from a set of manually indexed documents. An application of the network model is described, followed by an indexing example and some experimental results about the indexing performance of the network model.
A toolkit for knowledge base population
Authors: Zheng Chen, Suzanne Tamang, Adam Lee, Heng Ji
Venue: Conference SIGIR 2011
Year: 2011

The main goal of knowledge base population (KBP) is to distill entity information (e.g., facts of a person) from multiple unstructured and semi-structured data sources, and incorporate the information into a knowledge base (KB). In this work, we intend to release an open source KBP toolkit that is publicly available for research purposes.
Topic-conditioned novelty detection
Authors: Yiming Yang, Jian Zhang, Jaime G. Carbonell, Chun Jin
Venue: Conference KDD 2002
Year: 2002

Automated detection of the first document reporting each new event in temporally-sequenced streams of documents is an open challenge. In this paper we propose a new approach which addresses this problem in two stages: 1) using a supervised learning algorithm to classify the on-line document stream into pre-defined broad topic categories, and 2) performing topic-conditioned novelty detection for documents in each topic. We also focus on exploiting named-entities for event-level novelty detection and using feature-based heuristics derived from the topic histories. Evaluating these methods using a set of broadcast news stories, our results show substantial performance gains over the traditional one-level approach to the novelty detection problem.
Enhancing cluster labeling using Wikipedia
Authors: David Carmel, Haggai Roitman, Naama Zwerdling
Venue: Conference SIGIR 2009
Year: 2009

This work investigates cluster labeling enhancement by utilizing Wikipedia, the free on-line encyclopedia. We describe a general framework for cluster labeling that extracts candidate labels from Wikipedia in addition to important terms that are extracted directly from the text. The "labeling quality" of each candidate is then evaluated by several independent judges and the top evaluated candidates are recommended for labeling.
Less is more: probabilistic models for retrieving fewer relevant documents
Authors: Harr Chen, David R. Karger
Venue: Conference SIGIR 2006
Year: 2006

Traditionally, information retrieval systems aim to maximize the number of relevant documents returned to a user within some window of the top. For that goal, the probability ranking principle, which ranks documents in decreasing order of probability of relevance, is provably optimal. However, there are many scenarios in which that ranking does not optimize for the users information need. One example is when the user would be satisfied with some limited number of relevant documents, rather than needing all relevant documents. We show that in such a scenario, an attempt to return many relevant documents can actually reduce the chances of finding any relevant documents.
Spectral geometry for simultaneously clustering and ranking query search results
Authors: Ying Liu, Wenyuan Li, Yongjing Lin, Liping Jing
Venue: Conference SIGIR 2008
Year: 2008

How best to present query search results is an important problem in search engines and information retrieval systems. When a single query retrieves many results, simply showing them as a long list will provide users with poor overview. Nowadays, ranking and clustering query search results have been two useful separate post-processing techniques to organize retrieved documents. In this paper, we proposed a spectral analysis method based on the content similarity networks to integrate the clustering and ranking techniques for improving literature search. The new approach organizes all these search results into categories intelligently and simultaneously rank the results in each category. A variety of theoretical and empirical studies have demonstrated that the presented method performs well in real applications, especially in biomedical literature retrieval. Moreover, any free text information can be analyzed with the new method, i.e., the proposed approach can be applied to various information systems, such as Web search engines and literature search service.
Noun sense induction using web search results
Authors: Goldee Udani, Shachi Dave, Anthony Davis, Tim Sibley
Venue: Conference SIGIR 2005
Year: 2005

This paper presents an algorithm for unsupervised noun sense induction, based on clustering of Web search results. The algorithm does not utilize labeled training instances or any other external knowledge source. Preliminary results on a small dataset show that this technique provides two advantages over other techniques in the literature: it detects real-world senses not found in dictionaries or other lexical resources, and it does not require that the number of word senses be specified in advance.
A comparative evaluation of different link types on enhancing document clustering
Authors: Xiaodan Zhang, Xiaohua Hu, Xiaohua Zhou
Venue: Conference SIGIR 2008
Year: 2008

With a growing number of works utilizing link information in enhancing document clustering, it becomes necessary to make a comparative evaluation of the impacts of different link types on document clustering. Various types of links between text documents, including explicit links such as citation links and hyperlinks, implicit links such as co-authorship links, and pseudo links such as content similarity links, convey topic similarity or topic transferring patterns, which is very useful for document clustering. In this study, we adopt a Relaxation Labeling (RL)-based clustering algorithm, which employs both content and linkage information, to evaluate the effectiveness of the aforementioned types of links for document clustering on eight datasets. The experimental results show that linkage is quite effective in improving content-based document clustering. Furthermore, a series of interesting findings regarding the impacts of different link types on document clustering are discovered through our experiments.
Pet: a statistical model for popular events tracking in social communities
Authors: Cindy Xide Lin, Bo Zhao, Qiaozhu Mei, Jiawei Han
Venue: Conference KDD 2010
Year: 2010

User generated information in online communities has been characterized with the mixture of a text stream and a network structure both changing over time. A good example is a web-blogging community with the daily blog posts and a social network of bloggers.