nils-herrmann/clean_scirepeval huggingface dataset:
Read, write, and navigation awareness in realistic multi-view collaborations

Read, write, and navigation awareness allow users of a multi-view collaborative editor to get fine-grained information about whether others are reading what they are editing, where others are editing, and to which areas of the document others have navigated, respectively. We derive new high-level tools that directly support the three kinds of awareness. The results of a decision-making study involving the use of these tools revealed that write awareness helps with fine-grained conflict prevention, read awareness reduces unnecessary verbal communication, and navigation awareness is most effective when coupled with read and write awareness. Thus, the study identifies specific uses of the three forms of awareness and motivates tools for supporting them.
A relevance feedback architecture for content-based multimedia information retrieval systems

Content-based multimedia information retrieval (MIR) has become one of the most active research areas in the past few years. Many retrieval approaches based on extracting and representing visual properties of multimedia data have been developed. While these approaches establish the viability of MIR based on visual features, techniques for incorporating human expertise directly during the query process to improve retrieval performance have not drawn enough attention. To address this limitation, this paper introduces a human-computer interaction based approach to MIR in which the user guides the system during retrieval using relevance feedback. Our experiments show that the retrieval performance improves significantly by incorporating humans in the retrieval process
Optimizing learning in image retrieval

Combining learning with vision techniques in interactive image retrieval has been an active research topic during the past few years. However, existing learning techniques either are based on heuristics or fail to analyze the working conditions. Furthermore, there is almost no in depth study on how to effectively learn from the users when there are multiple visual features in the retrieval system. To address these limitations, in this paper we present a vigorous optimization formulation of the learning process and solve the problem in a principled way. By using Lagrange multipliers, we have derived explicit solutions, which are both optimal and fast to compute. Extensive comparisons against state-of-the-art techniques have been performed. Experiments were carried out on a large-size heterogeneous image collection consisting of 17,000 images. Retrieval performance was tested under a wide range of conditions. Various evaluation criteria, including precision-recall curve and rank measure, have demonstrated the effectiveness and robustness of the proposed technique.
Partial-Duplicate Clustering and Visual Pattern Discovery on Web Scale Image Database

In this paper, we study the problem of discovering visual patterns and partial-duplicate images, which is fundamental to visual concept representation and image parsing, but very challenging when the database is extremely large, such as billions of images indexed by a commercial search engine. Although extensive research with sophisticated algorithms has been conducted for either partial-duplicate clustering or visual pattern discovery, most of them can not be easily extended to this scale, since both are clustering problems in nature and require pairwise comparisons. To tackle this computational challenge, we introduce a novel and highly parallelizable framework to discover partial-duplicate images and visual patterns in a unified way in distributed computing systems. We emphasize the nested property of local features, and propose the generalized nested feature (GNF) as a mid-level representation for regions and local patterns. Initial coarse clusters are then discovered by GNFs, upon which n-gram GNF is defined to represent co-occurrent visual patterns. After that, efficient merging and refining algorithms are used to get the partial-duplicate clusters, and logical combinations of probabilistic GNF models are leveraged to represent the visual patterns of partially duplicate images. Extensive experiments show the parallelizable property and effectiveness of the algorithms on both partial-duplicate clustering and visual pattern discovery. With 2000 machines, it costs about eight and 400 minutes to process one million and 40 million images respectively, which is quite efficient compared to previous methods.
Two-Dimensional Active Learning for image classification

In this paper, we propose a two-dimensional active learning scheme and show its application in image classification. Traditional active learning methods select samples only along the sample dimension. While this is the right strategy in binary classification, it is sub-optimal for multi-label classification. In multi-label classification, we argue that, for each selected sample, only a part of more effective labels are necessary to be annotated while others can be inferred by exploring the correlations among the labels. The reason is that the contributions of different labels to minimizing the classification error are different due to the inherent label correlations. To this end, we propose to select sample-label pairs, rather than only samples, to minimize a multi-label Bayesian classification error bound. This new active learning strategy not only considers the sample dimension but also the label dimension, and we call it Two-Dimensional Active Learning (2DAL). We also show that the traditional active learning formulation is a special case of 2DAL when there is only one label. Extensive experiments conducted on two real-world applications show that the 2DAL significantly outperforms the best existing approaches which did not take label correlation into account.
Learning of Multimodal Representations With Random Walks on the Click Graph

In multimedia information retrieval, most classic approaches tend to represent different modalities of media in the same feature space. With the click data collected from the users' searching behavior, existing approaches take either one-to-one paired data (text-image pairs) or ranking examples (text-query-image and/or image-query-text ranking lists) as training examples, which do not make full use of the click data, particularly the implicit connections among the data objects. In this paper, we treat the click data as a large click graph, in which vertices are images/text queries and edges indicate the clicks between an image and a query. We consider learning a multimodal representation from the perspective of encoding the explicit/implicit relevance relationship between the vertices in the click graph. By minimizing both the truncated random walk loss as well as the distance between the learned representation of vertices and their corresponding deep neural network output, the proposed model which is named multimodal random walk neural network (MRW-NN) can be applied to not only learn robust representation of the existing multimodal data in the click graph, but also deal with the unseen queries and images to support cross-modal retrieval. We evaluate the latent representation learned by MRW-NN on a public large-scale click log data set Clickture and further show that MRW-NN achieves much better cross-modal retrieval performance on the unseen queries/images than the other state-of-the-art methods.
Image Classification With Kernelized Spatial-Context

The goal of image classification is to classify a collection of unlabeled images into a set of semantic classes. Many methods have been proposed to approach this goal by leveraging visual appearances of local patches in images. However, the spatial context between these local patches also provides significant information to improve the classification accuracy. Traditional spatial contextual models, such as two-dimensional hidden Markov model, attempt to construct one common model for each image category to depict the spatial structures of the images in this class. However due to large intra-class variances in an image category, one single model has difficulties in representing various spatial contexts in different images. In contrast, we propose to construct a prototype set of spatial contextual models by leveraging the kernel methods rather than only one model. Such an algorithm combines the advantages of rich representation ability of spatial contextual models as well as the powerful classification ability of kernel method. In particular, we propose a new distance measure between different spatial contextual models by integrating joint appearance-spatial image features. Such a distance measure can be efficiently computed in a recursive formulation that scales well to image size. Extensive experiments demonstrate that the proposed approach significantly outperforms the state-of-the-art approaches.
MPEG-7 enhanced ubi-multimedia access —Convergence of user experience and technology

Unlike early versions of MPEG, MPEG-7 is shifting from a data-compression standard to a multimedia content description standard. As a result, it will have much broader implications for the human-computer interaction research. There are countless ways that the MPEG-7 metadata description framework can enhance userspsila experiences in accessing multimedia content. In this paper, we will report the algorithms that we developed for automatic generation of MPEG-7psilas description schemes (DSs): a video Table of Contents DS to enhance active video browsing, a Summary DS to enable direct use of a meta-data producerpsilas annotation, and a Still Image DS to allow interactive content-based image retrieval. Experimental results and user studies demonstrated the usefulness of MPEG-7psilas meta-data framework and effectiveness of our proposed automatic DS generation techniques.
Web conferencing systems for ubi-media collaboration: Achievements and challenges

Web conferencing systems (WCSs)have entered a golden age of rapid development, as evidenced by more than a dozen commercial products and an expanding consumer population in the past five years. While WCSs have revolutionized the way people communicate and collaborate, there is still a lot of room for improvement in both the technologies and the users experience. In this paper, we summarize the latest achievements in WCS design and implementation from both scientific research and technical engineering, identify open questions and key challenges that deserve more research attention, and discuss interesting directions to explore for better solutions. We envision this paper will stimulate more researchers to explore ubi-media collaboration via WCS.
Optimal radial contour tracking by dynamic programming

A common problem in most active contour methods is that the recursive searching scheme can only return a local optimal solution. Furthermore, the internal energy of the snake is not strong enough to control the shape of the contour. To overcome these difficulties, we develop a causal internal energy term based on a radial contour representation to encode the smooth constraint of the contour, and develop a global shape priori to control contour's shape and position based on object's dynamics. The causality nature of the representation allows us to efficiently find global optimal solution using dynamic programming. To validate the efficacy and robustness of the proposed approach, we apply this approach to track people in bad illumination and cluttered environments. We report promising results in the paper.
Exploring video structure beyond the shots

While existing shot-based video analysis approaches provide users with better access to the video than the raw data stream does, they are still not sufficient for meaningful video browsing and retrieval, since: (1) the shots in a long video are still too many to be presented to the user; and (2) shots do not capture the underlying semantic structure of the video, based on which the user may wish to browse/retrieve the video. To explore video structure at the semantic level this paper presents an effective approach for video scene structure construction, in which shots are grouped into semantic-related scenes. The output of the proposed algorithm provides a structured video that greatly facilitates user's access. Experiments based on real-world movie videos validate the effectiveness of the proposed approach.
Cross-Domain Human Action Recognition

Conventional human action recognition algorithms cannot work well when the amount of training videos is insufficient. We solve this problem by proposing a transfer topic model (TTM), which utilizes information extracted from videos in the auxiliary domain to assist recognition tasks in the target domain. The TTM is well characterized by two aspects: 1) it uses the bag-of-words model trained from the auxiliary domain to represent videos in the target domain; and 2) it assumes each human action is a mixture of a set of topics and uses the topics learned from the auxiliary domain to regularize the topic estimation in the target domain, wherein the regularization is the summation of Kullback-Leibler divergences between topic pairs of the two domains. The utilization of the auxiliary domain knowledge improves the generalization ability of the learned topic model. Experiments on Weizmann and KTH human action databases suggest the effectiveness of the proposed TTM for cross-domain human action recognition.
Secure information retrieval and reveal for mobile apparatus based on 2D Barcode digital signature

This paper introduces the mobile apparatus must make their best tradeoffs for the innate limitations of the physical properties and will benefit from the cloud services via wireless broadband access. Then a secure information retrieval and reveal framework based on two-dimensional barcode digital signature scheme is proposed for mobile apparatus. In addition to the secure protocol with cipher retrieval and barcode reveal presented, a case study is also presented to demonstrate how the framework is feasible and applicable especially for mobile commerce.
Multitask Autoencoder Model for Recovering Human Poses

Human pose recovery in videos is usually conducted by matching 2-D image features and retrieving relevant 3-D human poses. In the retrieving process, the mapping between images and poses is critical. Traditional methods assume this mapping relationship as local joint detection or global joint localization, which limits recovery performance of these methods since this two tasks are actually unified. In this paper, we propose a novel pose recovery framework by simultaneously learning the tasks of joint localization and joint detection. To obtain this framework, multiple manifold learning is used and the shared parameter is calculated. With them, multiple manifold regularizers are integrated and generalized eigendecomposition is utilized to achieve parameter optimization. In this way, pose recovery is boosted by both global mapping and local refinement. Experimental results on two popular datasets demonstrates that the recovery error has been reduced by 10%–20%, which proves the performance improvement of the proposed method.
Adaptive key frame extraction using unsupervised clustering

Key frame extraction has been recognized as one of the important research issues in video information retrieval. Although progress has been made in key frame extraction, the existing approaches are either computationally expensive or ineffective in capturing salient visual content. We first discuss the importance of key frame selection; and then review and evaluate the existing approaches. To overcome the shortcomings of the existing approaches, we introduce a new algorithm for key frame extraction based on unsupervised clustering. The proposed algorithm is both computationally simple and able to adapt to the visual content. The efficiency and effectiveness are validated by large amount of real-world videos.
Joint Multiview Segmentation and Localization of RGB-D Images Using Depth-Induced Silhouette Consistency

In this paper, we propose an RGB-D camera localization approach which takes an effective geometry constraint, i.e. silhouette consistency, into consideration. Unlike existing approaches which usually assume the silhouettes are provided, we consider more practical scenarios and generate the silhouettes for multiple views on the fly. To obtain a set of accurate silhouettes, precise camera poses are required to propagate segmentation cues across views. To perform better localization, accurate silhouettes are needed to constrain camera poses. Therefore the two problems are intertwined with each other and require a joint treatment. Facilitated by the available depth, we introduce a simple but effective silhouette consistency energy term that binds traditional appearance-based multiview segmentation cost and RGB-D frame-to-frame matching cost together. Optimization of the problem w.r.t. binary segmentation masks and camera poses naturally fits in the graph cut minimization framework and the Gauss-Newton non-linear least-squares method respectively. Experiments show that the proposed approach achieves state-of-the-arts performance on both tasks of image segmentation and camera localization.
Supporting content-based queries over images in MARS

While advances in technology allow us to generate, transmit and store large amounts of digital images, video and audio, research in the indexing and retrieval of multimedia information is still at its infancy. To address the challenges in building an effective multimedia database system, we have built the Multimedia Analysis and Retrieval System (MARS) prototype. This paper summarises the retrieval subsystem of MARS and its support for content-based queries over image features. Content-based retrieval techniques have been extensively studied for textual documents in the area of automatic information retrieval. Our objective in MARS is to exploit these existing techniques for content-based retrieval over images and multimedia databases.
Automated region segmentation using attraction-based grouping in spatial-color-texture space

A great deal of attention has been paid to image content-based retrieval systems (CBRS). One important goal in CBRS is to extract local low level image features such as color, texture and shape, to allow queries based on these features. A large CBRS containing tens of thousands of images requires an automatic feature-extraction method since human aided segmentation is impractical. We address this problem in a particular application setting by using an attraction-based grouping method in spatial-color-texture space. The attraction concept makes this approach similar to human aided segmentation. Experimental results show that the method is reasonably better than existing methods, and has the potential to be used in other CBRS-related applications.
High-Order Distance-Based Multiview Stochastic Learning in Image Classification

How do we find all images in a larger set of images which have a specific content? Or estimate the position of a specific object relative to the camera? Image classification methods, like support vector machine (supervised) and transductive support vector machine (semi-supervised), are invaluable tools for the applications of content-based image retrieval, pose estimation, and optical character recognition. However, these methods only can handle the images represented by single feature. In many cases, different features (or multiview data) can be obtained, and how to efficiently utilize them is a challenge. It is inappropriate for the traditionally concatenating schema to link features of different views into a long vector. The reason is each view has its specific statistical property and physical interpretation. In this paper, we propose a high-order distance-based multiview stochastic learning (HD-MSL) method for image classification. HD-MSL effectively combines varied features into a unified representation and integrates the labeling information based on a probabilistic framework. In comparison with the existing strategies, our approach adopts the high-order distance obtained from the hypergraph to replace pairwise distance in estimating the probability matrix of data distribution. In addition, the proposed approach can automatically learn a combination coefficient for each view, which plays an important role in utilizing the complementary information of multiview data. An alternative optimization is designed to solve the objective functions of HD-MSL and obtain different views on coefficients and classification scores simultaneously. Experiments on two real world datasets demonstrate the effectiveness of HD-MSL in image classification.
PASS: Peer-Aware Silence Suppression for Internet Voice Conferences

A novel tandem-free solution for multiparty VoIP conferences called PASS (peer-aware silence suppression) is presented. Similar to traditional tandem-free solutions, PASS introduces a limit on the number of concurrent speakers in a conference. But in contrast to traditional solutions, PASS silence suppression and speaker selection are completely distributed, running on each client. No speaker selection is performed at the bridge at all. This configuration leads to better scalability, lower bandwidth occupation and jitter buffer delay, and higher compatibility with a wide variety of network topologies. The key component of PASS, distributed silence suppression and speaker selection, is realized through a robust approach proposed in this paper. Based on a voice activity measure derived using machine learning techniques, this approach is able to reliably suppress silence in complex environments, and perform accurate and transparent speaker selection as well
Parametric contour tracking using unscented Kalman filter

This paper presents an efficient method to integrate various spatial-temporal constraints to regularize the contour tracking. Specifically, the global shape prior, contour smoothness and object dynamics are addressed. First, the contour is represented as a parametric shape, based on which a causal smoothness constraint can be developed to exploit the local spatial constraint. The causality nature of the constraint allows us to do efficient probabilistic contour detection using the powerful hidden Markov model (HMM). Finally, a unscented Kalman filter (UKF) is applied to estimate object parameters based on the nonlinear observation model (i.e. the relationship between the detected contour points and the contour parameters) and the object dynamics. Better than other variants of the recursive least mean square estimators (e.g., extended Kalman filter), the UKF approximates nonlinear systems up to the second order (third for Gaussian prior) with similar computational cost. This novel tracking algorithm is running in real-time and robust to severe distractions due to the comprehensive spatial-temporal constraints. It is applied to track people in bad illumination and cluttered environments. Promising results are reported.
Content-based image retrieval with relevance feedback in MARS

Technology advances in the areas of image processing (IP) and information retrieval (IR) have evolved separately for a long time. However, successful content-based image retrieval systems require the integration of the two. There is an urgent need to develop integration mechanisms to link the image retrieval model to text retrieval model, such that the well established text retrieval techniques can be utilized. Approaches of converting image feature vectors (IF domain) to weighted-term vectors (IR domain) are proposed in this paper. Furthermore, the relevance feedback technique from the IR domain is used in content-based image retrieval to demonstrate the effectiveness of this conversion. Experimental results show that the image retrieval precision increases considerably by using the proposed integration approach.