nils-herrmann/clean_scirepeval huggingface dataset:
Retrieval of commercials by video semantics

Videos convey information through several planes of communication, encompassing what is represented in the images how the images are linked together and how the subject is imaged. This feature is stressed in commercials where colors, editing effects, rhythms, and object motion are exploited to influence human purchasing habits. In this paper, based on researches in the marketing field, a link is formalized between low level features of a commercial video and feelings that the video would inspire in the observer. This link is used to define high level indices capturing the main semantics of the video. These indices are embedded in a video retrieval system to support access to a database of video based on their semantics.
Automatic caption localization in videos using salient points

Broadcasters are demonstrating interest in building digital archives of their assets for reuse of archive materials for TV programs, on-line availability, and archiving. This requires tools for video indexing and retrieval by content exploiting high-level video information such as that contained in super-imposed text captions. In this paper we present a method to automatically detect and localize captions in digital video using temporal and spatial local properties of salient points in video frames. Results of experiments on both high-resolutionDV sequences and standard VHS videos are presented and discussed.
Computer analysis of TV spots: the semiotics perspective

Commercial videos exploit simultaneously several planes of expression (colors, editing effects, viewpoints and rhythms, etc.) to encode the message being vehiculated to the audience in the most persuasive way. In this paper, previous research in the marketing and semiotics fields is translated into a multimedia engineering perspective by formalizing a set of rules mapping visual features at the perceptual level onto higher level features capturing the main narrative structures of the spot. The rules are embedded in a video retrieval system supporting access to a database of commercials based either on semiotic video categories or on global similarity with respect to a template spot.
Content based annotation and retrieval of news videos

Effective retrieval and browsing by content of videos is based on high level information associated with visual data. Automatic extraction of high level content descriptors, requires to exploit the semantical characteristics of video types. In this paper a complete system for content based annotation and retrieval of news videos is presented, indexing of the video stream is fully automated and is based both on visual features extracted from video shots and on textual strings extracted from captions and speech.
Deep Generative Adversarial Compression Artifact Removal

Compression artifacts arise in images whenever a lossy compression algorithm is applied. These artifacts eliminate details present in the original image, or add noise and small structures; because of these effects they make images less pleasant for the human eye, and may also lead to decreased performance of computer vision algorithms such as object detectors. To eliminate such artifacts, when decompressing an image, it is required to recover the original image from a disturbed version. To this end, we present a feed-forward fully convolutional residual network model trained using a generative adversarial framework. To provide a baseline, we show that our model can be also trained optimizing the Structural Similarity (SSIM), which is a better loss with respect to the simpler Mean Squared Error (MSE). Our GAN is able to produce images with more photorealistic details than MSE or SSIM based networks. Moreover we show that our approach can be used as a pre-processing step for object detection in case images are degraded by compression to a point that state-of-the art detectors fail. In this task, our GAN method obtains better performance than MSE or SSIM trained networks.
Effective Codebooks for human action categorization

In this paper we propose a new method for human action categorization by using an effective combination of novel gradient and optic flow descriptors, and creating a more effective codebook modeling the ambiguity of feature assignment in the traditional bag-of-words model. Recent approaches have represented video sequences using a bag of spatio-temporal visual words, following the successful results achieved in object and scene classification. Codebooks are usually obtained by k-means clustering and hard assignment of visual features to the best representing codeword. Our main contribution is two-fold. First, we define a new 3D gradient descriptor that combined with optic flow outperforms the state-of-the-art, without requiring fine parameter tuning. Second, we show that for spatio-temporal features the popular k-means algorithm is insufficient because cluster centers are attracted by the denser regions of the sample distribution, providing a non-uniform description of the feature space and thus failing to code other informative regions. Therefore, we apply a radius-based clustering method and a soft assignment that considers the information of two or more relevant candidates. This approach generates a more effective codebook resulting in a further improvement of classification performances. We extensively test our approach on standard KTH and Weizmann action datasets showing its validity and outperforming other recent approaches.
Advanced man-machine interface for cultural heritage

This paper describes a system for advanced man-machine interaction based on computer vision technology allowing users to manipulate information displayed on large wall panels by using their own hands as pointing devices. The system gets its input from a pair of color video cameras (placed so as to have the user in view); a personal computer performs image analysis and updates interaction parameters. Such parameters, which reflect current user status, are then transformed into graphic interface commands and output to the screen through a beamer. Graphic interface operation is the same as with standard computer mice; the main points of innovation concern naturality of interaction, low intrusiveness and a priori training, and low equipment cost. An experimental version of the system is currently being employed to provide museum visitors with advanced interactive services.
Demo paper: Stamat: A framework for Social Topics and Media Analysis

Analysis of trending topics in social networks, based both on textual and multimedia content, can be used by content providers to measure the buzz around their channels, and even to create new material based on the current memes propagating across Twitter, Google+, Facebook etc. STAMAT is a framework designed to provide a set of tools for social media analysis, and to create, through content curation, personalized web sites and magazines.
Specification by-Example of Virtual Agents Behavior

The development of virtual agents running within graphic environments which emulate real-life contexts may largely benefit from the use of visual specification by-example. To support this specification, the development system must be able to interpret the examples and cast their underlying rules into an internal representation language. This language must find a suitable trade-off among a number of contrasting requirements regarding expressiveness, automatic executability, and suitability to the automatic representation of rules deriving from the analysis of examples. A language is presented which attains this trade-off by combining together an operational and a declarative fragment to separately represent the autonomous execution of each individual agent and its interaction with the environment, respectively. While the declarative part permits to capture interaction rules emerging from specification examples, the operational part supports the automatic execution in the operation of the virtual environment. A system is presented which embeds this language within a visual shell to support a behavioral training in which the animation rules of virtual agents are defined through visual examples.
Retrieval by content of commercials based on dynamics of color flows

The huge amount of data associated with videos complicates the extraction of salient information content descriptors from low level video features. Commercials are a video category where a large part of the content depends on low level perceptual features such as colors and color dynamics. These are related to changes-in terms of shrinking, growth and translation-of colored regions over consecutive frames. By changing its shape and position over consecutive frames, each region defines a 3D volume: a color flow. In this paper, a system is presented that allows retrieval of commercials based on color flow similarity. The system relies on automatic extraction of color flows, and description of color flows based on 3D wavelet decomposition.
MuseumVisitors: A dataset for pedestrian and group detection, gaze estimation and behavior understanding

In this paper we describe a new dataset, under construction, acquired inside the National Museum of Bargello in Florence. It was recorded with three IP cameras at a resolution of 1280 × 800 pixels and an average framerate of five frames per second. Sequences were recorded following two scenarios. The first scenario consists of visitors watching different artworks (individuals), while the second one consists of groups of visitors watching the same artworks (groups). This dataset is specifically designed to support research on group detection, occlusion handling, tracking, re-identification and behavior analysis. In order to ease the annotation process we designed a user friendly web interface that allows to annotate: bounding boxes, occlusion area, body orientation and head gaze, group belonging, and artwork under observation. We provide a comparison with other existing datasets that have group and occlusion annotations. In order to assess the difficulties of this dataset we have also performed some tests exploiting seven representative state-of-the-art pedestrian detectors.
Local shape estimation from a single keypoint

This paper presents a novel approach to estimate local homography of points belong to a given surface. While others works attempt this by using iterative algorithms developed for template matching, our method introduces a direct estimation of the transformation. It performs the following steps. First, a training set of features captures appearance and geometry information about keypoints taken from multiple views of the surface. Then incoming keypoints are matched against the training set in order to retrieve a cluster of features representing their identity. Finally the retrieved clusters are used to estimate the local pose of the regions around keypoints. Thanks to the high accuracy, outliers and bad estimates are filtered out by multiscale Summed Square Difference (SSD) test.
Metric 3D reconstruction and texture acquisition of surfaces of revolution from a single uncalibrated view

Image analysis and computer vision can be effectively employed to recover the three-dimensional structure of imaged objects, together with their surface properties. In this paper, we address the problem of metric reconstruction and texture acquisition from a single uncalibrated view of a surface of revolution (SOR). Geometric constraints induced in the image by the symmetry properties of the SOR structure are exploited to perform self-calibration of a natural camera, 3D metric reconstruction, and texture acquisition. By exploiting the analogy with the geometry of single axis motion, we demonstrate that the imaged apparent contour and the visible segments of two imaged cross sections in a single SOR view provide enough information for these tasks. Original contributions of the paper are: single view self-calibration and reconstruction based on planar rectification, previously developed for planar surfaces, has been extended to deal also with the SOR class of curved surfaces; self-calibration is obtained by estimating both camera focal length (one parameter) and principal point (two parameters) from three independent linear constraints for the SOR fixed entities; the invariant-based description of the SOR scaling function has been extended from affine to perspective projection. The solution proposed exploits both the geometric and topological properties of the transformation that relates the apparent contour to the SOR scaling function. Therefore, with this method, a metric localization of the SOR occluded parts can be made, so as to cope with them correctly. For the reconstruction of textured SORs, texture acquisition is performed without requiring the estimation of external camera calibration parameters, but only using internal camera parameters obtained from self-calibration.
Natural Interfaces to Enhance Visitors' Experiences

Museums and exhibitions don't communicate. These places are often just a collection of objects, standing deaf in front of visitors. In many cases, objects are accompanied by textual descriptions, usually too short or long to be useful for the visitor. In the last decade, progress in multimedia has allowed for new, experimental forms of communication (using computer technologies) in public spaces. Implementations have ranged from simply using standard PCs with multimedia applications that show thumbnails of image data integrated with text to large theaters that immerse users in virtual worlds or reproduce and display 3D models of masterpieces. Often designers just apply the technology available to traditional museum schemes, without paying much attention to the visitor's experience, particularly to the ways they expect users to interact with the system or to the cognitive and aesthetic factors involved.
A mosaic-based query language for video databases

Content based video retrieval, CBVR, is an active research topic. To make this feasible, the development of a query language which is simple (i.e. intuitive), yet powerful, is essential. The authors propose a mosaic based visual language for CBVR. Mosaicing is a well known technique in computer vision and graphics for creating a comprehensive overview of a scene reproduced in a set of images. The language supports the specification of queries encompassing motion of objects and camera, as well as visual features, by arranging visual components on a mosaic image. The proposed language was then used to implement a graphical interface that allows users to specify and refine queries to a video database.
Retrieval of 3D objects using curvature maps and weighted walkthroughs

In the field of content-based retrieval of multimedia documents, 3D models have recently gained increasing attention for a number of reasons: advancements in 3D hardware and software technologies; their ever decreasing prices and increasing availability; affordable 3D authoring tools; the establishment of such open standards for 3D data interchange as VRML and X3D. We present a solution for content-based retrieval of 3D models that relies on a description of each model by means of a curvature map. After an initial preprocessing of the model, differential properties of points on the surface of the 3D object are evaluated; the model surface is then warped into an ellipsoid, and is mapped onto a 2D image, retaining the curvature information of the original model. Matching is performed by comparing the map of the query against the maps of database models. In particular, this approach exploits weighted walkthroughs to describe curvature maps. Experimental results reveal that this technique, initially developed to describe images, can be successfully applied to curvature maps.
Current Multimedia Data Formats and Semantic Computing: A Practical Example and the Challenges for the Future

Research is doing exciting progress in understanding images as more than just a set of pixels, videos as more than just an even greater set of pixels synchronized with time, or audio streams as more than just an array of samples. Current implementations of multimedia data formats, such as MPEG, however, still focus on the storage of pixels and samples. This article uses a straightforward example of a system where a pixel-based video stream is superimposed on a vector-based animation to show that the lack of capability to store higher-level semantic information leaves the developers only two choices: Either accept suboptimal quality or design a new non-standard format. The paper presents an in-depth discussion of a practical example and draws conclusions for the future directions of multimedia data formats.
Spatio-Temporal Closed-Loop Object Detection

Object detection is one of the most important tasks of computer vision. It is usually performed by evaluating a subset of the possible locations of an image, that are more likely to contain the object of interest. Exhaustive approaches have now been superseded by object proposal methods. The interplay of detectors and proposal algorithms has not been fully analyzed and exploited up to now, although this is a very relevant problem for object detection in video sequences. We propose to connect, in a closed-loop, detectors and object proposal generator functions exploiting the ordered and continuous nature of video sequences. Different from tracking we only require a previous frame to improve both proposal and detection: no prediction based on local motion is performed, thus avoiding tracking errors. We obtain three to four points of improvement in mAP and a detection time that is lower than Faster Regions with CNN features (R-CNN), which is the fastest Convolutional Neural Network (CNN) based generic object detector known at the moment.
Transport measurements over an Ethernet LAN

The performance evaluation of transport protocols is a relevant step for designing efficient distributed applications. This analysis can be performed either with an unloaded medium or in the presence of background load. In this latter case, the effects of the interaction with other users are made evident, but remarkable difficulties exist to obtain a reproducible network load and to achieve accuracy in measurements. In this paper, performance measurements in the presence of background load are presented for TCP and UDP in the Internet family. And SPP and IDP in the Xerox Network System family. Experiments were carried out over an Ethernet LAN with MS-DOS AT-class stations.
Visual capture and understanding of hand pointing actions in a 3-D environment

We present a nonintrusive system based on computer vision for human-computer interaction in three-dimensional (3-D) environments controlled by hand pointing gestures. Users are allowed to walk around in a room and manipulate information displayed on its walls by using their own hands as pointing devices. Once captured and tracked in real-time using stereo vision, hand pointing gestures are remapped onto the current point of interest, thus reproducing in an advanced interaction scenario the "drag and click" behavior of traditional mice. The system, called PointAt (patent pending), enjoys a careful modeling of both user and optical subsystem, and visual algorithms for self-calibration and adaptation to both user peculiarities and environmental changes. The concluding sections provide an insight into system characteristics, performance, and relevance for real applications.
Compact Hash Codes for Efficient Visual Descriptors Retrieval in Large Scale Databases

In this paper, we present an efficient method for visual descriptors retrieval based on compact hash codes computed using a multiple k-means assignment. The method has been applied to the problem of approximate nearest neighbor (ANN) search of local and global visual content descriptors, and it has been tested on different datasets: three large scale standard datasets of engineered features of up to one billion descriptors (BIGANN) and, supported by recent progress in convolutional neural networks (CNNs), on CIFAR-10, MNIST, INRIA Holidays, Oxford 5K, and Paris 6K datasets; also, the recent DEEP1B dataset, composed by one billion CNN-based features, has been used. Experimental results show that, despite its simplicity, the proposed method obtains a very high performance that makes it superior to more complex state-of-the-art methods.
Group Re-identification via Unsupervised Transfer of Sparse Features Encoding

Person re-identification is best known as the problem of associating a single person that is observed from one or more disjoint cameras. The existing literature has mainly addressed such an issue, neglecting the fact that people usually move in groups, like in crowded scenarios. We believe that the additional information carried by neighboring individuals provides a relevant visual context that can be exploited to obtain a more robust match of single persons within the group. Despite this, re-identifying groups of people compound the common single person re-identification problems by introducing changes in the relative position of persons within the group and severe self-occlusions. In this paper, we propose a solution for group re-identification that grounds on transferring knowledge from single person reidentification to group re-identification by exploiting sparse dictionary learning. First, a dictionary of sparse atoms is learned using patches extracted from single person images. Then, the learned dictionary is exploited to obtain a sparsity-driven residual group representation, which is finally matched to perform the re-identification. Extensive experiments on the i-LIDS groups and two newly collected datasets show that the proposed solution outperforms stateof-the-art approaches.
Content-Based Retrieval of 3-D Objects Using Spin Image Signatures

Retrieval by content of 3D models is becoming more and more important due to the advancements in 3D hardware and software technologies for acquisition, authoring and display of 3D objects, their ever-increasing availability at affordable costs, and the establishments of open standards for 3D data interchange. In this paper, we present a new method, referred to as spin image signatures, that develops on the original spin images approach, with adaptations to support effective retrieval by content. According to the method proposed, a set of spin images is derived for each model, to obtain a view-independent description of its 3D shape and a signature is evaluated for each spin image in the set. Clustering is hence performed on the set of spin image signatures to obtain a compact representation. Experimental results are presented, showing the effectiveness of the spin image signatures method for retrieval, also in comparison with other methods, and its sensitivity to model deformations
A Three-Dimensional Iconic Environment for Image Database Querying

Retrieval by contents of images from pictorial databases can be effectively performed through visual icon-based systems. In these systems, the representation of pictures with 2D strings, which are derived from symbolic projections, provides an efficient and natural way to construct iconic indexes for pictures and is also an ideal representation for the visual query. With this approach, retrieval is reduced to matching two symbolic strings. However, using 2D-string representations, spatial relationships between the objects represented in the image might not be exactly specified. Ambiguities arise for the retrieval of images of 3D scenes. In order to allow the unambiguous description of object spatial relationships, in this paper, following the symbolic projections approach, images are referred to by considering spatial relationships in the 3D imaged scene. A representation language is introduced that expresses positional and directional relationships between objects in three dimensions, still preserving object spatial extensions after projections. Iconic retrieval from pictorial databases with 3D interfaces is discussed and motivated. A system for querying by example with 3D icons, which supports this language, is also presented.