nils-herrmann/clean_scirepeval huggingface dataset:
Novel variational approach for the perspective shape from shading problem using calibrated images

In this paper, We propose a new approach for solving the Lambertian shape from shading (SFS) problem. The proposed technique employs image information as well as camera calibration parameters. We claim that the incorporation of camera parameters results in a better surface reconstruction. In this work, we slightly modify this more realistic model in order to take into account the attenuation of the illumination due to distance. We propose a new variational formulation that relates an evolving surface model with image information, taking into consideration that the image is taken by a perspective camera with known parameters. A new energy functional is formulated to incorporate brightness, smoothness, and integrability constraints. All of these terms assume a hyper surface that evolves in time to meet their criteria. Gradient descent optimization with Euler-Lagrange is used for optimization. We constructed a database of real surfaces as a ground-truth for the experiments. The proposed approach shows superior performance when compared with other approaches in the literature.
Optical flow-based enhancement of spatio-temporal detection in videos

Accurate optical flow techniques are widely used in spatio-temporal object detection in videos. However, the computational complexity of the currently used techniques limits the effectiveness of spatio-temporal detection in applications such as action detection and event recognition. Therefore, in this paper we aim at employing rapid yet accurate optical flow techniques to promote the effectiveness of the detection system. The proposed design uses novel optical flow estimation techniques that are based on learned flow basis, known as PCA-Flow and PCA-Layers. PCA-Flow estimates dense flow from a linear flow model based on principle components of natural flow. PCA-Layers is an extension of PCA-Flow. PCA-Layers technique uses Markov random field (MRF) to combine several motion layers into dense optical flow. The motion in each layer is estimated by PCA-Flow. Our experimental results show that our approach maintains the overall performance of the baseline framework while 64% reduction in the computation time is achieved.
Variational approach for segmentation of lung nodules

Lung nodules from low dose CT (LDCT) scans may be used for early detection of lung cancer. However, these nodules vary in size, shape, texture, location, and may suffer from occlusion within the tissue. This paper presents an approach for segmentation of lung nodules detected by a prior step. First, regions around the detected nodules are segmented; using automatic seed point placement levels sets. The outline of the nodule region is further improved using the curvature characteristics of the segmentation boundary. We illustrate the effectiveness of this method for automatic segmentation of the Juxta-pleural nodules.
Probabilistic shape-based segmentation method using level sets

In this study, a novel probabilistic, geometric and dynamic shape-based level sets method is proposed. The shape prior is coupled with the intensity information to enhance the segmentation results. The two-dimensional principal component analysis method is applied on the training shapes to represent the shape variation with enough number of shape projections in the training step. The shape model is constructed using the implicit representation of the projected shapes. A new energy functional is proposed (i) to embed the shape model into the image domain and (ii) to estimate the shape coefficients. The proposed method is validated on synthetic and clinical images with various challenges such as the noise, occlusion and missing information. The authors compare their method with some of related works. Experiments show that the proposed segmentation method is more accurate and robust than other alternatives under different challenges.  Colour figures are available in the online version of this paper.
A kidney segmentation approach from DCE-MRI using level sets

Acute rejection is the most common reason of graft failure after kidney transplantation, and early detection is crucial to survive the transplanted kidney function. Automatic classification of normal and acute rejection transplants from dynamic contrast enhanced magnetic resonance imaging (DCEMRI), is of great importance. Kidney segmentation is the first step for such classification. The image intensity inside the kidney is used as an indication of failure/success. Differentiating between different cases cases is implemented by comparing subsequential kidney scans signals. So, this process is mainly dependent on segmentation. This paper introduces a new shape-based segmentation approach based on level sets. Training shapes are collected from different real data sets to represent the shape variations. Signed distance functions are used to represent these shapes. The methodology incorporates image and shape prior information in a variational framework. The shape registration is considered the backbone of the approach where more general transformations can be used to handle the process. We introduce a novel shape dissimilarity measure that enables the use of different (inhomogeneous) scales. The approach gives successful results compared with other techniques restricted to transformations with homogeneous scales. Results for segmenting kidney images will be illustrated and compared with other approaches to show the efficiency of the proposed technique.
Active microscopic cellular image annotation by superposable graph transduction with imbalanced labels

Systematic content screening of cell phenotypes in microscopic images has been shown promising in gene function understanding and drug design. However, manual annotation of cells and images in genome-wide studies is cost prohibitive. In this paper, we propose a highly efficient active annotation framework, in which a small amount of expert input is leveraged to rapidly and effectively infer the labels over the remaining unlabeled data. We formulate this as a graph based transductive learning problem and develop a novel method for label propagation. Specifically, a label regularizer method is proposed to handle the important label imbalance issue, typically seen in the cellular image screening applications. We also design a new scheme which breaks the graph into linear superposition of contributions from individual labeled samples. We take advantage of such a superposable representation to achieve fast annotation in an interactive setting. Extensive evaluations over toy data and realistic cellular images confirm the superiority of the proposed method over existing alternatives.
Statistical fusion of multiple cues for image tampering detection

Image forensic research has resulted in a number of tampering detection techniques utilizing cues culled from understanding of various natural image characteristics and modeling of tampering artifacts. Fusion of multiple cues provides promises for improving the detection robustness, however has never been systematically studied before. By fusing multiple cues, the tampering detection process does not rely entirely on a single detector and hence can be robust in face of missing or unreliable detectors. In this paper, we propose a statistical fusion framework based on discriminative random fields (DRF) to integrate multiple cues suitable for forgery detection, such as double quantization artifacts and camera response function inconsistency. The detection results using individual cues are used as observation from which the DRF model parameters and the most likely node labels are inferred indicating whether a local block belongs to a tampered foreground or the authentic background. Such inference results also provide information about localization of the suspect spliced regions. The proposed framework is effective and general - outperforming individual detectors over systematic evaluation and easily extensible to other detectors using different cues.
WebClip: a WWW video editing/browsing system

WebClip is a complete working prototype for editing/browsing MPEG-1 and MPEG-2 compressed video distributively over the World Wide Web. It uses a general system architecture to store, retrieve, and edit MPEG-1 or MPEG-2 compressed video over the network. It uses an innovative distributed network support architecture. It also uses our unique CVEPS (Compressed video editing, parsing, and search) technologies.
Assistive Image Comment Robot—A Novel Mid-Level Concept-Based Representation

We present a general framework and working system for predicting likely affective responses of the viewers in the social media environment after an image is posted online. Our approach emphasizes a mid-level concept representation, in which intended affects of the image publisher is characterized by a large pool of visual concepts (termed PACs) detected from image content directly instead of textual metadata, evoked viewer affects are represented by concepts (termed VACs) mined from online comments, and statistical methods are used to model the correlations among these two types of concepts. We demonstrate the utilities of such approaches by developing an end-to-end Assistive Comment Robot application, which further includes components for multi-sentence comment generation, interactive interfaces, and relevance feedback functions. Through user studies, we showed machine suggested comments were accepted by users for online posting in 90 percent of completed user sessions, while very favorable results were also observed in various dimensions (plausibility, preference, and realism) when assessing the quality of the generated image comments.
A statistical framework for fusing mid-level perceptual features in news story segmentation

News story segmentation is essential for video indexing, summarization and intelligence exploitation. In this paper, we present a general statistical framework, called exponential model or maximum entropy model that can systematically select the most significant mid-level features of various types (visual, audio, and semantic) and learn the optimal ways in fusing their combinations in story segmentation. The model utilizes a family of weighted, exponential functions to account for the contributions from different features. The Kullbak-Leibler divergence measure is used in an optimization procedure to iteratively estimate the model parameters, and automatically select the optimal features. The framework is scalable in incorporating new features and adapting to new domains and also discovers how these feature sets contribute to the segmentation work. When tested on foreign news programs, the proposed techniques achieve significant performance improvement over prior work using ad hoc algorithms and slightly better gain over the state of the art using HMM-based models.
Label Propagation from ImageNet to 3D Point Clouds

Recent years have witnessed a growing interest in understanding the semantics of point clouds in a wide variety of applications. However, point cloud labeling remains an open problem, due to the difficulty in acquiring sufficient 3D point labels towards training effective classifiers. In this paper, we overcome this challenge by utilizing the existing massive 2D semantic labeled datasets from decade-long community efforts, such as Image Net and Label Me, and a novel ``cross-domain'' label propagation approach. Our proposed method consists of two major novel components, Exemplar SVM based label propagation, which effectively addresses the cross-domain issue, and a graphical model based contextual refinement incorporating 3D constraints. Most importantly, the entire process does not require any training data from the target scenes, also with good scalability towards large scale applications. We evaluate our approach on the well-known Cornell Point Cloud Dataset, achieving much greater efficiency and comparable accuracy even without any 3D training data. Our approach shows further major gains in accuracy when the training data from the target scenes is used, outperforming state-of-the-art approaches with far better efficiency.
Region feature based similarity searching of semantic video objects

New video representations based on semantic objects (e.g., MPEG-4) provide great potential for content-based video searching. In this paper we present an efficient query model for similarity searching of video objects based on localized region features and spatial-temporal structures. The query model is barred on an existing framework on region-level video query, but addresses several new issues like right spatio-temporal relationships effective performance of the proposed approach.
Video mining: pattern discovery versus pattern recognition

We examine the significance of video mining as pattern discovery in multimedia content. We examine the underlying issue of pattern discovery versus pattern recognition, since most past work has not drawn such a sharp distinction. We argue that while the term "pattern discovery" implies a purely unsupervised approach, in practice a mixture of unsupervised and supervised techniques will have to be used. We compare conventional data mining with video mining and observe that a key difference is in the multilayered semantics of multimedia content. We then identify significant challenges posed by video mining.
Deep Image Set Hashing

In applications involving matching of image sets, the information from multiple images must be effectively exploited to represent each set. State-of-the-art methods use probabilistic distribution or subspace to model a set and use specific distance measure to compare two sets. These methods are slow to compute and not compact to use in a large scale scenario. Learning-based hashing is often used in large scale image retrieval as they provide a compact representation of each sample and the Hamming distance can be used to efficiently compare two samples. However, most hashing methods encode each image separately and discard knowledge that multiple images in the same set represent the same object or person. We investigate the set hashing problem by combining both set representation and hashing in a single deep neural network. An image set is first passed to a CNN module to extract image features, then these features are aggregated using two types of set feature to capture both set specific and database-wide distribution information. The computed set feature is then fed into a multilayer perceptron to learn a compact binary embedding trained with triplet loss. We extensively evaluate our approach on multiple image datasets and show highly competitive performance compared to state-of-the-art methods.
Content-based dynamic resource allocation for VBR video in bandwidth limited networks

In bandwidth-limited access networks, dynamic resource allocation can increase the link utilization and decrease the required network buffering. In this paper, we propose a new policy in which the visual content of video is used to determine the bandwidth needed for its transmission. The policy is based on a novel framework of content-based video traffic modeling. Based on simulation results, the proposed content-based dynamic resource allocation scheme has shown that a reduction of 55% to 70% in network resources could be reached, as compared to existing approaches (e.g. RVBR and RCBR).
An Exploration of Parameter Redundancy in Deep Networks with Circulant Projections

We explore the redundancy of parameters in deep neural networks by replacing the conventional linear projection in fully-connected layers with the circulant projection. The circulant structure substantially reduces memory footprint and enables the use of the Fast Fourier Transform to speed up the computation. Considering a fully-connected neural network layer with d input nodes, and d output nodes, this method improves the time complexity from O(d2) to O(dlogd) and space complexity from O(d2) to O(d). The space savings are particularly important for modern deep convolutional neural network architectures, where fully-connected layers typically contain more than 90% of the network parameters. We further show that the gradient computation and optimization of the circulant projections can be performed very efficiently. Our experiments on three standard datasets show that the proposed approach achieves this significant gain in storage and efficiency with minimal increase in error rate compared to neural networks with unstructured projections.
Color-mood analysis of films based on syntactic and psychological models

The emergence of peer-to-peer networking and the increase of home PC storage capacity are necessitating efficient scaleable methods for video clustering, recommending and browsing. Based on film theories and psychological models, color-mood is an important factor affecting user emotional preferences. We propose a compact set of features for color-mood analysis and subgenre discrimination. We introduce two color representations for scenes and full films in order to extract the essential moods from the films: a global measure for the color palette and a discriminative measure for the transitions of the moods in the movie. We captured the dominant color ratio and the pace of the movie. Despite the simplicity and efficiency of the features, the classification accuracy was surprisingly good, about 80%, possibly thanks to the prevalence of the color-mood association in feature films
A fully automated content-based video search engine supporting spatiotemporal queries

The rapidity with which digital information, particularly video, is being generated has necessitated the development of tools for efficient search of these media. Content-based visual queries have been primarily focused on still image retrieval. In this paper, we propose a novel, interactive system on the Web, based on the visual paradigm, with spatiotemporal attributes playing a key role in video retrieval. We have developed innovative algorithms for automated video object segmentation and tracking, and use real-time video editing techniques while responding to user queries. The resulting system, called VideoQ , is the first on-line video search engine supporting automatic object-based indexing and spatiotemporal queries. The system performs well, with the user being able to retrieve complex video clips such as those of skiers and baseball players with ease.
Recent Advances and Challenges of Semantic Image/Video Search

We present an overview of recent advances and major challenges in image and video search, with a specific focus on large-scale semantic concept detection and indexing. Such semantic indexing paradigm has been driven by the increasing availability of the large resources of corpora, novel labeling approaches, innovative image features, and machine learning techniques for visual content recognition. We will discus key approaches, recent results, and novel applications in text-to-concept semantic search and multi-modal retrieval models. Open issues and major opportunities are also presented.
Compressed-domain techniques for image/video indexing and manipulation

As massive amount of visual materials are captured and stored in visual information systems, effective and efficient image indexing and manipulation techniques are required. Most visual materials in visual information systems are stored in some compressed forms. Therefore, it is desirable to explore image technologies for feature extraction and image manipulation in the compressed domain. In other words, image feature extraction and manipulation are performed on compressed images/video without decoding, or with minimal decoding only. Although the compressed-domain approach imposes many constraints, it provides great potential for reducing computational complexity, because of reduction of the amount of data after compression. This paper provides an overview of our research in this area. Specifically, it describes the results and the future directions of our work on compressed-domain texture feature extraction, image matching, image manipulation, and video indexing.
Hash-SVM: Scalable Kernel Machines for Large-Scale Visual Classification

This paper presents a novel algorithm which uses compact hash bits to greatly improve the efficiency of non-linear kernel SVM in very large scale visual classification problems. Our key idea is to represent each sample with compact hash bits, over which an inner product is defined to serve as the surrogate of the original nonlinear kernels. Then the problem of solving the nonlinear SVM can be transformed into solving a linear SVM over the hash bits. The proposed Hash-SVM enjoys dramatic storage cost reduction owing to the compact binary representation, as well as a (sub-)linear training complexity via linear SVM. As a critical component of Hash-SVM, we propose a novel hashing scheme for arbitrary non-linear kernels via random subspace projection in reproducing kernel Hilbert space. Our comprehensive analysis reveals a well behaved theoretic bound of the deviation between the proposed hashing-based kernel approximation and the original kernel function. We also derive requirements on the hash bits for achieving a satisfactory accuracy level. Several experiments on large-scale visual classification benchmarks are conducted, including one with over 1 million images. The results show that Hash-SVM greatly reduces the computational complexity (more than ten times faster in many cases) while keeping comparable accuracies.
Video Event Detection by Inferring Temporal Instance Labels

Video event detection allows intelligent indexing of video content based on events. Traditional approaches extract features from video frames or shots, then quantize and pool the features to form a single vector representation for the entire video. Though simple and efficient, the final pooling step may lead to loss of temporally local information, which is important in indicating which part in a long video signifies presence of the event. In this work, we propose a novel instance-based video event detection approach. We represent each video as multiple 'instances', defined as video segments of different temporal intervals. The objective is to learn an instance-level event detection model based on only video-level labels. To solve this problem, we propose a large-margin formulation which treats the instance labels as hidden latent variables, and simultaneously infers the instance labels as well as the instance-level classification model. Our framework infers optimal solutions that assume positive videos have a large number of positive instances while negative videos have the fewest ones. Extensive experiments on large-scale video event datasets demonstrate significant performance gains. The proposed method is also useful in explaining the detection results by localizing the temporal segments in a video which is responsible for the positive detection.
Next-generation content representation, creation, and searching for new-media applications in education

Content creation, editing, and searching are extremely time-consuming tasks that often require substantial training and experience, especially when high-quality audio and video are involved. New media represents a new paradigm for multimedia information representation and processing, in which the emphasis is placed on the actual content. It thus brings the tasks of content creation and searching much closer to actual users and enables them to be active producers of audio-visual information rather than passive recipients. We discuss the state of the art and present next-generation techniques for content representation, searching, creation and editing. We discuss our experiences in developing a Web-based distributed compressed video editing and searching system (WebClip), a media-representation language (Flavor) and an object-based video authoring system (Zest) based on it, and a large image/video search engine for the World Wide Web (WebSEEk). We also present a case study of new media applications based on specific planned multimedia education experiments with the above systems in several K-12 schools in Manhattan, NY.