Reviewerly/exHarmony huggingface dataset:
Deep Learning Based Full-Reference and No-Reference Quality Assessment Models for Compressed UGC Videos

In this paper, we propose a deep learning based video quality assessment (VQA) framework to evaluate the quality of the compressed user’s generated content (UGC) videos. The proposed VQA framework consists of three modules, the feature extraction module, the quality regression module, and the quality pooling module. For the feature extraction module, we fuse the features from intermediate layers of the convolutional neural network (CNN) network into final quality-aware feature representation, which enables the model to make full use of visual information from low-level to high-level. Specifically, the structure and texture similarities of feature maps extracted from all intermediate layers are calculated as the feature representation for the full reference (FR) VQA model, and the global mean and standard deviation of the final feature maps fused by intermediate feature maps are calculated as the feature representation for the no reference (NR) VQA model. For the quality regression module, we use the fully connected (FC) layer to regress the quality-aware features into frame-level scores. Finally, a subjectively-inspired temporal pooling strategy is adopted to pool frame-level scores into the video-level score. The proposed model achieves the best performance among the state-of-the-art FR and NR VQA models on the Compressed UGC VQA database and also achieves pretty good performance on the in-the-wild UGC VQA databases.
WOAD: Weakly Supervised Online Action Detection in Untrimmed Videos

Online action detection in untrimmed videos aims to identify an action as it happens, which makes it very important for real-time applications. Previous methods rely on tedious annotations of temporal action boundaries for training, which hinders the scalability of online action detection systems. We propose WOAD, a weakly supervised framework that can be trained using only video-class labels. WOAD contains two jointly-trained modules, i.e., temporal proposal generator (TPG) and online action recognizer (OAR). Supervised by video-class labels, TPG works offline and targets at accurately mining pseudo frame-level labels for OAR. With the supervisory signals from TPG, OAR learns to conduct action detection in an online fashion. Experimental results on THUMOS’14, ActivityNet1.2 and ActivityNet1.3 show that our weakly-supervised method largely outperforms weakly-supervised baselines and achieves comparable performance to the previous strongly-supervised methods. Beyond that, WOAD is flexible to leverage strong supervision when it is available. When strongly supervised, our method obtains the state-of-the-art results in the tasks of both online per-frame action recognition and online detection of action start.
Low-speed cameras system for 3D-DIC vibration measurements in the kHz range

The digital image correlation (DIC) was used in this paper to obtain full-field measurements of a target vibrating at a frequency higher than the maximum cameras’ frame rate. The down-sampling technique was implemented to compensate for the cameras’ moderate frame rate, thus getting an accurate displacement acquisition even at 6.5 kHz. Two innovative methods to support the DIC application were introduced. The use of fringe projection (or structured light), initially applied on the sample at rest, reduced the effort and time required for the stereo matching task’s solution and improved this setting’s accuracy and reliability. Additionally, a new time-domain image filtering was proposed and tested to improve the quality of the obtained DIC maps. In combination with the down-sampling, the effect of this filtering technique was tested in this work at (approx.) 2500 and 6500 Hz by measuring the response of a bladed disk to sinusoidal excitation. Evidence of improved results was observed for both frequencies for amplitudes in the range of 10 µm.
Detecting Deep-Fake Videos from Aural and Oral Dynamics

A face-swap deep fake replaces a person’s face – from eyebrows to chin – with another face. A lip-sync deep fake replaces a person’s mouth region to be consistent with an impersonated or synthesized audio track. An overlooked aspect in the creation of these deep-fake videos is the human ear. Statically, the shape of the human ear has been shown to provide a biometric signal. Dynamically, movement of the mandible (lower jaw) causes changes in the shape of the ear and ear canal. While the facial identity in a face-swap deep fake may accurately depict the co-opted identity, the ears belong to the original identity. While the mouth in a lip-sync deep fake may be well synchronized with the audio, the dynamics of the ear motion will be de-coupled from the mouth and jaw motion. We describe a forensic technique that exploits these static and dynamic aural properties.
The role of reality enhancing technologies in teaching and learning of mathematics

Virtual and Augmented Reality (VR and AR, respectively) are technologies with potential impact in education, and can facilitate subjects, such as mathematics, that may be difficult for students. This manuscript presents a comprehensive review of works using VR and/or AR for teaching mathematics at all academic levels during the timespan from 2015 to June 2020. The work contributes by providing an overview of the findings, challenges, and trends for the use of these technologies. Results from the review indicate that both VR and AR have seen acceptance by students but using the technologies might be complicated if lectures are not designed adequately. Moreover, the number of publications involving AR and VR in teaching and learning maths has increased over time, and the demographics indicate that the technology is taking prevalence in developing countries.
BIIoVT: Blockchain-Based Secure Storage Architecture for Intelligent Internet of Vehicular Things

Today, the rapid growth of vehicles connected to the Internet enables provides various services to consumers, including traffic management, traffic safety, and entertainment. Vehicular ad-hoc network is one of the most prominent and emerging technologies on the Internet of Vehicular Things (IoVT). This technology offers to fulfill requirements such as robust information exchange, and infotainment among vehicles for the smart city environment. Still, it has some challenges such as centralization, storage, security, and privacy because all city vehicular networks send vehicles and road-related information data directly to the cloud. This article proposes BIIoVT: Blockchain-based secure storage architecture for Intelligent Internet of Vehicular Things to mitigate the above-mention issues. Blockchain provides security and privacy at each city's vehicular networks and decentralized storage at the cloud layer with a distributed hash table. It also examines how the vehicular network offers a secure platform. The validation results of the proposed architecture show an outstanding balance of secure storage and efficiency for the IoVT compared to existing methods.
A Novel Asymmetric Hyperchaotic Image Encryption Scheme Based on Elliptic Curve Cryptography

Most of the image encryption schemes based on chaos have so far employed symmetric key cryptography, which leads to a situation where the key cannot be transmitted in public channels, thus limiting their extended application. Based on the elliptic curve cryptography (ECC), we proposed a public key image encryption method where the hash value derived from the plain image was encrypted by ECC. Furthermore, during image permutation, a novel algorithm based on different-sized block was proposed. The plain image was firstly divided into five planes according to the amount of information contained in different bits: the combination of the low 4 bits, and other four planes of high 4 bits respectively. Second, for different planes, the corresponding method of block partition was followed by the rule that the higher the bit plane, the smaller the size of the partitioned block as a basic unit for permutation. In the diffusion phase, the used hyperchaotic sequences in permutation were applied to improve the efficiency. Lots of experimental simulations and cryptanalyses were implemented in which the NPCR and UACI are 99.6124% and 33.4600% respectively, which all suggested that it can effectively resist statistical analysis attacks and chosen plaintext attacks.
Unsupervised Learning of 3D Object Categories from Videos in the Wild

Our goal is to learn a deep network that, given a small number of images of an object of a given category, reconstructs it in 3D. While several recent works have obtained analogous results using synthetic data or assuming the avail-ability of 2D primitives such as keypoints, we are interested in working with challenging real data and with no manual annotations. We thus focus on learning a model from multiple views of a large collection of object instances. We contribute with a new large dataset of object centric videos suitable for training and benchmarking this class of models. We show that existing techniques leveraging meshes, voxels, or implicit surfaces, which work well for reconstructing isolated objects, fail on this challenging data. Finally, we propose a new neural network design, called warp-conditioned ray embedding (WCR), which significantly improves reconstruction while obtaining a detailed implicit representation of the object surface and texture, also compensating for the noise in the initial SfM reconstruction that bootstrapped the learning process. Our evaluation demonstrates performance improvements over several deep monocular reconstruction baselines on existing benchmarks and on our novel dataset. For additional material please visit: https://henzler.github.io/publication/unsupervised_videos/.
A State-of-the-Art Review for Gastric Histopathology Image Analysis Approaches and Future Development

Gastric cancer is a common and deadly cancer in the world. The gold standard for the detection of gastric cancer is the histological examination by pathologists, where Gastric Histopathological Image Analysis (GHIA) contributes significant diagnostic information. The histopathological images of gastric cancer contain sufficient characterization information, which plays a crucial role in the diagnosis and treatment of gastric cancer. In order to improve the accuracy and objectivity of GHIA, Computer-Aided Diagnosis (CAD) has been widely used in histological image analysis of gastric cancer. In this review, the CAD technique on pathological images of gastric cancer is summarized. Firstly, the paper summarizes the image preprocessing methods, then introduces the methods of feature extraction, and then generalizes the existing segmentation and classification techniques. Finally, these techniques are systematically introduced and analyzed for the convenience of future researchers.
Predicting the objective and priority of issue reports in software repositories

Software repositories such as GitHub host a large number of software entities. Developers collaboratively discuss, implement, use, and share these entities. Proper documentation plays an important role in successful software management and maintenance. Users exploit Issue Tracking Systems, a facility of software repositories, to keep track of issue reports, to manage the workload and processes, and finally, to document the highlight of their team’s effort. An issue report is a rich source of collaboratively-curated software knowledge, and can contain a reported problem, a request for new features, or merely a question about the software product. As the number of these issues increases, it becomes harder to manage them manually. GitHub provides labels for tagging issues, as a means of issue management. However, about half of the issues in GitHub’s top 1000 repositories do not have any labels. In this work, we aim at automating the process of managing issue reports for software teams. We propose a two-stage approach to predict both the objective behind opening an issue and its priority level using feature engineering methods and state-of-the-art text classifiers. To the best of our knowledge, we are the first to fine-tune a Transformer for issue classification. We train and evaluate our models in both project-based and cross-project settings. The latter approach provides a generic prediction model applicable for any unseen software project or projects with little historical data. Our proposed approach can successfully predict the objective and priority level of issue reports with $$82\%$$ (fine-tuned RoBERTa) and $$75\%$$ (Random Forest) accuracy, respectively. Moreover, we conducted human labeling and evaluation on unlabeled issues from six unseen GitHub projects to assess the performance of the cross-project model on new data. The model achieves $$90\%$$ accuracy on the sample set. We measure inter-rater reliability and obtain an average Percent Agreement of $$85.3\%$$ and Randolph’s free-marginal Kappa of 0.71 that translate to a substantial agreement among labelers.
Online Attention Accumulation for Weakly Supervised Semantic Segmentation

Object attention maps generated by image classifiers are usually used as priors for weakly supervised semantic segmentation. However, attention maps usually locate the most discriminative object parts. The lack of integral object localization maps heavily limits the performance of weakly supervised segmentation approaches. This paper attempts to investigate a novel way to identify entire object regions in a weakly supervised manner. We observe that image classifiers' attention maps at different training phases may focus on different parts of the target objects. Based on this observation, we propose an online attention accumulation (OAA) strategy that utilizes the attention maps at different training phases to obtain more integral object regions. Specifically, we maintain a cumulative attention map for each target category in each training image and utilize it to record the discovered object regions at different training phases. Albeit OAA can effectively mine more object regions for most images, for some training images, the range of the attention movement is not large, limiting the generation of integral object attention regions. To overcome this problem, we propose incorporating an attention drop layer into the online attention accumulation process to enlarge the range of attention movement during training explicitly. Our method (OAA) can be plugged into any classification network and progressively accumulate the discriminative regions into cumulative attention maps as the training process goes. Additionally, we also explore utilizing the final cumulative attention maps to serve as the pixel-level supervision, which can further assist the network in discovering more integral object regions. When applying the resulting attention maps to the weakly supervised semantic segmentation task, our approach improves the existing state-of-the-art methods on the PASCAL VOC 2012 segmentation benchmark, achieving a mIoU score of 67.2 percent on the test set.
Holistic 3D Human and Scene Mesh Estimation from Single View Images

The 3D world limits the human body pose and the human body pose conveys information about the surrounding objects. Indeed, from a single image of a person placed in an indoor scene, we as humans are adept at resolving ambiguities of the human pose and room layout through our knowledge of the physical laws and prior perception of the plausible object and human poses. However, few computer vision models fully leverage this fact. In this work, we pro-pose a holistically trainable model that perceives the 3D scene from a single RGB image, estimates the camera pose and the room layout, and reconstructs both human body and object meshes. By imposing a set of comprehensive and sophisticated losses on all aspects of the estimations, we show that our model outperforms existing human body mesh methods and indoor scene reconstruction methods. To the best of our knowledge, this is the first model that outputs both object and human predictions at the mesh level, and performs joint optimization on the scene and human poses.
QHSL: A quantum hue, saturation, and lightness color model

Image processing with any potential quantum computing hardware requires a quantum color model capable of capturing and manipulating color information in images. In this study, a quantum hue, saturation, and lightness (QHSL) model is proposed as a first attempt to encode perceptually relevant triplet color components using the properties of quantum mechanics (i.e., entanglement and parallelism). The proposed color model was used to define a representation of two-dimensional QHSL images for storage and transformation with fewer computing resources. The configuration of color-assignment attributes within this QHSL representation offer useful applications for image analysis. Specifically, a pseudocolor technique with flexible gray depth divisions is presented for highlighting fine visual details.
“BNN - BN = ?”: Training Binary Neural Networks without Batch Normalization

Batch normalization (BN) is a key facilitator and considered essential for state-of-the-art binary neural networks (BNN). However, the BN layer is costly to calculate and is typically implemented with non-binary parameters, leaving a hurdle for the efficient implementation of BNN training. It also introduces undesirable dependence between samples within each batch. Inspired by the latest advance on Batch Normalization Free (BN-Free) training [7], we extend their framework to training BNNs, and for the first time demonstrate that BNs can be completely removed from BNN training and inference regimes. By plugging in and customizing techniques including adaptive gradient clipping, scale weight standardization, and specialized bottleneck block, a BN-free BNN is capable of maintaining competitive accuracy compared to its BN-based counterpart. Extensive experiments validate the effectiveness of our proposal across diverse BNN backbones and datasets. For example, after removing BNs from the state-of-the-art ReActNets [38], it can still be trained with our proposed methodology to achieve 92.08%, 68.34%, and 68.0% accuracy on CIFAR-10, CIFAR-100, and ImageNet respectively, with marginal performance drop (0.23% ∼ 0.44% on CIFAR and 1.40% on ImageNet). Codes and pre-trained models are available at: https://github.com/VITA-Group/BNN_NoBN.
Bad Characters: Imperceptible NLP Attacks

Several years of research have shown that machine-learning systems are vulnerable to adversarial examples, both in theory and in practice. Until now, such attacks have primarily targeted visual models, exploiting the gap between human and machine perception. Although text-based models have also been attacked with adversarial examples, such attacks struggled to preserve semantic meaning and indistinguishability. In this paper, we explore a large class of adversarial examples that can be used to attack text-based models in a black-box setting without making any human-perceptible visual modification to inputs. We use encoding-specific perturbations that are imperceptible to the human eye to manipulate the outputs of a wide range of Natural Language Processing (NLP) systems from neural machine-translation pipelines to web search engines. We find that with a single imperceptible encoding injection – representing one invisible character, homoglyph, reordering, or deletion – an attacker can significantly reduce the performance of vulnerable models, and with three injections most models can be functionally broken. Our attacks work against currently-deployed commercial systems, including those produced by Microsoft and Google, in addition to open source models published by Facebook, IBM, and HuggingFace. This novel series of attacks presents a significant threat to many language processing systems: an attacker can affect systems in a targeted manner without any assumptions about the underlying model. We conclude that text-based NLP systems require careful input sanitization, just like conventional applications, and that given such systems are now being deployed rapidly at scale, the urgent attention of architects and operators is required.
In the light of feature distributions: moment matching for Neural Style Transfer

Style transfer aims to render the content of a given image in the graphical/artistic style of another image. The fundamental concept underlying Neural Style Transfer (NST) is to interpret style as a distribution in the feature space of a Convolutional Neural Network, such that a desired style can be achieved by matching its feature distribution. We show that most current implementations of that concept have important theoretical and practical limitations, as they only partially align the feature distributions. We propose a novel approach that matches the distributions more precisely, thus reproducing the desired style more faithfully, while still being computationally efficient. Specifically, we adapt the dual form of Central Moment Discrepancy (CMD), as recently proposed for domain adaptation, to minimize the difference between the target style and the feature distribution of the output image. The dual interpretation of this metric explicitly matches all higher-order centralized moments and is therefore a natural extension of existing NST methods that only take into account the first and second moments. Our experiments confirm that the strong theoretical properties also translate to visually better style transfer, and better disentangle style from semantic image content.
Free-viewpoint Indoor Neural Relighting from Multi-view Stereo

We introduce a neural relighting algorithm for captured indoors scenes, that allows interactive free-viewpoint navigation. Our method allows illumination to be changed synthetically, while coherently rendering cast shadows and complex glossy materials. We start with multiple images of the scene and a three-dimensional mesh obtained by multi-view stereo (MVS) reconstruction. We assume that lighting is well explained as the sum of a view-independent diffuse component and a view-dependent glossy term concentrated around the mirror reflection direction. We design a convolutional network around input feature maps that facilitate learning of an implicit representation of scene materials and illumination, enabling both relighting and free-viewpoint navigation. We generate these input maps by exploiting the best elements of both image-based and physically based rendering. We sample the input views to estimate diffuse scene irradiance, and compute the new illumination caused by user-specified light sources using path tracing. To facilitate the network's understanding of materials and synthesize plausible glossy reflections, we reproject the views and compute mirror images . We train the network on a synthetic dataset where each scene is also reconstructed with MVS. We show results of our algorithm relighting real indoor scenes and performing free-viewpoint navigation with complex and realistic glossy reflections, which so far remained out of reach for view-synthesis techniques.
The Intersection of Users, Roles, Interactions, and Technologies in Creativity Support Tools

Creativity Support Tools (CSTs) have become an integral part of artistic creation. The range of CST technologies is broad—from fabricators to generative algorithms to robots. The interaction approaches for CSTs are accordingly broad. CSTs combine specific technologies and interaction types to serve a spectrum of roles and users. In this work, we tackle a comprehensive understanding of how the intersections of users, roles, interactions, and technologies form a design space for CSTs. We accomplish this by reviewing 111 art-creation CSTs from HCI and computing research and analyzing how diverse aspects of CSTs relate to each other. Our findings identify patterns for designing CSTs, which can give guidance to future CST designers. We also highlight under-explored types of CSTs within the HCI community, providing future directions that CST researchers can pursue given the current trajectory of technological advancement. This work contributes an integrating perspective to understand the landscape of art-creation CSTs.