Reviewerly/exHarmony huggingface dataset:
Progressive Temporal Feature Alignment Network for Video Inpainting

Video inpainting aims to fill spatiotemporal "corrupted" regions with plausible content. To achieve this goal, it is necessary to find correspondences from neighbouring frames to faithfully hallucinate the unknown con-tent. Current methods achieve this goal through attention, flow-based warping, or 3D temporal convolution. However, flow-based warping can create artifacts when optical flow is not accurate, while temporal convolution may suffer from spatial misalignment. We propose ‘Progressive Temporal Feature Alignment Network’, which progressively enriches features extracted from the current frame with the feature warped from neighbouring frames using optical flow. Our approach corrects the spatial misalignment in the temporal feature propagation stage, greatly improving visual quality and temporal consistency of the inpainted videos. Using the proposed architecture, we achieve state-of-the-art performance on the DAVIS and FVI datasets compared to existing deep learning approaches. Code is available at https://github.com/MaureenZOU/TSAM.
R-CTSVM+: Robust capped L1-norm twin support vector machine with privileged information

In the new paradigm, learning using privileged information (LUPI) creates a more informative strategy for tasks to achieve better prediction. SVM based methods including SVM+ and TSVM+, have achieved considerable success in LUPI on the clean data. However, these methods typically suffer from the noise and outliers contained in the data, which leads to larger fluctuations in performance. To handle this problem, in this paper, we propose a novel Robust Capped L1-norm Twin Support Vector Machine with Privileged Information (R-CTSVM+). The proposed pair of regularization functions (up- and down-bound) can definitely help to increase the learning model's tolerance to disturbance, because the up-bound function aims to maximize the lower bound of the perturbation of both main feature and privilege feature, meanwhile, the down-bound function aims to minimize the upper bound of the perturbation of both main feature and privilege feature. Moreover, as the widely employed squared L2-norm distance typically exaggerates the impact of outliers, we adopt the capped L1 regularized distance to further guarantee the robustness of the model. We present that the resulted optimization problem is theoretically converged and can be solved using an effective alternating optimization procedure. Experimental results on benchmark datasets indicate that the proposed robust model can produce superior performance in the case where data samples contain much noise and outliers.
Data Poisoning Attack against Recommender System Using Incomplete and Perturbed Data

Recent studies reveal that recommender systems are vulnerable to data poisoning attack due to their openness nature. In data poisoning attack, the attacker typically recruits a group of controlled users to inject well-crafted user-item interaction data into the recommendation model's training set to modify the model parameters as desired. Thus, existing attack approaches usually require full access to the training data to infer items' characteristics and craft the fake interactions for controlled users. However, such attack approaches may not be feasible in practice due to the attacker's limited data collection capability and the restricted access to the training data, which sometimes are even perturbed by the privacy preserving mechanism of the service providers. Such design-reality gap may cause failure of attacks. In this paper, we fill the gap by proposing two novel adversarial attack approaches to handle the incompleteness and perturbations in user-item interaction data. First, we propose a bi-level optimization framework that incorporates a probabilistic generative model to find the users and items whose interaction data is sufficient and has not been significantly perturbed, and leverage these users and items' data to craft fake user-item interactions. Moreover, we reverse the learning process of recommendation models and develop a simple yet effective approach that can incorporate context-specific heuristic rules to handle data incompleteness and perturbations. Extensive experiments on two datasets against three representative recommendation models show that the proposed approaches can achieve better attack performance than existing approaches.
Secure reversible data hiding in encrypted images based on adaptive prediction-error labeling

Reversible data hiding ensures that the legitimate recipient can completely recover the original image and additional data. However, in order to hide the original image content, the content-owner might encrypt the image locally first. Thus reversible data hiding in encrypted images (RDHEI) has attracted wide attention, which can be used for cloud storage and privacy protection. In this paper, a high-capacity and secure RDHEI method based on adaptive prediction-error labeling (APL) is proposed, which takes advantage of the Laplacian-like distribution of prediction errors. To increase the reserved room before encryption, we present two approaches that combine APL and encryption. One pessimistically exposes the shuffled labels in encrypted images, while the other optimistically performs the encryption on the labels. Dependent of both approaches, the original image and additional data can be completely and separately recovered if the recipient has the secret key. More importantly, security of RDHEI is emphasized, not only the invisibility of image content, but also the undetectability of additional data. Experimental results show that our method is able to achieve higher embedding capacity and relevant security compared with prior arts.
The Study of AR-Based Learning for Natural Science Inquiry Activities in Taiwan’s Elementary School from the Perspective of Sustainable Development

Experiential activities are the most direct form of ecological teaching, so outdoor education is an important way of learning. Through direct observation and life experience, students will learn about the ecological environment, understand the importance of the ecological environment, raise their awareness of environmental protection, and put into practice the protection of the ecological environment. This study involved plant teaching activities that incorporated school plant learning paths into environmental education, so that students could learn about flora, experience nature, and take care of the environment. We created an augmented reality application for use in schools that takes the user on an ecological tour of the plants in the diet of butterflies. The application formed the localized and special mobile learning content in a school. Students were allowed to walk out of the classroom and use their mobile device to engage in autonomous learning, and we then determined their acceptance of the augmented reality application. Through this application we were able to tell whether students were able to adapt to learning with technology combined with traditional teaching methods. This application was tested on a group of elementary school students, who were then interviewed using quantitative and qualitative research methods to understand the users’ feedback.
Contrastive Neural Architecture Search with Neural Architecture Comparators

One of the key steps in Neural Architecture Search (NAS) is to estimate the performance of candidate architectures. Existing methods either directly use the validation performance or learn a predictor to estimate the performance. However, these methods can be either computationally expensive or very inaccurate, which may severely affect the search efficiency and performance. Moreover, as it is very difficult to annotate architectures with accurate performance on specific tasks, learning a promising performance predictor is often non-trivial due to the lack of labeled data. In this paper, we argue that it may not be necessary to estimate the absolute performance for NAS. On the contrary, we may need only to understand whether an architecture is better than a baseline one. However, how to exploit this comparison information as the reward and how to well use the limited labeled data remains two great challenges. In this paper, we propose a novel Contrastive Neural Architecture Search (CTNAS) method which performs architecture search by taking the comparison results between architectures as the reward. Specifically, we design and learn a Neural Architecture Comparator (NAC) to compute the probability of candidate architectures being better than a baseline one. Moreover, we present a baseline updating scheme to improve the baseline iteratively in a curriculum learning manner. More critically, we theoretically show that learning NAC is equivalent to optimizing the ranking over architectures. Extensive experiments in three search spaces demonstrate the superiority of our CTNAS over existing methods.
DeepACG: Co-Saliency Detection via Semantic-aware Contrast Gromov-Wasserstein Distance

The objective of co-saliency detection is to segment the co-occurring salient objects in a group of images. To address this task, we introduce a new deep network architecture via semantic-aware contrast Gromov-Wasserstein distance (DeepACG). We first adopt the Gromov-Wasserstein (GW) distance to build dense 4D correlation volumes for all pairs of image pixels within the image group. These dense correlation volumes enable the network to accurately discover the structured pair-wise pixel similarities among the common salient objects. Second, we develop a semantic-aware co-attention module (SCAM) to enhance the foreground co-saliency through predicted categorical information. Specifically, SCAM recognizes the semantic class of the foreground co-objects, and this information is then modulated to the deep representations to localize the related pixels. Third, we design a contrast edge-enhanced module (EEM) to capture richer contexts and preserve fine-grained spatial information. We validate the effectiveness of our model using three largest and most challenging benchmark datasets (Cosal2015, CoCA, and CoSOD3k). Extensive experiments have demonstrated the substantial practical merit of each module. Compared with the existing works, DeepACG shows significant improvements and achieves state-of-the-art performance.
A Multimodal Data Processing System for LiDAR-Based Human Activity Recognition

Increasingly, the task of detecting and recognizing the actions of a human has been delegated to some form of neural network processing camera or wearable sensor data. Due to the degree to which the camera can be affected by lighting and wearable sensors scantiness, neither one modality can capture the required data to perform the task confidently. That being the case, range sensors, like light detection and ranging (LiDAR), can complement the process to perceive the environment more robustly. Most recently, researchers have been exploring ways to apply convolutional neural networks to 3-D data. These methods typically rely on a single modality and cannot draw on information from complementing sensor streams to improve accuracy. This article proposes a framework to tackle human activity recognition by leveraging the benefits of sensor fusion and multimodal machine learning. Given both RGB and point cloud data, our method describes the activities being performed by subjects using regions with a convolutional neural network (R-CNN) and a 3-D modified Fisher vector network. Evaluated on a custom captured multimodal dataset demonstrates that the model outputs remarkably accurate human activity classification (90%). Furthermore, this framework can be used for sports analytics, understanding social behavior, surveillance, and perhaps most notably by autonomous vehicles (AVs) to data-driven decision-making policies in urban areas and indoor environments.
AutoClips: An Automatic Approach to Video Generation from Data Facts

Abstract Data videos, a storytelling genre that visualizes data facts with motion graphics, are gaining increasing popularity among data journalists, non‐profits, and marketers to communicate data to broad audiences. However, crafting a data video is often time‐consuming and asks for various domain knowledge such as data visualization, animation design, and screenwriting. Existing authoring tools usually enable users to edit and compose a set of templates manually, which still cost a lot of human effort. To further lower the barrier of creating data videos, this work introduces a new approach, AutoClips, which can automatically generate data videos given the input of a sequence of data facts. We built AutoClips through two stages. First, we constructed a fact‐driven clip library where we mapped ten data facts to potential animated visualizations respectively by analyzing 230 online data videos and conducting interviews. Next, we constructed an algorithm that generates data videos from data facts through three steps: selecting and identifying the optimal clip for each of the data facts, arranging the clips into a coherent video, and optimizing the duration of the video. The results from two user studies indicated that the data videos generated by AutoClips are comprehensible, engaging, and have comparable quality with human‐made videos.
Soft Skills in Software Engineering Technicians Education

Abstract The research work analyzes the problem of forming software engineering technicians’ soft skills, approaches to defining the list, detail and the maturity level of software engineering technicians’ soft skills and basic ways of their formation, in particular: development of new academic programs or individual innovative courses, the problem-oriented approach, the AGILE approach, gamification of learning, development of interdisciplinary courses, etc. The obtained data enable developing methodological recommendations to form software engineering technicians’ soft skills when studying the humanities and social sciences and experimentally investigating their efficiency.
Evaluation of deep learning‐based artificial intelligence techniques for breast cancer detection on mammograms: Results from a retrospective study using a BreastScreen Victoria dataset

This study aims to evaluate deep learning (DL)-based artificial intelligence (AI) techniques for detecting the presence of breast cancer on a digital mammogram image.We evaluated several DL-based AI techniques that employ different approaches and backbone DL models and tested the effect on performance of using different data-processing strategies on a set of digital mammographic images with annotations of pathologically proven breast cancer.Our evaluation uses the area under curve (AUC) and accuracy (ACC) for performance measurement. The best evaluation result, based on 349 test cases (930 test images), was an AUC of 0.8979 [95% confidence interval (CI) 0.873, 0.923] and ACC of 0.8178 [95% CI 0.785, 0.850]. This was achieved by an AI technique that utilises a certain family of DL models, namely ResNet, as its backbone, combines the global features extracted from the whole mammogram and the local features extracted from the automatically detected cancer and non-cancer local regions in the whole image, and leverages background cropping and text removal, contrast adjustment and more training data.DL-based AI techniques have shown promising results in retrospective studies for many medical image analysis applications. Our study demonstrates a significant opportunity to boost the performance of such techniques applied to breast cancer detection by exploring different types of approaches, backbone DL models and data-processing strategies. The promising results we have obtained suggest further development of AI reading services could transform breast cancer screening in the future.
A 3D GAN for Improved Large-pose Facial Recognition

Facial recognition using deep convolutional neural networks relies on the availability of large datasets of face images. Many examples of identities are needed, and for each identity, a large variety of images are needed in order for the network to learn robustness to intra-class variation. In practice, such datasets are difficult to obtain, particularly those containing adequate variation of pose. Generative Adversarial Networks (GANs) provide a potential solution to this problem due to their ability to generate realistic, synthetic images. However, recent studies have shown that current methods of disentangling pose from identity are inadequate. In this work we incorporate a 3D morphable model into the generator of a GAN in order to learn a nonlinear texture model from in-the-wild images. This allows generation of new, synthetic identities, and manipulation of pose, illumination and expression without compromising the identity. Our synthesised data is used to augment training of facial recognition networks with performance evaluated on the challenging CFP and CPLFW datasets.
HDR Environment Map Estimation for Real-Time Augmented Reality

We present a method to estimate an HDR environment map from a narrow field-of-view LDR camera image in real-time. This enables perceptually appealing reflections and shading on virtual objects of any material finish, from mirror to diffuse, rendered into a real environment using augmented reality. Our method is based on our efficient convolutional neural network, EnvMapNet, trained end-to-end with two novel losses, ProjectionLoss for the generated image, and ClusterLoss for adversarial training. Through qualitative and quantitative comparison to state-of-the-art methods, we demonstrate that our algorithm reduces the directional error of estimated light sources by more than 50%, and achieves 3.7 times lower Frechet Inception Distance (FID). We further showcase a mobile application that is able to run our neural network model in under 9 ms on an iPhone XS, and render in real-time, visually coherent virtual objects in previously unseen real-world environments.
AIFit: Automatic 3D Human-Interpretable Feedback Models for Fitness Training

I went to the gym today, but how well did I do? And where should I improve? Ah, my back hurts slightly... User engagement can be sustained and injuries avoided by being able to reconstruct 3d human pose and motion, relate it to good training practices, identify errors, and provide early, real-time feedback. In this paper we introduce the first automatic system, AIFit, that performs 3d human sensing for fitness training. The system can be used at home, outdoors, or at the gym. AIFit is able to reconstruct 3d human pose, shape, and motion, reliably segment exercise repetitions, and identify in real-time the deviations between standards learnt from trainers, and the execution of a trainee. As a result, localized, quantitative feedback for correct execution of exercises, reduced risk of injury, and continuous improvement is possible. To support research and evaluation, we introduce the first large scale dataset, Fit3D, containing over 3 million images and corresponding 3d human shape and motion capture ground truth configurations, with over 37 repeated exercises, covering all the major muscle groups, performed by instructors and trainees. Our statistical coach is governed by a global parameter that captures how critical it should be of a trainee’s performance. This is an important aspect that helps adapt to a student’s level of fitness (i.e. beginner vs. advanced vs. expert), or to the expected accuracy of a 3d pose reconstruction method. We show that, for different values of the global parameter, our feedback system based on 3d pose estimates achieves good accuracy compared to the one based on ground-truth motion capture. Our statistical coach offers feedback in natural language, and with spatio-temporal visual grounding.
Privacy-preserving Data Aggregation against Malicious Data Mining Attack for IoT-enabled Smart Grid

Internet of Things (IoT)-enabled smart grids can achieve more reliable and high-frequency data collection and transmission compared with existing grids. However, this frequent data processing may consume a lot of bandwidth, and even put the user’s privacy at risk. Although many privacy-preserving data aggregation schemes have been proposed to solve the problem, they still suffer from some security weaknesses or performance deficiency, such as lack of satisfactory data confidentiality and resistance to malicious data mining attack. To address these issues, we propose a novel privacy-preserving data aggregation scheme (called PDAM) for IoT-enabled smart grids, which can support efficient data source authentication and integrity checking, secure dynamic user join and exit. Unlike existing schemes, the PDAM is resilient to the malicious data mining attack launched by internal or external attackers and can achieve perfect data confidentiality against not only a malicious aggregator but also a curious control center for an authorized user. The detailed security and performance analysis show that our proposed PDAM can satisfy several well-known security properties and desirable efficiency for a smart grid system. Moreover, the comparative studies and experiments demonstrate that the PDAM is superior to other recently proposed works in terms of both security and performance.
Animating Pictures with Eulerian Motion Fields

In this paper, we demonstrate a fully automatic method for converting a still image into a realistic animated looping video. We target scenes with continuous fluid motion, such as flowing water and billowing smoke. Our method relies on the observation that this type of natural motion can be convincingly reproduced from a static Eulerian motion description, i.e. a single, temporally constant flow field that defines the immediate motion of a particle at a given 2D location. We use an image-to-image translation network to encode motion priors of natural scenes collected from on-line videos, so that for a new photo, we can synthesize a corresponding motion field. The image is then animated using the generated motion through a deep warping technique: pixels are encoded as deep features, those features are warped via Eulerian motion, and the resulting warped feature maps are decoded as images. In order to produce continuous, seamlessly looping video textures, we propose a novel video looping technique that flows features both for-ward and backward in time and then blends the results. We demonstrate the effectiveness and robustness of our method by applying it to a large collection of examples including beaches, waterfalls, and flowing rivers.
Efficient Cross-Modality Graph Reasoning for RGB-Infrared Person Re-Identification

The modality and pose variance between RGB and infrared (IR) images are two key challenges for RGB-IR person re-identification. Existing methods mainly focus on leveraging pixel or feature alignment to handle the intra-class variations and cross-modality discrepancy. However, these methods are hard to keep semantic identity consistency between global and local representation, which the consistency is important for the cross-modality pedestrian re-identification task. In this work, we propose a novel cross-modality graph reasoning method (CGRNet) to globally model and reason over relations between modalities and context, and to keep semantic identity consistency between global and local representation. Specifically, we propose a local modality-similarity module to put the distribution of modality-specific features into a common subspace without losing identity information. Besides, we squeeze the input feature of RGB and IR images into a channel-wise global vector, and through graph reasoning, the identity relationship and modality relationship in each vector are inferred. Extensive experiments on two datasets demonstrate the superior performance of our approach over the existing state-of-the-art. The code is available at <uri>https://github.com/fegnyujian/CGRNet</uri>.