Reviewerly/exHarmony huggingface dataset:
Knowledge Enhanced Neural Fashion Trend Forecasting

Fashion trend forecasting is a crucial task for both academia andindustry. Although some efforts have been devoted to tackling this challenging task, they only studied limited fashion elements with highly seasonal or simple patterns, which could hardly reveal thereal fashion trends. Towards insightful fashion trend forecasting,this work focuses on investigating fine-grained fashion element trends for specific user groups. We first contribute a large-scale fashion trend dataset (FIT) collected from Instagram with extracted time series fashion element records and user information. Furthermore, to effectively model the time series data of fashion elements with rather complex patterns, we propose a Knowledge Enhanced Recurrent Network model (KERN) which takes advantage of the capability of deep recurrent neural networks in modeling time series data. Moreover, it leverages internal and external knowledgein fashion domain that affects the time-series patterns of fashion element trends. Such incorporation of domain knowledge further enhances the deep learning model in capturing the patterns of specific fashion elements and predicting the future trends. Extensive experiments demonstrate that the proposed KERN model can effectively capture the complicated patterns of objective fashion elements, therefore making preferable fashion trend forecast.
High Capacity Reversible Data Hiding in Encrypted Image Based on Intra-Block Lossless Compression

The cover image is generally encrypted by a stream cipher in existing reversible data hiding in encrypted image (RDHEI) methods. As pixel correlation is seriously damaged, more than one pixel should be employed to carry one bit such that the quite limited capacity is achieved. To overcome this issue, a new RDHEI method with high capacity, that preserves pixel correlation and exploits it to vacate embedding room, is proposed in this paper. First, we propose a block-level encryption scheme which combines block-level stream cipher and block-level permutation, and all blocks are classified into usable blocks (UBs) and unusable blocks (NUBs) by preserving the correlation of pixels in blocks. Then, UB is reconstructed to vacate room for data embedding, because the pixels in blocks share the same most significant bits (MSBs). To ensure reversibility, the number of NUBs between current UB and the previous one is also embedded along with additional data, and the blocks are rearranged in a reversible way such that UBs are always in front of NUBs. Experimental results show that not only the embedding capacity is significantly improved but also the hidden data can be losslessly extracted, and the cover image can be perfectly recovered.
Vision-Dialog Navigation by Exploring Cross-Modal Memory

Vision-dialog navigation posed as a new holy-grail task in vision-language disciplinary targets at learning an agent endowed with the capability of constant conversation for help with natural language and navigating according to human responses. Besides the common challenges faced in visual language navigation, vision-dialog navigation also requires to handle well with the language intentions of a series of questions about the temporal context from dialogue history and co-reasoning both dialogs and visual scenes. In this paper, we propose the Cross-modal Memory Network (CMN) for remembering and understanding the rich information relevant to historical navigation actions. Our CMN consists of two memory modules, the language memory module (L-mem) and the visual memory module (V-mem). Specifically, L-mem learns latent relationships between the current language interaction and a dialog history by employing a multi-head attention mechanism. V-mem learns to associate the current visual views and the cross-modal memory about the previous navigation actions. The cross-modal memory is generated via a vision-to-language attention and a language-to-vision attention. Benefiting from the collaborative learning of the L-mem and the V-mem, our CMN is able to explore the memory about the decision making of historical navigation actions which is for the current step. Experiments on the CVDN dataset show that our CMN outperforms the previous state-of-the-art model by a significant margin on both seen and unseen environments.
Varicolored Image De-Hazing

The quality of images captured in bad weather is often affected by chromatic casts and low visibility due to the presence of atmospheric particles. Restoration of the color balance is often ignored in most of the existing image de-hazing methods. In this paper, we propose a varicolored end-to-end image de-hazing network which restores the color balance in a given varicolored hazy image and recovers the haze-free image. The proposed network comprises of 1) Haze color correction (HCC) module and 2) Visibility improvement (VI) module. The proposed HCC module provides required attention to each color channel and generates a color balanced hazy image. While the proposed VI module processes the color balanced hazy image through novel inception attention block to recover the haze-free image. We also propose a novel approach to generate a large-scale varicolored synthetic hazy image database. An ablation study has been carried out to demonstrate the effect of different factors on the performance of the proposed network for image de-hazing. Three benchmark synthetic datasets have been used for quantitative analysis of the proposed network. Visual results on a set of real-world hazy images captured in different weather conditions demonstrate the effectiveness of the proposed approach for varicolored image de-hazing.
Multi-View Representation Learning With Deep Gaussian Processes

Multi-view representation learning is a promising and challenging research topic, which aims to integrate multiple data information from different views to improve the learning performance. The recent deep Gaussian processes (DGPs) have the advantages of good uncertainty estimates, powerful non-linear mapping ability and great generalization capability, which can be used as an excellent data representation learning method. However, DGPs only focus on single view data and are rarely applied to the multi-view scenario. In this paper, we propose a multi-view representation learning algorithm with deep Gaussian processes (named MvDGPs), which inherits the advantages of deep Gaussian processes and multi-view representation learning, and can learn more effective representation of multi-view data. The MvDGPs consist of two stages. The first stage is multi-view data representation learning, which is mainly used to learn more comprehensive representations of multi-view data. The second stage is classifier design, which aims to select an appropriate classifier to better employ the representations obtained in the first stage. In contrast with DGPs, MvDGPs support asymmetrical modeling depths for different views of data, resulting in better characterizations of the discrepancies among different views. Experimental results on real-world multi-view data sets verify the effectiveness of the proposed algorithm, which indicates that MvDGPs can integrate the complementary information in multiple views to discover a good representation of the data.
3D Sketch-Aware Semantic Scene Completion via Semi-Supervised Structure Prior

The goal of the Semantic Scene Completion (SSC) task is to simultaneously predict a completed 3D voxel representation of volumetric occupancy and semantic labels of objects in the scene from a single-view observation. Since the computational cost generally increases explosively along with the growth of voxel resolution, most current state-of-the-arts have to tailor their framework into a low-resolution representation with the sacrifice of detail prediction. Thus, voxel resolution becomes one of the crucial difficulties that lead to the performance bottleneck. In this paper, we propose to devise a new geometry-based strategy to embed depth information with low-resolution voxel representation, which could still be able to encode sufficient geometric information, e.g., room layout, object's sizes and shapes, to infer the invisible areas of the scene with well structure-preserving details. To this end, we first propose a novel 3D sketch-aware feature embedding to explicitly encode geometric information effectively and efficiently. With the 3D sketch in hand, we further devise a simple yet effective semantic scene completion framework that incorporates a light-weight 3D Sketch Hallucination module to guide the inference of occupancy and the semantic labels via a semi-supervised structure prior learning strategy. We demonstrate that our proposed geometric embedding works better than the depth feature learning from habitual SSC frameworks. Our final model surpasses state- of-the-arts consistently on three public benchmarks, which only requires 3D volumes of 60 × 36 × 60 resolution for both input and output.
Multi-Fusion Residual Memory Network for Multimodal Human Sentiment Comprehension

Multimodal human sentiment comprehension refers to recognizing human affection from multiple modalities. There exist two key issues for this problem. First, it is difficult to explore time-dependent interactions between modalities and focus on the important time steps. Second, processing the long fused sequence of utterances is susceptible to the forgetting problem due to the long-term temporal dependency. In this article, we introduce a hierarchical learning architecture to classify utterance-level sentiment. To address the first issue, we perform time-step level fusion to generate fused features for each time step, which explicitly models time-restricted interactions by incorporating information across modalities at the same time step. Furthermore, based on the assumption that acoustic features directly reflect emotional intensity, we pioneer emotion intensity attention to focus on the time steps where emotion changes or intense affections take place. To handle the second issue, we propose Residual Memory Network (RMN) to process the fused sequence. RMN utilizes some techniques such as directly passing the previous state into the next time step, which helps to retain the information from many time steps ago. We show that our method achieves state-of-the-art performance on multiple datasets. Results also suggest that RMN yields competitive performance on sequence modeling tasks.
ActionBytes: Learning From Trimmed Videos to Localize Actions

This paper tackles the problem of localizing actions in long untrimmed videos.Different from existing works, which all use annotated untrimmed videos during training, we learn only from short trimmed videos.This enables learning from large-scale datasets originally designed for action classification.We propose a method to train an action localization network that segments a video into interpretable fragments, we call ActionBytes.Our method jointly learns to cluster ActionBytes and trains the localization network using the cluster assignments as pseudolabels.By doing so, we train on short trimmed videos that become untrimmed for ActionBytes.In isolation, or when merged, the ActionBytes also serve as effective action proposals.Experiments demonstrate that our boundary-guided training generalizes to unknown action classes and localizes actions in long videos of Thumos14, MultiThumos, and ActivityNet1.2.Furthermore, we show the advantage of Ac-tionBytes for zero-shot localization as well as traditional weakly supervised localization, that train on long videos, to achieve state-of-the-art results.
Single Image Reflection Removal With Physically-Based Training Images

Recently, deep learning-based single image reflection separation methods have been exploited widely. To benefit the learning approach, a large number of training image pairs (i.e., with and without reflections) were synthesized in various ways, yet they are away from a physically-based direction. In this paper, physically based rendering is used for faithfully synthesizing the required training images, and a corresponding network structure and loss term are proposed. We utilize existing RGBD/RGB images to estimate meshes, then physically simulate the light transportation between meshes, glass, and lens with path tracing to synthesize training data, which successfully reproduce the spatially variant anisotropic visual effect of glass reflection. For guiding the separation better, we additionally consider a module, backtrack network (BT-net) for backtracking the reflections, which removes complicated ghosting, attenuation, blurred and defocused effect of glass/lens. This enables obtaining a priori information before having the distortion. The proposed method considering additional a priori information with physically simulated training data is validated with various real reflection images and shows visually pleasant and numerical advantages compared with state-of-the-art techniques.
End-to-End Camera Calibration for Broadcast Videos

The increasing number of vision-based tracking systems deployed in production have necessitated fast, robust camera calibration. In the domain of sport, the majority of current work focuses on sports where lines and intersections are easy to extract, and appearance is relatively consistent across venues. However, for more challenging sports like basketball, those techniques are not sufficient. In this paper, we propose an end-to-end approach for single moving camera calibration across challenging scenarios in sports. Our method contains three key modules: 1) area-based court segmentation, 2) camera pose estimation with embedded templates, 3) homography prediction via a spatial transform network (STN). All three modules are connected, enabling end-to-end training. We evaluate our method on a new college basketball dataset and demonstrate state of the art performance in variable and dynamic environments. We also validate our method on the World Cup 2014 dataset to show its competitive performance against the state-of-the-art methods. Lastly, we show that our method is two orders of magnitude faster than the previous state of the art on both datasets.
Deep Pyramidal Pooling With Attention for Person Re-Identification

Learning discriminative, view-invariant and multi-scale representations of object appearance with different semantic levels is of paramount importance for person Re-Identification (ReID). Recently, the community has focused on learning deep Re-ID models to capture a single holistic representation. To improve the achieved results, additional visual attributes and object part-driven models have been considered, inevitably introducing additional human annotation labor or computational efforts. In this paper, we argue that pyramid-inspired methods capturing multi-scale information may overcome such requirements. Precisely, multi-scale pooled regions representing visual information of an object are integrated within a novel deep architecture factorizing them into discriminative features at multiple semantic levels. These are exploited through an attention mechanism later considered in an identification-similarity multi-task loss, trained by means of a curriculum learning strategy. Extensive results on three person ReID benchmarks demonstrate that better performance than existing methods are achieved. Code is available at https://github.com/iN1k1.
Augmented Reality-Based Learning Environment to Enhance Teaching-Learning Experience in Geometry Education

Virtual information and physical objects are deployed in regular teaching all over, and until recently, blending these two environments has been a very difficult task at best. Understanding the concepts of geometry and their three-dimensional (3-D) space is still considered a difficult subject area for some students. Therefore, a requirement of a learning innovation arises for learning geometry to overcome the problems faced while understanding geometry by students’. The main objective of the paper is to develop Augmented Reality (AR)- based geometry learning for the android and iOS platforms than to deploy the applications among students for teaching 3-D geometry in high school students. This technology is considered to the realm of science and mathematics classroom and supports theoretical underpinnings in understanding the benefits as well as limitations of augmented reality-based learning environment (ARLE) experiences. One of the topics which are difficult for the students to understand is geometry in mathematics education. To address the problem, the article introduces a framework of mobile- based ARLE system.
Using Siamese Networks with Transfer Learning for Face Recognition on Small-Samples Datasets

Nowadays, computer-based face recognition is a mature and reliable mechanism that is significantly used in many access control scenarios along with other biometric methods. Face recognition consists of two subtasks including Face Verification and Face Identification. By comparing a pair of images, Face Verification determines whether those images are related to one person or not; and Face Identification has to identify a specific face within a set of available faces in the database. There are many challenges in face recognition such as angle, illumination, pose, facial expression, noise, resolution, occlusion and the few number of one-class samples with several classes. In this paper, we are carrying out face recognition by utilizing transfer learning in a siamese network which consists of two similar CNNs. In the siamese network, a pair of two face images is given to the network as input, then the network extracts the features of this pair of images and finally, it determines whether the pair of images belongs to one person or not by using a similarity criterion. The results show that the proposed model is comparable with advanced models that are trained on datasets containing large numbers of samples. furthermore, it improves the accuracy of face recognition in comparison with methods which are trained using datasets with a few number of samples, and the mentioned accuracy is claimed to be 95.62% on LFW dataset.
A content recommendation system for effective e-learning using embedded feature selection and fuzzy DT based CNN

This paper proposes a new content recommendation system which combines the newly proposed embedded feature selection method and the new Fuzzy Temporal Logic based Decision Tree incorporated Convolutional Neural Network classifier. The newly proposed embedded feature selection called Fuzzy Decision Tree and Weighted Gini-Index based Feature Selection Algorithm (FDTWGI-FSA) that contains the existing incorporated the Fuzzy Decision Tree (FDT) and the Weighted Gini-index based Feature Selection Algorithm (WGIFSA) for getting optimized feature subset. Moreover, an enhanced CNN and Fuzzy Temporal Decision Tree for performing the deep learning process which is able to identify the exact e-content from the huge volume of data with the help of the recommended features by the proposed embedded feature selection method. The exact e-content can be identified after performing the five-layer network structure for extracting the relevant features and it also can be classified by applying the Fuzzy Temporal Decision Tree for the e-learners. Finally, the proposed content recommendation system provides exact content to the e-learners according to their level of understanding and it also satisfies them by providing the exact high level contents. The experiments have been conducted for evaluating the proposed content recommendation system and compared with the existing classifier including the standard CNN.
Air-writing recognition using smart-bands

We propose a novel approach for textual input which is based on air-writing recognition using smart-bands. The proposed approach enables the user to hand-write in the air in an intuitive and natural way, where text is recognized by analyzing the motion signals captured by an off-the-shelf smart-band worn by the user. Unlike existing studies that proposed the use of motion signals to recognize written letters, our approach does not require an extra dedicated device, nor it imposes unnecessary limitations on the writing process of the user. To test the feasibility of the new approach, we developed two air-writing recognition methods: a user-dependent method, based on K-Nearest-Neighbors with Dynamic-Time-Warping as the distance measure, and a user-independent method, based on a Convolutional-Neural-Network. The first creates a tailored model for each user, using a set of reference samples collected from the user in an enrollment phase, and therefore has the potential to be more accurate. The latter involves a preliminary training phase which generates a single model to fit all users, and therefore does not require an enrollment phase for new users. In order to evaluate our methods, we collected 15 sets of the English alpha-bet letters (written on the air and collected using a smart-band) from 55 different subjects. The results of our evaluation demonstrate the ability of the proposed methods to successfully recognize air-written letters with a high degree of accuracy, obtaining 89.2% average accuracy for the user-dependent method, and 83.2% average accuracy (95.6% when applying an auto-correction phase) for the user-independent method.
Mix2FLD: Downlink Federated Learning After Uplink Federated Distillation With Two-Way Mixup

This letter proposes a novel communication-efficient and privacy-preserving distributed machine learning framework, coined Mix2FLD. To address uplink-downlink capacity asymmetry, local model outputs are uploaded to a server in the uplink as in federated distillation (FD), whereas global model parameters are downloaded in the downlink as in federated learning (FL). This requires a model output-to-parameter conversion at the server, after collecting additional data samples from devices. To preserve privacy while not compromising accuracy, linearly mixed-up local samples are uploaded, and inversely mixed up across different devices at the server. Numerical evaluations show that Mix2FLD achieves up to 16.7% higher test accuracy while reducing convergence time by up to 18.8% under asymmetric uplink-downlink channels compared to FL.
A Power Associative Loop Structure for the Construction of Non-Linear Components of Block Cipher

In the symmetric key cryptography, the purpose of the substitution box is to generate confusion and hence improve the security of the whole cryptosystem.For this reason, many algebraic substitution boxes are constructed depending upon the associative algebras of Galois fields.In this paper, the power associative loop is used for the construction of substitution boxes.This novel structure comprises excellent features which include, the inverse of zero element, non-associativity and fewer constraints as compared to the cyclic group and Galois field.In comparison to existing substitution boxes, the substitution boxes based upon power associative loop are comparatively easy and the above-mentioned properties offer the number of structures to construct highly nonlinear substitution boxes.To obtain the number of substitution boxes, we further applied the symmetric group of order 16 on the proposed substitution box.The evaluation of proposed boxes with different algebraic and statistical analyses like nonlinearity test, strict avalanche criterion, bit independence criterion, linear approximation probability and differential approximation probability indicate the strength of proposed substitution boxes.Majority logic criterion results depict that proposed substitution boxes have better cryptographic strength to apply in different techniques of secure communication.