Reviewerly/exHarmony huggingface dataset:
Accelerated masked transformer for dense video captioning

Dense video captioning aims to generate dense descriptions for all possible events in an untrimmed video. The task is challenging that it requires accurately localizing events in the video and simultaneously describe each event with a sentence. Current approaches usually decompose this task into two independent stages—the proposal localization stage and the caption generation stage, resulting in a suboptimal solution. Masked Transformer (MT) model [30] has been proposed to integrate the two stages and optimize them in an end-to-end philosophy. Despite the superior performance that the MT has achieved, its runtime efficiency is unsatisfactory which severely limits its applicability in real-world scenarios. In this paper, we devise an improved Accelerated Masked Transformer (AMT) model that enjoys the dual-benefit of effectiveness and efficiency. Taking MT as our reference model, we respectively introduce accelerating strategies to the two stages: 1) in the proposal localization stage, we introduce a lightweight anchor-free proposal in company with a local attention mechanism; and 2) in the caption generation stage, we introduce the single-shot feature masking strategy along with an average attention mechanism. Extensive experiments on two benchmark datasets ActivityNet-Caption and YouCookII demonstrate that AMT achieves competitive performance on both datasets with significant speed improvement. On the ActivityNet-Caption dataset, AMT reduces up to 2× running time with comparable performance when compared to the reference MT model.
Temporal Graph Neural Networks for Social Recommendation

In social recommendation, the purchase decision of users is influenced by their basic preference of items, as well as the social influence of peers. Such social connections had been proved to be effective in modeling users' preference of items. However, most models in social recommender literature only considered two types of relations, i.e., user-item relation in interaction network and user-user relation in social network. The temporal sequential information of items, i.e., item-item relation, can also be utilized to infer the preference of users, but had been ignored in almost all of the graph based recommendation models. Two issues of such temporal information had not been well studied in social recommender systems: the temporal strength information, i.e., the real purchase time of an item, and its influence on social relations. To address the above issues, we propose a novel Temporal Enhanced Graph Model for Social Recommendation (TGRec). In TGRec, the purchase time information between items is characterized as a special temporal relation, and the purchase decision of users depends on three factors: (1) a user's basic preference of items, (2) the collaborative influence of peers, (3) the temporal impact of previous items bought by the user. Experimental results on three real-world commerce datasets demonstrate the effectiveness of our model for social recommendation, showing the usefulness of modeling the temporal information in heterogeneous graph.
Constrained Domain Adaptation for Image Segmentation

We propose to adapt segmentation networks with a constrained formulation, which embeds domain-invariant prior knowledge about the segmentation regions. Such knowledge may take the form of simple anatomical information, e.g., structure size or shape, estimated from source samples or known a priori. Our method imposes domain-invariant inequality constraints on the network outputs of unlabeled target samples. It implicitly matches prediction statistics between target and source domains with permitted uncertainty of prior knowledge. We address our constrained problem with a differentiable penalty, fully suited for standard stochastic gradient descent approaches, removing the need for computationally expensive Lagrangian optimization with dual projections. Unlike current two-step adversarial training, our formulation is based on a single loss in a single network, which simplifies adaptation by avoiding extra adversarial steps, while improving convergence and quality of training. The comparison of our approach with state-of-the-art adversarial methods reveals substantially better performance on the challenging task of adapting spine segmentation across different MRI modalities. Our results also show a robustness to imprecision of size priors, approaching the accuracy of a fully supervised model trained directly in a target domain.Our method can be readily used for various constraints and segmentation problems.
Comparative Study of Artificial Neural Network Based Channel Equalization Methods for mmWave Communications

In this paper, we compare two artificial neural networks (ANNs) approaches designed to perform channel equalization for millimeter-wave (mmWave) signals operating in the 28 GHz frequency band. We used an in-house deterministic Three-Dimensional Ray-Launching (3D-RL) code to simulate the spatial structure of mmWave channels considering the material properties of the obstacles within the scenario at the frequency under analysis. We performed offline training of a multilayer perceptron (MLP) neural network with the simulated mmWave channels to equalize the received signal. We also performed online training of an extreme learning machine (ELM) neural network to directly get the equalized symbols at the receiver, given as input the received mmWave signal. The ANN solutions were tested in terms of the achievable spectral efficiency, bit error rate, and time to process. We compared the ANN techniques to the minimum mean square error and the zero-forcing equalizers, considering an orthogonal frequency-division multiplexing communication based on the 5G New Radio standard. We present numerical results on the performance of the proposed ANNs and show that the ELM strategy outperforms the MLP method, requiring significantly less processing time than the reviewed equalization methods.
Temporally smooth online action detection using cycle-consistent future anticipation

Many video understanding tasks work in the offline setting by assuming that the input video is given from the start to the end. However, many real-world problems require the online setting, making a decision immediately using only the current and the past frames of videos such as in autonomous driving and surveillance systems. In this paper, we present a novel solution for online action detection by using a simple yet effective RNN-based networks called the Future Anticipation and Temporally Smoothing network (FATSnet). The proposed network consists of a module for anticipating the future that can be trained in an unsupervised manner with the cycle-consistency loss, and another component for aggregating the past and the future for temporally smooth frame-by-frame predictions. We also propose a solution to relieve the performance loss when running RNN-based models on very long sequences. Evaluations on TVSeries, THUMOS14, and BBDB show that our method achieve the state-of-the-art performances compared to the previous works on online action detection.
Privacy preserving biometric authentication using Chaos on remote untrusted server

Biometric-based authentication systems are promising because of unique biometric features for each person. Generally, biometric features are generated , and biometric templates or bio-hash codes are calculated and stored in Authentication Data Table (ADT) along with userid. However, such hashing based authentication is vulnerable to precomputed table attacks. Also, in some cases, the service providers outsource the ADT to cloud or untrusted server. In this paper, a novel biometric-based authentication system is proposed using two servers: crypto-match server and untrusted storage server. The system utilizes biometric image cryptosystem, cryptographic hashing, and Paillier cryptosystem. In the proposed cryptosystem, keystreams are generated from Henon and logistic maps. The control parameters of these chaotic maps are calculated from the input biometric image. The proposed biometric encryption is able to resist statistical and differential attacks and its security level is also validated through various analyzes. Also, the attacks related to the privacy of ADT are analyzed.
Evaluate and improve the quality of neural style transfer

Recent studies have made tremendous progress in neural style transfer (NST) and various methods have been advanced. However, evaluating and improving the stylization quality remain two important open challenges. Committed to these two aspects, in this paper, we first decompose the quality of style transfer into three quantifiable factors, i.e., the content fidelity (CF), global effects (GE) and local patterns (LP). Then, two novel approaches are further presented for exploiting these factors to improve the stylization quality. The first, named cascade style transfer (CST), utilizes the factors to guide the cascade combination of existing NST methods to absorb their merits and avoid their own shortcomings. The second, dubbed multi-objective network (MO-Net), directly optimizes these factors to balance their performance and achieves more harmonious stylized results. Extensive experiments demonstrate the effectiveness and superiority of our proposed factors and methods.
Emotion Analysis in Text using TF-IDF

A myriad of the population has adapted to the evolving technology, which includes text communication. Users advertently or inadvertently share emotions. As we know, emotions are one of the most critical aspects of human life; they impact human's behavior, thinking, compelling of action, and most important, decision making. There are many alleged emotions known to us, and each having its significance. In this era of modern technology, it is hard to find any unexplored area; this applies to emotion. People express their emotions through text a lot nowadays, which has led the Emotion Recognition as an important research area. Extracting emotion is a very complicated task. This paper shows a new approach to detect emotion based on TFIDF, and it is a measure that reflects the value a word holds in a document. In this method, emotion is classified into six types. There are other researches on the simple distinction between positive and negative emotion, but this does not add much to understanding human emotion. Emotion is extracted from different sentences, and data representation is based on semantic structure. It generalizes each sentence into six major predefined emotion sets. The evaluation shows that this method is well accomplished to categorize a sentence into different emotion categories and with a reasonable accuracy rate.
Attention-shift based deep neural network for fine-grained visual categorization

Fine-grained visual categorization (FGVC) has attracted extensive attention in recent years. The general pipeline of current FGVC techniques is to 1) locate the discriminative regions; 2) extract features from each region independently; and 3) feed the integrated features to a classifier. In this paper, we re-investigate the pipeline from the view of human visual recognition mechanisms. The perceiving of discriminative regions is a temporal processing by the human visual system (HVS) via the attention-shift mechanism. However, the existing independent feature extracting and one-pass feeding strategy ignore the inherent semantic relationships among discriminative regions, and thus is improper to model the attention-shift process properly. Therefore, in this paper, we propose a novel end-to-end FGVC network structure named Attention-Shift based Deep Neural Network (AS-DNN) to locate the discriminative regions automatically and encode the semantic correlations iteratively. AS-DNN consists of two channels: 1) the global perception channel Cglb and 2) the attention-shift channel Csft, simulating the global perception and the attention-shift mechanism, respectively. Experimental results show that AS-DNN achieves state-of-the-art performances by outperforming both the CNN-based weakly or strongly-supervised FGVC algorithms on several widely-used fine-grained datasets, and the visualization of attention regions exhibit that the proposed method can locate the discriminative regions robustly in complex backgrounds and postures.
Multilevel Anomaly Detection Through Variational Autoencoders and Bayesian Models for Self-Aware Embodied Agents

Anomaly detection constitutes a fundamental step in developing self-aware autonomous agents capable of continuously learning from new situations, as it enables to distinguish novel experiences from already encountered ones. This paper combines Dynamic Bayesian Networks (DBNs) and Neural Networks (NNs) and proposes a method for detecting anomalies in video data at different abstraction levels. We use a Variational Autoencoder (VAE) to reduce the dimensionality of video frames, and Optical Flows between subsequent images, generating a latent space that captures both visual and dynamical information and that is comparable to low-dimensional sensory data (e.g., positioning, steering angle). An Adapted Markov Jump Particle Filter is employed to predict the following frames and detect anomalies in video data. Our method’s evaluation is executed using different video data from a semi-autonomous vehicle performing different tasks in a closed environment. Tests on benchmark anomaly detection datasets have additionally been conducted.
Towards an attestation architecture for blockchain networks

Abstract If blockchain networks are to become the building blocks of the infrastructure for the future digital economy, then several challenges related to the resiliency and survivability of blockchain networks need to be addressed. The survivability of a blockchain network is influenced by the diversity of its nodes. Trustworthy device-level attestations permits nodes in a blockchain network to provide truthful evidence regarding their current configuration, operational state, keying material and other system attributes. In the current work we review the recent developments towards a standard attestation architecture and evidence conveyance protocols. We explore the applicability and benefits of a standard attestation architecture to blockchain networks. Finally, we discuss a number of open challenges related to node attestations that has arisen due to changing model of blockchain network deployments, such as the use of virtualization and containerization technologies for nodes in cloud infrastructures.
3D Face From X: Learning Face Shape From Diverse Sources

We present a novel method to jointly learn a 3D face parametric model and 3D face reconstruction from diverse sources. Previous methods usually learn 3D face modeling from one kind of source, such as scanned data or in-the-wild images. Although 3D scanned data contain accurate geometric information of face shapes, the capture system is expensive and such datasets usually contain a small number of subjects. On the other hand, in-the-wild face images are easily obtained and there are a large number of facial images. However, facial images do not contain explicit geometric information. In this paper, we propose a method to learn a unified face model from diverse sources. Besides scanned face data and face images, we also utilize a large number of RGB-D images captured with an iPhone X to bridge the gap between the two sources. Experimental results demonstrate that with training data from more sources, we can learn a more powerful face model.
Sent2Vec: A New Sentence Embedding Representation With Sentimental Semantic

Text classification is considered as one of the primary task in many Natural Language Processing (NLP) applications. In industrial applications of NLP, sentimental analysis is a task to understand how satisfied a user is after receiving a service or buying a product. The traditional approach is to convert a text into a format of numeric vector before feeding into machine learning algorithm. This representation of a word refers to word embedding. However the traditional embedding methods often model the syntactic context of words but ignore the sentiment information of text [1]. This can impact on the accuracy of a classification model to predict the correct sentimental score for a text. In this paper, we present Sent2Vec, an alternative embedding representation that includes the sentimental semantic of a sentence in its embedding vector. We utilized the unsupervised Smoothed Inverse Frequency (uSIF) sentence embedding method in the Sent2Vec neural network over a multi million samples dataset. The new sentence embedding presented, can be used as features in downstream (un)supervised tasks, which also leads to better or comparable results compared to sophisticated methods. Furthermore, with a simple logistic regression classifier, Sent2Vec reaches competitive performance to state-of-the-art results on several datasets when combined with GloVe(6B).
GestOnHMD: Enabling Gesture-based Interaction on Low-cost VR Head-Mounted Display

Low-cost virtual-reality (VR) head-mounted displays (HMDs) with the integration of smartphones have brought the immersive VR to the masses, and increased the ubiquity of VR. However, these systems are often limited by their poor interactivity. In this paper, we present GestOnHMD, a gesture-based interaction technique and a gesture-classification pipeline that leverages the stereo microphones in a commodity smartphone to detect the tapping and the scratching gestures on the front, the left, and the right surfaces on a mobile VR headset. Taking the Google Cardboard as our focused headset, we first conducted a gesture-elicitation study to generate 150 user-defined gestures with 50 on each surface. We then selected 15, 9, and 9 gestures for the front, the left, and the right surfaces respectively based on user preferences and signal detectability. We constructed a data set containing the acoustic signals of 18 users performing these on-surface gestures, and trained the deep-learning classification pipeline for gesture detection and recognition. Lastly, with the real-time demonstration of GestOnHMD, we conducted a series of online participatory-design sessions to collect a set of user-defined gesture-referent mappings that could potentially benefit from GestOnHMD.
Earthquake Early Warning System Using Ncheck and Hard-Shared Orthogonal Multitarget Regression on Deep Learning

Realizing an effective earthquake early warning system (EEWS) in the case of extensive regions and noisy signals is challenging, particularly in East Java, Indonesia. This letter proposes the rapid detection of the p-wave arrival and determination of the earthquake’s hypocenter and magnitude using deep learning. The Ncheck algorithm is used for noise handling for picking the p-arrival on a multistation waveform as a form of picking target window prediction (PTWP). Then, multitarget regression (MTR) with a hard-shared orthogonal optimization model is proposed for earthquake parameter determination. The data sets used contained data of earthquakes recorded at three stations from the Indonesian seismic network in East Java; 2009–2017 data were used for training and validation, and 2019 data were used for real-time testing. The results show that the PTWP for picking p-arrival has a mean absolute error (MAE) of 0.12 s, and the MTR for earthquake magnitude, longitude, latitude, depth, and origin time detection shows MAEs of 0.21 M, 9.44, 18.72, 27.81 km, and 2.78 s, respectively.
Mitigating Data Poisoning Attacks On a Federated Learning-Edge Computing Network

Edge Computing (EC) has seen a continuous rise in its popularity as it provides a solution to the latency and communication issues associated with edge devices transferring data to remote servers. EC achieves this by bringing the cloud closer to edge devices. Even though EC does an excellent job of solving the latency and communication issues, it does not solve the privacy issues associated with users transferring personal data to the nearby edge server. Federated Learning (FL) is an approach that was introduced to solve the privacy issues associated with data transfers to distant servers. FL attempts to resolve this issue by bringing the code to the data, which goes against the traditional way of sending the data to remote servers. In FL, the data stays on the source device, and a Machine Learning (ML) model used to train the local data is brought to the end device instead. End devices train the ML model using local data and then send the model updates back to the server for aggregation. However, this process of asking random devices to train a model using its local data has potential risks such as a participant poisoning the model using malicious data for training to produce bogus parameters. In this paper, an approach to mitigate data poisoning attacks in a federated learning setting is investigated. The application of the approach is highlighted, and the practical and secure nature of this approach is illustrated as well using numerical results.
A feature invariant generative adversarial network for head and neck MRI/CT image synthesis

With the emergence of online MRI radiotherapy treatments, MR-based workflows have increased in importance in the clinical workflow. However proper dose planning still requires CT images to calculate dose attenuation due to bony structures. In this paper, we present a novel deep image synthesis model that generates in an unsupervised manner CT images from diagnostic MRI for radiotherapy planning. The proposed model based on a generative adversarial network (GAN) consists of learning a new invariant representation to generate synthetic CT (sCT) images based on high frequency and appearance patterns. This new representation encodes each convolutional feature map of the convolutional GAN discriminator, leading the training of the proposed model to be particularly robust in terms of image synthesis quality. Our model includes an analysis of common histogram features in the training process, thus reinforcing the generator such that the output sCT image exhibits a histogram matching that of the ground-truth CT. This CT-matched histogram is embedded then in a multi-resolution framework by assessing the evaluation over all layers of the discriminator network, which then allows the model to robustly classify the output synthetic image. Experiments were conducted on head and neck images of 56 cancer patients with a wide range of shape sizes and spatial image resolutions. The obtained results confirm the efficiency of the proposed model compared to other generative models, where the mean absolute error yielded by our model was 26.44(0.62), with a Hounsfield unit error of 45.3(1.87), and an overall Dice coefficient of 0.74(0.05), demonstrating the potential of the synthesis model for radiotherapy planning applications.