Reviewerly/exHarmony huggingface dataset:
Mixed Reality Tunneling Effects for Stereoscopic Untethered Video-See-Through Head-Mounted Displays

We present mixed reality (MR) tunneling, a novel method to balance the trade-off between limited render performance and high visual quality of video see-through head-mounted displays (VST-HMDs) through fusing images of two types of camera sensors with different resolutions and frame rates. By merging a color video stream from an external stereoscopic camera with the grayscale VST commonly integrated into today’s standalone virtual reality (VR) headsets, we create a perceptually high-resolution and wide field of view VSTHMD prototype. The external high-resolution VST displayed at the central foveal to the para-peripheral region of the human visual field complements the low-resolution, low-latency grayscale VST at the far peripheral region, producing a tunneling effect, which simulates the human foveal and peripheral vision, with the potential to reduce cybersickness as in the tunneling effect in immersive VR. We propose two extensions to the MR tunneling method. The first one accommodates the user’s head movement speed by fading out the external VST when fast head movements are detected, thus potentially compensating for video streaming latency. The second one is a foveated MR tunneling effect, which displays the center of the external VST based on the tracked user eye movements. We evaluated the three MR tunneling methods in a within-subject study with 24 participants. The user study demonstrates the potential of our prototype and techniques based on the example of an assembly task that requires hand-eye coordination, untethered locomotion, and fine motor skills. The results demonstrate that, although not significant, the MR tunneling effects lead to higher overall usability, less perceived motion sickness, and a better sense of presence. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Augmenting Weather Sensor Data with Remote Knowledge Graphs

The latest analytical models are becoming frequently used in meteorological science research. For instance, machine learning and deep learning models are being trained for weather forecasting. A solid machine learning model can give trustworthy findings that aid individuals in making weather-related decisions. However, the performance of analytical models is largely determined not only by the design of the model body but also by the input features. We address common issues of modern meteorological studies that take sensor data as the input for various analytical models. In contrast to the traditional practice of combining and preprocessing many fixed sensor data bulks to create an augmented dataset, we tunnel into remote knowledge graphs to fetch and augment the sensor data in a scalable way. As a consequence, we reduce the amount of time and storage space to preprocess diverse data in preparation for analytical models leveraging the high interoperability between knowledge graphs.
One-bit Active Query with Contrastive Pairs

How to achieve better results with fewer labeling costs remains a challenging task. In this paper, we present a new active learning framework, which for the first time incorporates contrastive learning into recently proposed one-bit supervision. Here one-bit supervision denotes a simple Yes or No query about the correctness of the model's prediction, and is more efficient than previous active learning methods requiring assigning accurate labels to the queried samples. We claim that such one-bit information is intrinsically in accordance with the goal of contrastive loss that pulls positive pairs together and pushes negative samples away. Towards this goal, we design an uncertainty metric to actively select samples for query. These samples are then fed into different branches according to the queried results. The Yes query is treated as positive pairs of the queried category for contrastive pulling, while the No query is treated as hard negative pairs for contrastive repelling. Additionally, we design a negative loss that penalizes the negative samples away from the incorrect predicted class, which can be treated as optimizing hard negatives for the corresponding category. Our method, termed as ObCP, produces a more powerful active learning framework, and experiments on several benchmarks demonstrate its superiority.
Design and Implementation of Decision-Tree (DT) Online Training Hardware Using Divider-Free GI Calculation and Speeding-Up Double-Root Classifier

This paper proposes a total solution in ASIC chip hardware implementation, fulfilling decision-tree (DT) online training and classification missions. We also develop three effective design techniques, including divider-free resister-in-parallel Gini Impurity calculation (DR-GIC), adaptive means with updated learning function (AM-ULF), and speeding-up classifier with double-root tree (SC-DRT). In the ASIC chip implementation with TSMC 40-nm multi-Vt CMOS process, the chip layout is well-verified and only has a total core area of 0.803 mm2. The maximum operating frequency is 429 MHz, dissipating average power of 73.7 mW. Regarding online training, it supports a maximum training size of 10.1KB data on chip. Accordingly, the worst-case training latency is only 9.98ms for dealing with 1024 data elements. For classification, the decision throughput is up to 4.29 GBps – 8.58 GBps. After being verified with various datasets for different applications, the ASIC chip with superior design performance offers a solid hardware implementation, realizing a total DT-based procedure from online training to classification.
StyleBabel: Artistic Style Tagging and Captioning

We present StyleBabel, a unique open access dataset of natural language captions and free-form tags describing the artistic style of over 135K digital artworks, collected via a novel participatory method from experts studying at specialist art and design schools. StyleBabel was collected via an iterative method, inspired by 'Grounded Theory': a qualitative approach that enables annotation while co-evolving a shared language for fine-grained artistic style attribute description. We demonstrate several downstream tasks for StyleBabel, adapting the recent ALADIN architecture for fine-grained style similarity, to train cross-modal embeddings for: 1) free-form tag generation; 2) natural language description of artistic style; 3) fine-grained text search of style. To do so, we extend ALADIN with recent advances in Visual Transformer (ViT) and cross-modal representation learning, achieving a state of the art accuracy in fine-grained style retrieval.
Geometric Features Informed Multi-person Human-Object Interaction Recognition in Videos

Human-Object Interaction (HOI) recognition in videos is important for analyzing human activity. Most existing work focusing on visual features usually suffer from occlusion in the real-world scenarios. Such a problem will be further complicated when multiple people and objects are involved in HOIs. Consider that geometric features such as human pose and object position provide meaningful information to understand HOIs, we argue to combine the benefits of both visual and geometric features in HOI recognition, and propose a novel Two-level Geometric feature-informed Graph Convolutional Network ( $$\text {2G-GCN}$$ ). The geometric-level graph models the interdependency between geometric features of humans and objects, while the fusion-level graph further fuses them with visual features of humans and objects. To demonstrate the novelty and effectiveness of our method in challenging scenarios, we propose a new multi-person HOI dataset ( $$\text {MPHOI-72}$$ ). Extensive experiments on $$\text {MPHOI-72}$$ (multi-person HOI), CAD-120 (single-human HOI) and Bimanual Actions (two-hand HOI) datasets demonstrate our superior performance compared to state-of-the-arts.
Domestic private banking solutions can be quite successful as an effective protection against whaling-style cyber attacks which are used as a basis for more complex targeted phishing

The growing anti-Russian cyber attacks from outside has increased dramatically by spring months primarily with phishing and DdoS. Later we observe unusual increase in interest in whaling, the methods of which are used as a basis for more complex targeted phishing. This rare variety of phishing is not too well known within local IS. However effective protection metods should be carried out taking into account the imbalance in the corporate strategy of Russian business evolution: between the current level of corporate relations development, and the existing methods of IS. Thus domestic private banking solutions can be quite successful with effective protection against whaling-style cyber threats.
TCAD: Unsupervised Anomaly Detection Based on Global Local Representation Differences

Multivariate time series anomaly detection is of great interest because of its wide range of applications. Since it is difficult to obtain accurate anomaly labels, many unsupervised anomaly detection algorithms have been developed. However, it is challenging to build an unsupervised multivariate anomaly detection model because we need to find a criterion with anomaly discriminative power. Previously, researchers have focused on extracting the association of time points with global sequences, while ignoring the association of time points with local sequences. In this paper, we propose a combined model TCAD based on Transformer and Resnet, which learns global and local features of sequences using Transformer and Resnet, and constrains the learning of rich global local representations using reconstructed differences and global local representation differences. In addition, this paper proposes an anomaly score based on global and local feature discrepancies. TCAD is extensively tested on four public datasets and two private datasets.
An industrial experience report on retro-inspection

To reinforce the quality of code delivery, especially to improve future coding quality, one global Information and Communication Technology (ICT) enterprise has institutionalized a retrospective style inspection (namely retro-inspection), which is similar to Fagan inspection but differs in terms of stage, participants, etc. This paper reports an industrial case study that aims to investigate the experiences and lessons from this software practice. To this end, we collected and analyzed various empirical evidence for data tri-angulation. The results reflect that retro-inspection distinguishes itself from peer code review by identifying more complicated and underlying defects, providing more indicative and suggestive comments. Many experienced inspectors indicate defects together with their rationale behind and offer suggestions for correction and prevention. As a result, retro-inspection can benefit not only quality assurance (like Fagan inspection), but also internal audit, inter-division communication, and competence promotion. On the other side, we identify several lessons of retro-inspection at this stage, e.g., developers' acceptance and organizers' predicament, for next-step improvement of this practice. To be specific, some recommendations are discussed for retro-inspection, e.g., more adequate preparation and more careful publicity. This study concludes that most of the expected benefits of retro-inspection can be empirically confirmed in this enterprise and its value on the progress to continuous maturity can be recognized organization-wide. The experiences on executing this altered practice in a large enterprise provide reference value on code quality assurance to other software organizations.
TL;DW? Summarizing Instructional Videos with Task Relevance and Cross-Modal Saliency

YouTube users looking for instructions for a specific task may spend a long time browsing content trying to find the right video that matches their needs. Creating a visual summary (abridged version of a video) provides viewers with a quick overview and massively reduces search time. In this work, we focus on summarizing instructional videos, an under-explored area of video summarization. In comparison to generic videos, instructional videos can be parsed into semantically meaningful segments that correspond to important steps of the demonstrated task. Existing video summarization datasets rely on manual frame-level annotations, making them subjective and limited in size. To overcome this, we first automatically generate pseudo summaries for a corpus of instructional videos by exploiting two key assumptions: (i) relevant steps are likely to appear in multiple videos of the same task (Task Relevance), and (ii) they are more likely to be described by the demonstrator verbally (Cross-Modal Saliency). We propose an instructional video summarization network that combines a context-aware temporal video encoder and a segment scoring transformer. Using pseudo summaries as weak supervision, our network constructs a visual summary for an instructional video given only video and transcribed speech. To evaluate our model, we collect a high-quality test set, WikiHow Summaries, by scraping WikiHow articles that contain video demonstrations and visual depictions of steps allowing us to obtain the ground-truth summaries. We outperform several baselines and a state-of-the-art video summarization model on this new benchmark.
A Fair, Verifiable and Privacy-Protecting Data Outsourcing Transaction Scheme Based on Smart Contracts

The continuous production of large-scale data makes data outsourcing computation a trend.In order to ensure that data outsourcing transaction is trusted and fair, it needs the supervision and judgment of a third party.However, the third-party intermediary increases the transaction cost, and there is also subjective unreliability.Smart contracts allow trusted transactions without a third party, which are automatic, traceable and irreversible.Therefore, in this paper, we propose a framework for data outsourcing computation transaction based on the smart contract, in which the transaction is verifiable, fair and privacy protected.In this framework, by improved the replication-based verifiable computation technology, the data in the dataset is evenly distributed to each server, and at least one data in the sub datasets of the adjacent servers is the same, only the results from the duplicate data are verified, so as to implement the verifiable transaction with the low transaction cost.Meanwhile, a punishment mechanism is adopted to solve the fairness of the transaction in the smart contract.In addition, the oblivious transfer protocol is used to implement the privacy protection of transaction data.We deploy the data outsourcing computation transaction framework in the simulation environment of the Ethereum blockchain, and the experimental results show that our proposed scheme is effective and has low overhead.Specifically, the data returned by the server is verifiable, and the data privacy of the client is protected.When the client and servers are dishonest, they will be punished so that the transaction is fair.Moreover, the cost of using the smart contract in transactions is almost negligible.
An XAI-based adversarial training approach for cyber-threat detection

Adversarial training is commonly used in the artificial intelligence literature to improve the robustness of deep neural models to adversarial samples. In addition, eXplainable Artificial Intelligence (XAI) has been recently investigated to improve the interpretability and explainability of black-box artificial systems such as deep neural models. In this study, we propose a methodology that combines adversarial training and XAI, in order to increase the accuracy of deep neural models trained for cyber-threat detection. In particular, we use the FGSM technique to generate the adversarial samples for the adversarial training stage, and SHAP to produce the local explanations of decisions made during the adversarial training stage. These local explanations are, subsequently, used to produce a new feature set that describes the effect of the original cyber-data characteristics on the classifications of the examples processed during the adversarial training stage. Leveraging this XAI-based information, we apply a transfer learning strategy, namely fine-tuning, to improve the accuracy performance of the deep neural model. Experiments conducted on two benchmark cybersecurity datasets prove the effectiveness of the proposed methodology in the multi-class classification of cyber-data.
Hallucinating Pose-Compatible Scenes

What does human pose tell us about a scene? We propose a task to answer this question: given human pose as input, hallucinate a compatible scene. Subtle cues captured by human pose—action semantics, environment affordances, object interactions—provide surprising insight into which scenes are compatible. We present a large-scale generative adversarial network for pose-conditioned scene generation. We significantly scale the size and complexity of training data, curating a massive meta-dataset containing over 19 million frames of humans in everyday environments. We double the capacity of our model with respect to StyleGAN2 to handle such complex data, and design a pose conditioning mechanism that drives our model to learn the nuanced relationship between pose and scene. We leverage our trained model for various applications: hallucinating pose-compatible scene(s) with or without humans, visualizing incompatible scenes and poses, placing a person from one generated image into another scene, and animating pose. Our model produces diverse samples and outperforms pose-conditioned StyleGAN2 and Pix2Pix/Pix2PixHD baselines in terms of accurate human placement (percent of correct keypoints) and quality (Fréchet inception distance).
Bitcoin Price Prediction Using Machine Learning and Deep Learning Algorithms

A cryptocurrency or ‘coins' is a virtual asset which can be used as an alternative to physical currency via a computer network that is not reliant on the government or any bank, to uphold or maintain it. This has led to the creation of numerous brands of cryptocurrencies. It was only after the boom in their price in 2011 that they began to be regarded as an investment asset. Since these coins are highly volatile in their pricing, there is a need for a good prediction of their closing price on which investment decisions can be made. To address this requirement, this paper studies the relative performances of different machine learning algorithms for a well-known cryptocurrency - the ‘Bitcoin’. The performance measures of different machine learning models were undertaken to get the accuracy of the models for Bitcoin and results were obtained. The results show that the Auto Regressive Integrated Moving Average (ARIMA) is better than the other models and has the least mean absolute error. It is observed that the quality of training data and amount of the dataset used plays an important role for a successful prediction. When comparing the predicted value of Bitcoin through ARIMA with its actual value, the results obtained are found to be comparable for the entire four months of analysis.
pPATE: A Pragmatic Private Aggregation of Teacher Ensembles Framework by Sparse Vector Technique Based Differential Privacy, Paillier Cryptosystem and Human-in-the-loop

With advances of deep learning models, Artificial Intelligence (AI) has been applied into various fields to aid human. Some domains where sensitive data with privacy concerns are pivotal, for example medical care, are no exception. Dealing with that, a private learning framework satisfying differential privacy - a gold standard to protect privacy, namely Private Aggregation of Teacher Ensembles (PATE) has gained popularity. However, this framework needs to train a large number of models in disjoint private training datasets, thereby in plethora of cases, it can not be leveraged. In this paper, we propose pPATE - a pragmatical framework that is based on PATE but it uses a sparse vector technique to achieve differential privacy and demonstrate that with small manual efforts of human (expert) in the development loop, our solution can train privacy-preserving models that have approximate accuracy as ground-truth models. Moreover, we extend PATE framework pragmatically in a distributed setting so that it not only aggregates privately but also secures confidentiality and privacy when multi-parties collaborate.
Simple But Powerful, a Language-Supervised Method for Image Emotion Classification

Image emotion classification is an important computer vision task to extract emotions from images. The methods for image emotion classification (IEC) are primarily based on label or distribution as a supervision signal, which neither has enough accessibility nor diversity, limiting the development of IEC research. Inspired by psychology research and the recent booming of large-scale pretrained language models. We figure out a language-supervised paradigm, which can cleverly combine the features of language and visual emotion to drive the visual model to gain stronger emotional discernment with language prompts. To practice the paradigm, we present a conceptually simple while empirically powerful framework for image emotion classification, SimEmotion. That we propose a prompt-based fine-tuning strategy to learn task-specific representations by composing a template with the emotion-level concept and entity-level information. Evaluations on four widely-used affective datasets, namely, Flickr and Instagram (FI), EmotionROI, Twitter I, and Twitter II, demonstrate that the proposed algorithm outperforms the state-of-the-art methods with a large margin (i.e., <inline-formula><tex-math notation="LaTeX">$8.42\%$</tex-math></inline-formula> absolute accuracy gain on EmotionROI) on image emotion classification tasks. Our codes will be publicly available for research purposes.
Improvement Graph Convolution Collaborative Filtering with Weighted Addition Input

Graph Neural Networks have been extensively applied in the field of machine learning to find features of graphs, and recommendation systems are no exception. The ratings of users on considered items can be represented by graphs which are input for many efficient models to find out the characteristics of the users and the items. From these insights, relevant items are recommended to users. However, user’s decisions on the items have varying degrees of effects on different users, and this information should be learned so as not to be lost in the process of information mining. In this publication, we propose to build an additional graph showing the recommended weight of an item to a target user to improve the accuracy of GNN models. Although the users’ friendships were not recorded, their correlation was still evident through the commonalities in consumption behavior. We build a model WiGCN (Weighted input GCN) to describe and experiment on well-known datasets. Conclusions will be stated after comparing our results with state-of-the-art such as GCMC, NGCF and LightGCN. The source code is also included ( https://github.com/trantin84/WiGCN ).