Reviewerly/exHarmony huggingface dataset:
Local Graph Convolutional Networks for Cross-Modal Hashing

Cross-modal hashing aims to map the data of different modalities into a common binary space to accelerate the retrieval speed. Recently, deep cross-modal hashing methods have shown promising performance by applying deep neural networks to facilitate feature learning. However, the known supervised deep methods mainly rely on the labeled information of datasets, which is insufficient to characterize the latent structures that exist among different modalities. To mitigate this problem, in this paper, we propose to use Graph Convolutional Networks (GCNs) to exploit the local structure information of datasets for cross-modal hash learning. Specifically, a local graph is constructed according to the neighborhood relationships between samples in deep feature spaces and fed into GCNs to generate graph embeddings. Then, a within-modality loss is designed to measure the inner products between deep features and graph embeddings so that hashing networks and GCNs can be jointly optimized. By taking advantage of GCNs to assist model's training, the performance of hashing networks can be improved. Extensive experiments on benchmarks verify the effectiveness of the proposed method.
Rain-component-aware capsule-GAN for single image de-raining

Images taken in the rain usually have poor visual quality, which may cause difficulties for vision-based analysis systems. The research aims to recover clean image content from a single rainy image by removing rain components without introducing any artifacts. Existing rain removal methods often model the rain component as noise, but it obviously has clear patterns instead of random noise. Motivated by this, we raise the idea to build modules to capture rain patterns for de-raining. A Rain-Component-Aware (RCA) network is proposed to capture the characteristics of the rain. We then integrate it into an image-conditioned generative adversarial network (image-cGAN) as a RCA loss to guide the generation of rainless images. This results in the proposed two-branch cGAN, where one branch aims at improving the image visual quality after de-raining, and the other aims at extracting rain patterns so that the rain could be effectively removed. To better capture the spatial relationship of different objects within an image, we incorporate the capsule structure in both generator and discriminator of cGAN, which further improves the quality of generated images. The proposed approach is hence named as RCA-cGAN. Benefited by the RCA loss based two-branch optimization and the capsule structure, RCA-cGAN achieves good de-raining effect. Extensive experimental results on several benchmark datasets show that the RCA network is effective to capture rain patterns and the proposed approach could produce much better de-raining images in terms of both subjective visual quality inspection and objective quantitative assessment.
Towards Multiple Black-boxes Attack via Adversarial Example Generation Network

The current research on adversarial attacks aims at a single model while the research on attacking multiple models simultaneously is still challenging. In this paper, we propose a novel black-box attack method, referred to as MBbA, which can attack multiple black-boxes at the same time. By encoding input image and its target category into an associated space, each decoder seeks the appropriate attack areas from the image through the designed loss functions, and then generates effective adversarial examples. This process realizes end-to-end adversarial example generation without involving substitute models for the black-box scenario. On the other hand, adopting the adversarial examples generated by MBbA for adversarial training, the robustness of the attacked models are greatly improved. More importantly, those adversarial examples can achieve satisfactory attack performance, even if these black-box models are trained with the adversarial examples generated by other black-box attack methods, which show good transferability. Finally, extensive experiments show that compared with other state-of-the-art methods: (1) MBbA takes the least time to obtain the most effective attack effects in multi-black-box attack scenario. Furthermore, MBbA achieves the highest attack success rates in a single black-box attack scenario; (2) the adversarial examples generated by MBbA can effectively improve the robustness of the attacked models and exhibit good transferability.
Exploring ensemble oversampling method for imbalanced keyword extraction learning in policy text based on three-way decisions and SMOTE

The e-government platform not only enables the government department to publish policy texts online, but also makes it easier for users to access the policy, especially for the convenience of understanding the policies by reading the keywords. For a given policy text, keywords take up only a small proportion, which can be seen as an unbalanced data set. Therefore, in this paper, we try to design automatic keyword extraction method of policy text with unbalanced data set. In order to achieve this goal, we firstly propose a new ensemble oversampling method to synthesize new data. In this case, we sample data from the training set by using Bagging method. During each sampling process, we train a logistic regression model to classify the training set. Based on the predicted probabilities, we utilize the classification confidence to divide training set into three regions by using three-way decisions (3WD). Then, we implement different strategies to synthesize new data. Besides, for keyword extraction of policy text, we conduct a series of experiments by using the classical supervised and unsupervised methods. In our experiment results, we can find that both in the public data sets and manual data sets, our sampling method can achieve better performance of F-measure and G-mean indexes, no matter what the supervised machine learning method is. This can also explain the advantage of 3WD. Different regions have different strategies to synthesize new data.
Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery

Automatic surgical gesture recognition is fundamentally important to enable intelligent cognitive assistance in robotic surgery. With recent advancement in robot-assisted minimally invasive surgery, rich information including surgical videos and robotic kinematics can be recorded, which provide complementary knowledge for understanding surgical gestures. However, existing methods either solely adopt uni-modal data or directly concatenate multi-modal representations, which can not sufficiently exploit the informative correlations inherent in visual and kinematics data to boost gesture recognition accuracies. In this regard, we propose a novel online approach of multi-modal relational graph network (i.e., MRG-Net) to dynamically integrate visual and kinematics information through interactive message propagation in the latent feature space. In specific, we first extract embeddings from video and kinematics sequences with temporal convolutional networks and LSTM units. Next, we identify multi-relations in these multi-modal embeddings and leverage them through a hierarchical relational graph learning module. The effectiveness of our method is demonstrated with state-of-the-art results on the public JIGSAWS dataset, outperforming current uni-modal and multi-modal methods on both suturing and knot typing tasks. Furthermore, we validated our method on in-house visual-kinematics datasets collected with da Vinci Research Kit (dVRK) platforms in two centers, with consistent promising performance achieved. Our code and data are released at: https://www.cse.cuhk.edu.hk/~yhlong/mrgnet.html.
Exploring Large and Complex Environments Fast and Efficiently

This paper describes a novel framework for autonomous exploration in large and complex environments. We show that the framework is efficient as a result of its hierarchical structure, where at one level it maintains a sparse representation of the environment and at another level, a dense representation is used within a local planning horizon around the robot. The exploration path is computed at the two levels, coarsely at the global scale and finely around the robot. Such a framework produces detailed paths in the vicinity of the robot, while trades off data resolution far away from the robot for computational efficiency. In experiments, we evaluate our method with a real robot exploring large and complex indoor and outdoor environments. Results show that our method is twice as efficient in covering spaces while using less than one-fifth of processing in comparison to state-of-the-art methods.
A feature consistency driven attention erasing network for fine-grained image retrieval

Large-scale fine-grained image retrieval based hashing learning method has two main problems. First, low dimension feature embedding can fasten the retrieval process but bring accuracy decrease due to much information loss. Second, fine-grained images lead to the same category query hash codes mapping into the different cluster in database hash latent space. To handle these issues, we propose a feature consistency driven attention erasing network (FCAENet) for fine-grained image retrieval. For the first issue, we propose an adaptive augmentation module in FCAENet, which is the selective region erasing module (SREM). SREM makes the network more robust on subtle differences of fine-grained task by adaptively covering some regions of raw images. The feature extractor and hash layer can learn more representative hash codes for fine-grained images by SREM. With regard to the second issue, we fully exploit the pair-wise similarity information and add the enhancing space relation loss (ESRL) in FCAENet to make the vulnerable relation stabler between the query hash code and database hash code. We conduct extensive experiments on five fine-grained benchmark datasets (CUB2011, Aircraft, NABirds, VegFru, Food101) for 12bits, 24bits, 32bits, 48bits hash codes. The results show that FCAENet achieves the state-of-the-art (SOTA) fine-grained image retrieval performance based on the hashing learning method.
Detecting violations of access control and information flow policies in data flow diagrams

The security of software-intensive systems is frequently attacked. High fines or loss in reputation are potential consequences of not maintaining confidentiality, which is an important security objective. Detecting confidentiality issues in early software designs enables cost-efficient fixes. A Data Flow Diagram (DFD) is a modeling notation, which focuses on essential, functional aspects of such early software designs. Existing confidentiality analyses on DFDs support either information flow control or access control, which are the most common confidentiality mechanisms. Combining both mechanisms can be beneficial but existing DFD analyses do not support this. This lack of expressiveness requires designers to switch modeling languages to consider both mechanisms, which can lead to inconsistencies. In this article, we present an extended DFD syntax that supports modeling both, information flow and access control, in the same language. This improves expressiveness compared to related work and avoids inconsistencies. We define the semantics of extended DFDs by clauses in first-order logic. A logic program made of these clauses enables the automated detection of confidentiality violations by querying it. We evaluate the expressiveness of the syntax in a case study. We attempt to model nine information flow cases and six access control cases. We successfully modeled fourteen out of these fifteen cases, which indicates good expressiveness. We evaluate the reusability of models when switching confidentiality mechanisms by comparing the cases that share the same system design, which are three pairs of cases. We successfully show improved reusability compared to the state of the art. We evaluated the accuracy of confidentiality analyses by executing them for the fourteen cases that we could model. We experienced good accuracy.
Boosting Atomic Orbit Search Using Dynamic-Based Learning for Feature Selection

Feature selection (FS) is a well-known preprocess step in soft computing and machine learning algorithms. It plays a critical role in different real-world applications since it aims to determine the relevant features and remove other ones. This process (i.e., FS) reduces the time and space complexity of the learning technique used to handle the collected data. The feature selection methods based on metaheuristic (MH) techniques established their performance over all the conventional FS methods. So, in this paper, we presented a modified version of new MH techniques named Atomic Orbital Search (AOS) as FS technique. This is performed using the advances of dynamic opposite-based learning (DOL) strategy that is used to enhance the ability of AOS to explore the search domain. This is performed by increasing the diversity of the solutions during the searching process and updating the search domain. A set of eighteen datasets has been used to evaluate the efficiency of the developed FS approach, named AOSD, and the results of AOSD are compared with other MH methods. From the results, AOSD can reduce the number of features by preserving or increasing the classification accuracy better than other MH techniques.
Information security policy non-compliance: Can capitulation theory explain user behaviors?

Informational security policies depend on user compliance to protect computing resources. When users are lax with security measures and fail to follow security related directives, data breaches are more likely to occur. We hypothesize that users are giving up on security compliance because they believe breaches are inevitable. We test the explanatory theory of security capitulation based on technology threat avoidance, protection motivation theory, and technology acceptance theory. Capitulation theory examines distrust of security, theft of privacy, vulnerability, security threats and security self-efficacy as predictors of capitulation, which influences motivation to protect computing assets. Results indicate that distrust of security, loss of privacy, vulnerability, threats, and self-efficacy are significantly related to capitulation. This research provides insight into emerging explanations of continuing organizational security failures.
User experience framework for understanding user experience in blockchain services

Since the advent of blockchain, interest in the technology has rapidly increased, bringing the launch of new blockchain services. User experience (UX) research provides a basis for systematically understanding and improving the experience of users, and must be carried out regardless of service type. However, it seems that research on the UX of blockchain services has not kept up with the speed with which blockchain technology has been implemented. To remedy this situation, this study proposes a UX framework for understanding the nature of UX in blockchain services. An extensive literature survey related to UX, blockchain technology, and blockchain services was conducted to define UX in blockchain services and identify the elements of UX. Blockchain functions and values that had been newly introduced in blockchain services were identified to help clarify the nature of UX in blockchain services. The proposed framework consists of two parts: a general UX and a technological UX. The elements and sub-elements of the defined UX are derived with detailed explanations. The proposed framework was applied to four blockchain services as a case study. The appropriateness and applicability of the framework are supported by the results of this case study. This study is expected to inspire researchers to provide enhanced UX in blockchain services in the future.
ARENA: The Augmented Reality Edge Networking Architecture

Many have predicted the future of the Web to be the integration of Web content with the real-world through technologies such as Augmented Reality (AR). This has led to the rise of Extended Reality (XR) Web Browsers used to shorten the long AR application development and deployment cycle of native applications especially across different platforms. As XR Browsers mature, we face new challenges related to collaborative and multi-user applications that span users, devices, and machines. These collaborative XR applications require: (1) networking support for scaling to many users, (2) mechanisms for content access control and application isolation, and (3) the ability to host application logic near clients or data sources to reduce application latency. In this paper, we present the design and evaluation of the AR Edge Networking Architecture (ARENA) which is a platform that simplifies building and hosting collaborative XR applications on WebXR capable browsers. ARENA provides a number of critical components including: a hierarchical geospatial directory service that connects users to nearby servers and content, a token-based authentication system for controlling user access to content, and an application/service runtime supervisor that can dispatch programs across any network connected device. All of the content within ARENA exists as endpoints in a PubSub scene graph model that is synchronized across all users. We evaluate ARENA in terms of client performance as well as benchmark end-to-end response-time as load on the system scales. We show the ability to horizontally scale the system to Internet-scale with scenes containing hundreds of users and latencies on the order of tens of milliseconds. Finally, we highlight projects built using ARENA and showcase how our approach dramatically simplifies collaborative multi-user XR development compared to monolithic approaches.
Fusing ASR Outputs in Joint Training for Speech Emotion Recognition

Alongside acoustic information, linguistic features based on speech transcripts have been proven useful in Speech Emotion Recognition (SER). However, due to the scarcity of emotion labelled data and the difficulty of recognizing emotional speech, it is hard to obtain reliable linguistic features and models in this research area. In this paper, we propose to fuse Automatic Speech Recognition (ASR) outputs into the pipeline for joint training SER. The relationship between ASR and SER is understudied, and it is unclear what and how ASR features benefit SER. By examining various ASR outputs and fusion methods, our experiments show that in joint ASR-SER training, incorporating both ASR hidden and text output using a hierarchical co-attention fusion approach improves the SER performance the most. On the IEMOCAP corpus, our approach achieves 63.4% weighted accuracy, which is close to the baseline results achieved by combining ground-truth transcripts. In addition, we also present novel word error rate analysis on IEMOCAP and layer-difference analysis of the Wav2vec 2.0 model to better understand the relationship between ASR and SER.
Staying Active While Staying Home: The Use of Physical Activity Technologies During Life Disruptions

One impact of the Covid-19 lockdowns was a restriction on people's ability to engage in physical activity in previously routine ways. This paper presents a two-stage mixed-method study exploring how people used technology to stay physically active during this period. We found that activity trackers reminded people to be active, while virtual coaching (i.e., video tutorials and online classes) helped them stay connected. The lockdown increased people's awareness of their activity levels and removed barriers to exercise, for example by giving them greater control over their time. However, it also created new challenges, with lack of time and space, injuries due to sudden changes in activity, and anxiety around lockdown, putting limits on physical activity. We highlight future directions that must be addressed to maximise the benefits of physical activity technologies for people trying to stay active during major life disruptions.
Ultra-High-Definition Image HDR Reconstruction via Collaborative Bilateral Learning

Existing single image high dynamic range (HDR) reconstruction methods attempt to expand the range of illuminance. They are not effective in generating plausible textures and colors in the reconstructed results, especially for high-density pixels in ultra-high-definition (UHD) images. To address these problems, we propose a new HDR reconstruction network for UHD images by collaboratively learning color and texture details. First, we propose a dual-path network to extract the content and chromatic features at a reduced resolution of the low dynamic range (LDR) input. These two types of features are used to fit bilateral-space affine models for real-time HDR reconstruction. To extract the main data structure of the LDR input, we propose to use 3D Tucker decomposition and reconstruction to prevent pseudo edges and noise amplification in the learned bilateral grid. As a result, the high-quality content and chromatic features can be reconstructed capitalized on guided bilateral upsampling. Finally, we fuse these two full-resolution feature maps into the HDR reconstructed results. Our proposed method can achieve real-time processing for UHD images (about 160 fps). Experimental results demonstrate that the proposed algorithm performs favorably against the state-of-the-art HDR reconstruction approaches on public benchmarks and real-world UHD images.
On Merits and Viability of Multi-Cloud Serverless

Serverless computing is a rapidly growing paradigm in the cloud industry that envisions functions as the computational building blocks of an application. Instead of forcing the application developer to provision cloud resources for their application, the cloud provider provisions the required resources for each function "under the hood." In this work, we envision virtual serverless providers (VSPs) to aggregate serverless offerings. In doing so, VSPs allow developers (and businesses) to get rid of vendor lock-in problems and exploit pricing and performance variation across providers by adaptively utilizing the best provider at each time, forcing the providers to compete to offer cheaper and superior services. We discuss the merits of a VSP and show that serverless systems are well-suited to cross-provider aggregation, compared to virtual machines. We propose a VSP system architecture and implement an initial version. Using experimental evaluations, our preliminary results show that a VSP can improve maximum sustained throughput by 1.2x to 4.2x, reduces SLO violations by 98.8%, and improves the total invocations' costs by 54%.
Self-Supervised Learning for Anomaly Detection With Dynamic Local Augmentation

Anomaly detection is an important problem for recent advances in machine learning.To this end, many attempts have emerged to detect unknown anomalies of the images by learning representations and designing score functions.In this paper, we propose a simple yet effective framework for unsupervised anomaly detection using self-supervised learning.We extend conventional self-supervised learning for an anomaly detection problem.In anomaly detection, anomalous patterns appear in the local regions of an image, so we employ dynamic local augmentation to generate a negative pair of the images from the normal training dataset.Specifically, in addition to learning the global representation of an image, our framework contrasts a normal sample to a locally augmented sample.To effectively apply the local augmentations regardless of a category or a random location of an image, we use dynamically weighted local augmentations to generate more suitable negative samples.We also present a novel scoring function for detecting unseen anomalous patterns.Our experiment demonstrates the effectiveness of our method, and we show that our framework achieves competitive performance compared to state-of-the-art methods on MVTec Anomaly Detection dataset.