Reviewerly/exHarmony huggingface dataset:
Incorporating Explicit Subtopics in Personalized Search

The key to personalized search is modeling user intents to tailor returned results for different users. Existing personalized methods mainly focus on learning implicit user interest vectors. In this paper, we propose ExpliPS, a personalized search model that explicitly incorporates query subtopics into personalization. It models the user’s current intent by estimating the user’s preference over the subtopics of the current query and personalizes the results over the weighted subtopics. We think that in such a way, personalized search could be more explainable and stable. Specifically, we first employ a semantic encoder to learn the representations of the user’s historical behaviours. Then with the historical behaviour representations, a subtopic preference encoder is devised to predict the user’s subtopic preferences on the current query. Finally, we rerank the candidates via a subtopic-aware ranker that prioritizes the documents relevant to the user-preferred subtopics. Experimental results show our model ExpliPS outperforms the state-of-the-art personalized web search models with explainable and stable results.
Continual Few-shot Learning with Transformer Adaptation and Knowledge Regularization

Continual few-shot learning, as a paradigm that simultaneously solves continual learning and few-shot learning, has become a challenging problem in machine learning. An eligible continual few-shot learning model is expected to distinguish all seen classes upon new categories arriving, where each category only includes very few labeled data. However, existing continual few-shot learning methods only consider the visual modality, where the distributions of new categories often indistinguishably overlap with old categories, thus resulting in the severe catastrophic forgetting problem. To tackle this problem, in this paper we study continual few-shot learning with the assistance of semantic knowledge by simultaneously taking both visual modality and semantic concepts of categories into account. We propose a Continual few-shot learning algorithm with Semantic knowledge Regularization (CoSR) for adapting to the distribution changes of visual prototypes through a Transformer-based prototype adaptation mechanism. Specifically, the original visual prototypes from the backbone are fed into the well-designed Transformer with corresponding semantic concepts, where the semantic concepts are extracted from all categories. The semantic-level regularization forces the categories with similar semantics to be closely distributed, while the opposite ones are constrained to be far away from each other. The semantic regularization improves the model’s ability to distinguish between new and old categories, thus significantly mitigating the catastrophic forgetting problem in continual few-shot learning. Extensive experiments on CIFAR100, miniImageNet, CUB200 and an industrial dataset with long-tail distribution demonstrate the advantages of our CoSR model compared with state-of-the-art methods.
The Benefits of Vulnerability Discovery and Bug Bounty Programs: Case Studies of Chromium and Firefox

Recently, bug-bounty programs have gained popularity and become a significant part of the security culture of many organizations. Bug-bounty programs enable organizations to enhance their security posture by harnessing the diverse expertise of crowds of external security experts (i.e., bug hunters). Nonetheless, quantifying the benefits of bug-bounty programs remains elusive, which presents a significant challenge for managing them. Previous studies focused on measuring their benefits in terms of the number of vulnerabilities reported or based on the properties of the reported vulnerabilities, such as severity or exploitability. However, beyond these inherent properties, the value of a report also depends on the probability that the vulnerability would be discovered by a threat actor before an internal expert could discover and patch it. In this paper, we present a data-driven study of the Chromium and Firefox vulnerability-reward programs. First, we estimate the difficulty of discovering a vulnerability using the probability of rediscovery as a novel metric. Our findings show that vulnerability discovery and patching provide clear benefits by making it difficult for threat actors to find vulnerabilities; however, we also identify opportunities for improvement, such as incentivizing bug hunters to focus more on development releases. Second, we compare the types of vulnerabilities that are discovered internally vs. externally and those that are exploited by threat actors. We observe significant differences between vulnerabilities found by external bug hunters, internal security teams, and external threat actors, which indicates that bug-bounty programs provide an important benefit by complementing the expertise of internal teams, but also that external hunters should be incentivized more to focus on the types of vulnerabilities that are likely to be exploited by threat actors.
EdgeMove: Pipelining Device-Edge Model Training for Mobile Intelligence

Training machine learning (ML) models on mobile and Web-of-Things (WoT) has been widely acknowledged and employed as a promising solution to privacy-preserving ML. However, these end-devices often suffer from constrained resources and fail to accommodate increasingly large ML models that crave great computation power. Offloading ML models partially to the cloud for training strikes a trade-off between privacy preservation and resource requirements. However, device-cloud training creates communication overheads that delay model training tremendously. This paper presents EdgeMove, the first device-edge training scheme that enables fast pipelined model training across edge devices and edge servers. It employs probing-based mechanisms to tackle the new challenges raised by device-edge training. Before training begins, it probes nearby edge servers’ training performance and bootstraps model training by constructing a training pipeline with an approximate model partitioning. During the training process, EdgeMove accommodates user mobility and system dynamics by probing nearby edge servers’ training performance adaptively and adapting the training pipeline proactively. Extensive experiments are conducted with two popular DNN models trained on four datasets for three ML tasks. The results demonstrate that EdgeMove achieves a 1.3 × -2.1 × speedup over the state-of-the-art scheme.
SeeGera: Self-supervised Semi-implicit Graph Variational Auto-encoders with Masking

Generative graph self-supervised learning (SSL) aims to learn node representations by reconstructing the input graph data. However, most existing methods focus on unsupervised learning tasks only and very few work has shown its superiority over the state-of-the-art graph contrastive learning (GCL) models, especially on the classification task. While a very recent model has been proposed to bridge the gap, its performance on unsupervised learning tasks is still unknown. In this paper, to comprehensively enhance the performance of generative graph SSL against other GCL models on both unsupervised and supervised learning tasks, we propose the SeeGera model, which is based on the family of self-supervised variational graph auto-encoder (VGAE). Specifically, SeeGera adopts the semi-implicit variational inference framework, a hierarchical variational framework, and mainly focuses on feature reconstruction and structure/feature masking. On the one hand, SeeGera co-embeds both nodes and features in the encoder and reconstructs both links and features in the decoder. Since feature embeddings contain rich semantic information on features, they can be combined with node embeddings to provide fine-grained knowledge for feature reconstruction. On the other hand, SeeGera adds an additional layer for structure/feature masking to the hierarchical variational framework, which boosts the model generalizability. We conduct extensive experiments comparing SeeGera with 9 other state-of-the-art competitors. Our results show that SeeGera can compare favorably against other state-of-the-art GCL methods in a variety of unsupervised and supervised learning tasks.
NetGuard: Protecting Commercial Web APIs from Model Inversion Attacks using GAN-generated Fake Samples

Recently more and more cloud service providers (e.g., Microsoft, Google, and Amazon) have commercialized their well-trained deep learning models by providing limited access via web API interfaces. However, it is shown that these APIs are susceptible to model inversion attacks, where attackers can recover the training data with high fidelity, which may cause serious privacy leakage.Existing defenses against model inversion attacks, however, hinder the model performance and are ineffective for more advanced attacks, e.g., Mirror [4]. In this paper, we proposed NetGuard, a novel utility-aware defense methodology against model inversion attacks (MIAs). Unlike previous works that perturb prediction outputs of the victim model, we propose to mislead the MIA effort by inserting engineered fake samples during the training process. A generative adversarial network (GAN) is carefully built to construct fake training samples to mislead the attack model without degrading the performance of the victim model. Besides, we adopt continual learning to further improve the utility of the victim model. Extensive experiments on CelebA, VGG-Face, and VGG-Face2 datasets show that NetGuard is superior to existing defenses, including DP [37] and Ad-mi [32] on state-of-the-art model inversion attacks, i.e., DMI [8], Mirror [4], Privacy [12], and Alignment [34].
Cross-Modality Mutual Learning for Enhancing Smart Contract Vulnerability Detection on Bytecode

Over the past couple of years, smart contracts have been plagued by multifarious vulnerabilities, which have led to catastrophic financial losses. Their security issues, therefore, have drawn intense attention. As countermeasures, a family of tools has been developed to identify vulnerabilities in smart contracts at the source-code level. Unfortunately, only a small fraction of smart contracts is currently open-sourced. Another spectrum of work is presented to deal with pure bytecode, but most such efforts still suffer from relatively low performance due to the inherent difficulty in restoring abundant semantics in the source code from the bytecode.
Message Function Search for Knowledge Graph Embedding

Recently, many promising embedding models have been proposed to embed knowledge graphs (KGs) and their more general forms, such as n-ary relational data (NRD) and hyper-relational KG (HKG). To promote the data adaptability and performance of embedding models, KG searching methods propose to search for suitable models for a given KG data set. But they are restricted to a single KG form, and the searched models are restricted to a single type of embedding model. To tackle such issues, we propose to build a search space for the message function in graph neural networks (GNNs). However, it is a non-trivial task. Existing message function designs fix the structures and operators, which makes them difficult to handle different KG forms and data sets. Therefore, we first design a novel message function space, which enables both structures and operators to be searched for the given KG form (including KG, NRD, and HKG) and data. The proposed space can flexibly take different KG forms as inputs and is expressive to search for different types of embedding models. Especially, some existing message function designs and some classic KG embedding models can be instantiated as special cases of our space. We empirically show that the searched message functions are data-dependent, and can achieve leading performance on benchmark KGs, NRD, and HKGs.
Provenance of Training without Training Data: Towards Privacy-Preserving DNN Model Ownership Verification

In the era of deep learning, it is critical to protect the intellectual property of high-performance deep neural network (DNN) models. Existing proposals, however, are subject to adversarial ownership forgery (e.g., methods based on watermarks or fingerprints) or require full access to the original training dataset for ownership verification (e.g., methods requiring the replay of the learning process). In this paper, we propose a novel Provenance of Training (PoT) scheme, the first empirical study towards verifying DNN model ownership without accessing any original dataset while being robust against existing attacks. At its core, PoT relies on a coherent model chain built from the intermediate checkpoints saved during model training to serve as the ownership certificate. Through an in-depth analysis of model training, we propose six key properties that a legitimate model chain shall naturally hold. In contrast, it is difficult for the adversary to forge a model chain that satisfies these properties simultaneously without performing actual training. We systematically analyze PoT’s robustness against various possible attacks, including the adaptive attacks that are designed given the full knowledge of PoT’s design, and further perform extensive empirical experiments to demonstrate our security analysis.
Cross-domain Recommendation with Behavioral Importance Perception

Cross-domain recommendation (CDR) aims to leverage the source domain information to provide better recommendation for the target domain, which is widely adopted in recommender systems to alleviate the data sparsity and cold-start problems. However, existing CDR methods mostly focus on designing effective model architectures to transfer the source domain knowledge, ignoring the behavior-level effect during the loss optimization process, where behaviors regarding different aspects in the source domain may have different importance for the CDR model optimization. The ignorance of the behavior-level effect will cause the carefully designed model architectures ending up with sub-optimal parameters, which limits the recommendation performance. To tackle the problem, we propose a generic behavioral importance-aware optimization framework for cross-domain recommendation (BIAO). Specifically, we propose a behavioral perceptron which predicts the importance of each source behavior according to the corresponding item’s global impact and local user-specific impact. The joint optimization process of the CDR model and the behavioral perceptron is formulated as a bi-level optimization problem. In the lower optimization, only the CDR model is updated with weighted source behavior loss and the target domain loss, while in the upper optimization, the behavioral perceptron is updated with implicit gradient from a developing dataset obtained through the proposed reorder-and-reuse strategy. Extensive experiments show that our proposed optimization framework consistently improves the performance of different cross-domain recommendation models in 7 cross-domain scenarios, demonstrating that our method can serve as a generic and powerful tool for cross-domain recommendation1.
Meta-GNAS: Meta-reinforcement learning for graph neural architecture search

Graph Neural architecture search (GNAS) has shown great success in designing many prominent models on non-Euclidean data. However, the existing GNAS methods need to search from scratch on new tasks, which is time-consuming and inefficient in real application scenarios. In this paper, we propose a meta-reinforcement learning method for Graph Neural Architecture Search (Meta-GNAS) to improve the learning efficiency on new tasks by leveraging the knowledge learned from previous tasks. As far as we know, it is the first work that applies meta-learning to GNAS tasks. Moreover, to further improve the efficiency in tackling a new task, we use a predictive model to evaluate the accuracy of sampled graph neural architecture, instead of training it from scratch. The experiment results demonstrate that the architecture designed by Meta-GNAS outperforms the state-of-art manually designed architectures, and the search speed is faster than other search methods, with an average search time of fewer than 210 GPU seconds on 6 datasets.
Identifying Creative Harmful Memes via Prompt based Approach

The creative nature of memes has made it possible for harmful content to spread quickly and widely on the internet. Harmful memes can range from spreading hate speech promoting violence, and causing emotional distress to individuals or communities. These memes are often designed to be misleading, manipulative, and controversial, making it challenging to detect and remove them from online platforms. Previous studies focused on how to fuse visual and language modalities to capture contextual information. However, meme analysis still severely suffers from data deficiency, resulting in insufficient learning of fusion modules. Further, using conventional pretrained encoders for text and images exhibits a greater semantic gap in feature spaces and leads to low performance. To address these gaps, this paper reformulates a harmful meme analysis as an auto-filling and presents a prompt-based approach to identify harmful memes. Specifically, we first transform multimodal data to a single (i.e., textual) modality by generating the captions and attributes of the visual data and then prepend the textual data in the prompt-based pre-trained language model. Experimental results on two benchmark harmful memes datasets demonstrate that our method outperformed state-of-the-art methods. We conclude with the transferability and robustness of our approach to identify creative harmful memes.
SignBERT+: Hand-Model-Aware Self-Supervised Pre-Training for Sign Language Understanding

Hand gesture serves as a crucial role during the expression of sign language. Current deep learning based methods for sign language understanding (SLU) are prone to over-fitting due to insufficient sign data resource and suffer limited interpretability. In this paper, we propose the first self-supervised pre-trainable SignBERT+ framework with model-aware hand prior incorporated. In our framework, the hand pose is regarded as a visual token, which is derived from an off-the-shelf detector. Each visual token is embedded with gesture state and spatial-temporal position encoding. To take full advantage of current sign data resource, we first perform self-supervised learning to model its statistics. To this end, we design multi-level masked modeling strategies (joint, frame and clip) to mimic common failure detection cases. Jointly with these masked modeling strategies, we incorporate model-aware hand prior to better capture hierarchical context over the sequence. After the pre-training, we carefully design simple yet effective prediction heads for downstream tasks. To validate the effectiveness of our framework, we perform extensive experiments on three main SLU tasks, involving isolated and continuous sign language recognition (SLR), and sign language translation (SLT). Experimental results demonstrate the effectiveness of our method, achieving new state-of-the-art performance with a notable gain.
A novel end-to-end dual-camera system for eye gaze synchrony assessment in face-to-face interaction

Quantification of face-to-face interaction can provide highly relevant information in cognitive and psychological science research. Current commercial glint-dependent solutions suffer from several disadvantages and limitations when applied in face-to-face interaction, including data loss, parallax errors, the inconvenience and distracting effect of wearables, and/or the need for several cameras to capture each person. Here we present a novel eye-tracking solution, consisting of a dual-camera system used in conjunction with an individually optimized deep learning approach that aims to overcome some of these limitations. Our data show that this system can accurately classify gaze location within different areas of the face of two interlocutors, and capture subtle differences in interpersonal gaze synchrony between two individuals during a (semi-)naturalistic face-to-face interaction.
Learning Target Point Seeking Weights Spatial–Temporal Regularized Correlation Filters for Visual Tracking

Spatial–temporal regularized correlation filtering is an effective tracking algorithm. Still, problems such as target occlusion, background clutter and out of view caused by target movement are inevitable in the running process. This paper proposes a target point seeking weights spatial–temporal regularized correlation filters to tackle target loss. The alternating direction method of multipliers is used to simplify the spatial–temporal regularized term in the objective function, obtain the optimal solution of the filter and auxiliary factor, reduce the algorithm’s complexity, and realize the adaptability of time and space. The target weight response model is obtained through the target point finding weight, the target motion information is extracted, the target motion state and motion trajectory are predicted, and the interference information such as the algorithm’s ability to identify the target and the ability to distinguish the target is enhanced. With the help of the weighted least squares method, the maximum response value of the weight is obtained, and the target position is determined. Experiments show that, compared with other mainstream correlation filtering algorithms, the proposed algorithm has higher tracking accuracy and stronger robustness under complex conditions.
Tangible User Interfaces for Enhancing User Experience of Virtual Reality Cultural Heritage Applications for Utilization in Educational Environment

Museums are traditionally considered learning environments and are ordinarily used for non-formal education. Physical museums, while being irreplaceable, are limited to a physical space, requiring mobility and physical presence. In addition, traditional exhibitions are not designed for interaction and physical exploration of artefacts. With the focus being shifted from museum exhibits to visitors’ experience, utilization of emerging technologies and co-creation of virtual museums not only helps in preservation of cultural heritage, but enhances the dissemination, engagement, and experience, while addressing the mobility and the plurality of voices and perspectives represented. In this work, we designed and developed the School House Virtual Museum with tangible user interfaces based on participatory, interdisciplinary, and co-creative methods with students and a larger community of researchers, artists, and practitioners working on heritage and memory. In a user study with 62 participants, usability and user experience were explored and the potential contribution of such virtual museums to learning, based on critical, cross-disciplinary, and participatory dialogue, both in cultural and educational institutions/programs has been investigated. The results have confirmed that the system has been well designed and developed, and the user experience was largely positive. The responses from educators and students confirmed that the application holds potential as a learning and education tool in either museums, schools, or when used independently.