Reviewerly/exHarmony huggingface dataset:
Causal Inference for Knowledge Graph Based Recommendation

Knowledge Graph (KG), as a side-information, tends to be utilized to supplement the collaborative filtering (CF) based recommendation model. By mapping items with the entities in KGs, prior studies mostly extract the knowledge information from the KGs and inject it into the representations of users and items. Despite their remarkable performance, they fail to model the user preference on attribute in the KG, since they ignore that (1) the structure information of KG may hinder the user preference learning, and (2) the user's interacted attributes will result in the bias issue on the similarity scores. With the help of causality tools, we construct the causal-effect relation between the variables in KG-based recommendation and identify the reasons causing the mentioned challenges. Accordingly, we develop a new framework, termed Knowledge Graph-based Causal Recommendation (KGCR), which implements the deconfounded user preference learning and adopts counterfactual inference to eliminate bias in the similarity scoring. Ultimately, we evaluate our proposed model on three datasets, including Amazon-book, LastFM, and Yelp2018 datasets. By conducting extensive experiments on the datasets, we demonstrate that KGCR outperforms several state-of-the-art baselines, such as KGNN-LS (Wang et al., 2019), KGAT (Wang et al., 2019) and KGIN (Wang et al., 2021).
Spatiotemporal Self-attention Modeling with Temporal Patch Shift for Action Recognition

Transformer-based methods have recently achieved great advancement on 2D image-based vision tasks. For 3D video-based tasks such as action recognition, however, directly applying spatiotemporal transformers on video data will bring heavy computation and memory burdens due to the largely increased number of patches and the quadratic complexity of self-attention computation. How to efficiently and effectively model the 3D self-attention of video data has been a great challenge for transformers. In this paper, we propose a Temporal Patch Shift (TPS) method for efficient 3D self-attention modeling in transformers for video-based action recognition. TPS shifts part of patches with a specific mosaic pattern in the temporal dimension, thus converting a vanilla spatial self-attention operation to a spatiotemporal one with little additional cost. As a result, we can compute 3D self-attention using nearly the same computation and memory cost as 2D self-attention. TPS is a plug-and-play module and can be inserted into existing 2D transformer models to enhance spatiotemporal feature learning. The proposed method achieves competitive performance with state-of-the-arts on Something-something V1 & V2, Diving-48, and Kinetics400 while being much more efficient on computation and memory cost. The source code of TPS can be found at https://github.com/MartinXM/TPS.
Global Learnable Attention for Single Image Super-Resolution

Self-similarity is valuable to the exploration of non-local textures in single image super-resolution (SISR). Researchers usually assume that the importance of non-local textures is positively related to their similarity scores. In this paper, we surprisingly found that when repairing severely damaged query textures, some non-local textures with low-similarity which are closer to the target can provide more accurate and richer details than the high-similarity ones. In these cases, low-similarity does not mean inferior but is usually caused by different scales or orientations. Utilizing this finding, we proposed a Global Learnable Attention (GLA) to adaptively modify similarity scores of non-local textures during training instead of only using a fixed similarity scoring function such as the dot product. The proposed GLA can explore non-local textures with low-similarity but more accurate details to repair severely damaged textures. Furthermore, we propose to adopt Super-Bit Locality-Sensitive Hashing (SB-LSH) as a preprocessing method for our GLA. With the SB-LSH, the computational complexity of our GLA is reduced from quadratic to asymptotic linear with respect to the image size. In addition, the proposed GLA can be integrated into existing deep SISR models as an efficient general building block. Based on the GLA, we constructed a Deep Learnable Similarity Network (DLSN), which achieves state-of-the-art performance for SISR tasks of different degradation types (e.g., blur and noise). Our code and a pre-trained DLSN have been uploaded to GitHub† for validation.
Non-Aligned Multi-View Multi-Label Classification via Learning View-Specific Labels

In the multi-view multi-label (MVML) classification problem , multiple views are simultaneously associated with multiple semantic representations.Multi-view multi-label learning inevitably has the problems of consistency, diversity, and non-alignment among views and the correlation among labels.Most of the existing multi-view multi-label methods for nonaligned views assume that each view has a common or shared label set, but because a single view cannot contain the entire label information, they often learn suboptimal results.Based on this, this paper proposes a non-aligned multi-view multilabel classification method that learns view-specific labels (LVSL), aiming to explicitly mine the information of view-specific labels and low-rank label structures in non-aligned views in a unified model framework.Furthermore, to alleviate insufficient available label information, we thoroughly explored the global and local structural information among labels.Specifically, first, we assume that there is structural consistency between the view and the label space and then construct the view-specific label model in turn.Second, to enrich the original label space information, we mine the consistent information of multiple views and the low-rank correlation information hidden among multiple labels.Finally, the contribution weight of each view is combined with learning the complementary information among the views in the decisionmaking stage, and extend the model to handle nonlinear data.The results of the proposed method compared with existing stateof-the-art algorithms on several datasets validate its effectiveness.
Sample Condensation in Online Continual Learning

Online Continual learning is a challenging learning scenario where the model must learn from a non-stationary stream of data where each sample is seen only once. The main challenge is to incrementally learn while avoiding catastrophic forgetting, namely the problem of forgetting previously acquired knowledge while learning from new data. A popular solution in these scenario is to use a small memory to retain old data and rehearse them over time. Unfortunately, due to the limited memory size, the quality of the memory will deteriorate over time. In this paper we propose OLCGM, a novel replay-based continual learning strategy that uses knowledge condensation techniques to continuously compress the memory and achieve a better use of its limited size. The sample condensation step compresses old samples, instead of removing them like other replay strategies. As a result, the experiments show that, whenever the memory budget is limited compared to the complexity of the data, OLCGM improves the final accuracy compared to state-of-the-art replay strategies.
A DTCWT-SVD Based Video Watermarking Resistant to Frame Rate Conversion

Videos can be easily tampered, copied and redistributed by attackers for illegal and monetary usage. Such behaviors severely jeopardize the interest of content owners. Despite huge efforts made in digital video watermarking for copyright protection, typical distortions in video transmission can still easily erase the embedded signal. Among them, temporal synchronization attacks is one of the most prevalent attacks. To address this issue, we present a new video watermarking based on joint Dual-Tree Cosine Wavelet Transform (DTCWT) and Singular Value Decomposition (SVD), which is resistant to frame rate conversion. We simulate the watermark embedding by adjusting the shape of candidate coefficient and perform group-level watermarking that includes moderate temporal redundancy to resist temporal synchronization attacks. Extensive experimental results show that the proposed scheme is more resilient to temporal synchronization attacks and performs better than the existing blind video watermarking schemes.
FINet: A Feature Interaction Network for SAR Ship Object-Level and Pixel-Level Detection

Deep learning-based detection methods have achieved great success in ship target detection in synthetic aperture radar (SAR) images. However, due to the interference of imaging mechanism, speckle noise, and sea and land clutter, ship detection in SAR images still suffers from difficult interpretation. It is found that most ship detection algorithms focus on object-level detection while ignoring pixel-level information. In order to further improve the recognition effectiveness and positioning accuracy of ships in SAR images, we present a novel ship detection method based on a feature interaction network (FINet) in SAR images from the perspective of object-level and pixel-level. FINet consists of an object-level detection network and a pixel-level detection network. The information of the two branches is fused through the feature interaction module (FIM), and then the object-level information and pixel-level information are enhanced by the feature guidance module (FGM). Finally, FINet utilizes object-level and pixel-level detection heads for prediction and regression to obtain object-level classification accuracy, positioning bounding box coordinates, and pixel-level binary classification results. The experimental results demonstrate that the classification effectiveness and localization accuracy of FINet are better than those of the comparison algorithms, and FINet achieves the best performance.
Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learning

We present DOME, a novel method for one-shot imitation learning, where a task can be learned from just a single demonstration and then be deployed immediately, without any further data collection or training. DOME does not require prior task or object knowledge, and can perform the task in novel object configurations and with distractors. At its core, DOME uses an image-conditioned object segmentation network followed by a learned visual servoing network, to move the robot's end-effector to the same relative pose to the object as during the demonstration, after which the task can be completed by replaying the demonstration's end-effector velocities. We show that DOME achieves near 100% success rate on 7 real-world everyday tasks, and we perform several studies to thoroughly understand each individual component of DOME. Videos and supplementary material are available at: https://www.robot-learning.uk/dome.
DreamingV2: Reinforcement Learning with Discrete World Models without Reconstruction

The present paper proposes a novel reinforce-ment learning method with world models, DreamingV2, a collaborative extension of DreamerV2 and Dreaming. Dream- erV2 is a cutting-edge model-based reinforcement learning from pixels that uses discrete world models to represent latent states with categorical variables. Dreaming is also a form of reinforcement learning from pixels that attempts to avoid the auto encoding process in general world model training by involving a reconstruction-free contrastive learning objective. The proposed DreamingV2 is a novel approach of adopting both the discrete representation of DreamingV2 and the reconstruction-free objective of Dreaming. Compared to DreamerV2 and other recent model-based methods without reconstruction, DreamingV2 achieves the best scores on five simulated challenging 3D robot arm tasks. We believe that DreamingV2 will be a reliable solution for robot learning since its discrete representation is suitable to describe discontinuous environments, and the reconstruction-free fashion well manages complex vision observations.
PI-Cut-Choo and Friends: Compact Blind Signatures via Parallel Instance Cut-and-Choose and More

Blind signature schemes are one of the best-studied tools for privacy-preserving authentication. Unfortunately, known constructions of provably secure blind signatures either rely on non-standard hardness assumptions, or require parameters that grow linearly with the number of concurrently issued signatures, or involve prohibitively inefficient general techniques such as general secure two-party computation. Recently, Katz, Loss and Rosenberg (ASIACRYPT’21) gave a technique that, for the security parameter n, transforms blind signature schemes secure for $$O(\log n)$$ concurrent executions of the blind signing protocol into ones that are secure for any $$\textsf{poly}(n)$$ concurrent executions. This transform has two drawbacks that we eliminate in this paper: 1) the communication complexity of the resulting blind signing protocol grows linearly with the number of signing interactions; 2) the resulting schemes inherit a very loose security bound from the underlying scheme and, as a result, require impractical parameter sizes. In this work, we give an improved transform for obtaining a secure blind signing protocol tolerating any $$\textsf{poly}(n)$$ concurrent executions from one that is secure for $$O(\log n)$$ concurrent executions. While preserving the advantages of the original transform, the communication complexity of our new transform only grows logarithmically with the number of interactions. Under the CDH and RSA assumptions, we improve on this generic transform in terms of concrete efficiency and give (1) a BLS-based blind signature scheme over a standard-sized group where signatures are of size roughly 3 KB and communication per signature is roughly 120 KB; and (2) an Okamoto-Guillou-Quisquater-based blind signature scheme with signatures and communication of roughly 9 KB and 8 KB, respectively.
Hierarchical Reinforcement Learning for Air Combat at DARPA's AlphaDogfight Trials

Autonomous control in high-dimensional, continuous state spaces is a persistent and important challenge in the fields of robotics and artificial intelligence. Because of high risk and complexity, the adoption of AI for autonomous combat systems has been a long-standing difficulty. In order to address these issues, DARPA's AlphaDogfight Trials (ADT) program sought to vet the feasibility of and increase trust in AI for autonomously piloting an F-16 in simulated air-to-air combat. Our submission to ADT solves the high-dimensional, continuous control problem using a novel hierarchical deep reinforcement learning approach consisting of a high-level policy selector and a set of separately trained low-level policies specialized for excelling in specific regions of the state space. Both levels of the hierarchy are trained using off-policy, maximum entropy methods with expert knowledge integrated through reward shaping. Our approach outperformed human expert pilots and achieved a second-place rank in the ADT championship event.
Communication-Efficient Online Federated Learning Strategies for Kernel Regression

This article presents communication-efficient approaches to federated learning (FL) for resource-constrained devices with access to streaming data. In particular, we first propose a partial-sharing-based framework for online federated learning (PSO-Fed), wherein clients update local models from a stream of data and exchange tiny fractions of the model with the server, reducing the communication overhead. In contrast to classical FL approaches, the proposed strategy provides clients who are not part of a global iteration with the freedom to update local models whenever new data arrives. Furthermore, by devising a client-side innovation check, we also propose an event-triggered PSO-Fed (ETPSO-Fed) that further reduces the computational burden of clients while enhancing communication efficiency. We implement the above-mentioned frameworks in the context of kernel regression, where clients perform local learning employing random Fourier features (RFFs)-based kernel least mean squares. In addition, we examine the mean and mean-square convergence of the proposed PSO-Fed. Finally, we conduct experiments to determine the efficacy of the proposed frameworks. Our results show that PSO-Fed and ETPSO-Fed can compete with Online-Fed while requiring significantly less communication overhead. Simulations demonstrate an 80% reduction in PSO-Fed and an 84.5% reduction in ETPSO-Fed communication overhead compared to Online-Fed. Notably, the proposed PSO-Fed strategies show good resilience against model-poisoning attacks without involving additional mechanisms.
Meta Spatio-Temporal Debiasing for Video Scene Graph Generation

Video scene graph generation (VidSGG) aims to parse the video content into scene graphs, which involves modeling the spatio-temporal contextual information in the video. However, due to the long-tailed training data in datasets, the generalization performance of existing VidSGG models can be affected by the spatio-temporal conditional bias problem. In this work, from the perspective of meta-learning, we propose a novel Meta Video Scene Graph Generation (MVSGG) framework to address such a bias problem. Specifically, to handle various types of spatio-temporal conditional biases, our framework first constructs a support set and a group of query sets from the training data, where the data distribution of each query set is different from that of the support set w.r.t. a type of conditional bias. Then, by performing a novel meta training and testing process to optimize the model to obtain good testing performance on these query sets after training on the support set, our framework can effectively guide the model to learn to well generalize against biases. Extensive experiments demonstrate the efficacy of our proposed framework.
Fast and Unsupervised Action Boundary Detection for Action Segmentation

To deal with the great number of untrimmed videos produced every day, we propose an efficient unsupervised action segmentation method by detecting boundaries, named action boundary detection (ABD). In particular, the proposed method has the following advantages: no training stage and low-latency inference. To detect action boundaries, we estimate the similarities across smoothed frames, which inherently have the properties of internal consistency within actions and external discrepancy across actions. Under this circumstance, we successfully transfer the boundary detection task into the change point detection based on the similarity. Then, non-maximum suppression (NMS) is conducted in local windows to select the smallest points as candidate boundaries. In addition, a clustering algorithm is followed to refine the initial proposals. Moreover, we also extend ABD to the online setting, which enables real-time action segmentation in long untrimmed videos. By evaluating on four challenging datasets, our method achieves state-of-the-art performance. Moreover, thanks to the efficiency of ABD, we achieve the best trade-off between the accuracy and the inference time compared with existing unsupervised approaches.
VulDeBERT: A Vulnerability Detection System Using BERT

Deep learning technologies recently received much attention to detect vulnerable code patterns accurately. This paper proposes a new deep learning-based vulnerability detection tool dubbed VulDeBERT by fine-tuning a pre-trained language model, Bidirectional Encoder Representations from Transformers (BERT), on the vulnerable code dataset. To support VulDeBERT, we develop a new code analysis tool to extract well-represented abstract code fragments from C and C++ source code. The experimental results show that VulDeBERT outperforms the state-of-the-art tool, VulDeePecker [1] for two security vul- nerability types (CWE-119 and CWE-399). For the CWE-119 dataset, VulDeBERT achieved an Fl score of 94.6 %, which is significantly better than VulDeePecker, achieving an Fl score of 86.6 % in the same settings. Again, for the CWE-399 dataset, VulDeBERT achieved an Fl score of 97.9 %, which is also better than VulDeePecker, achieving an Fl score of 95 % in the same settings.
EleGANt: Exquisite and Locally Editable GAN for Makeup Transfer

Most existing methods view makeup transfer as transferring color distributions of different facial regions and ignore details such as eye shadows and blushes. Besides, they only achieve controllable transfer within predefined fixed regions. This paper emphasizes the transfer of makeup details and steps towards more flexible controls. To this end, we propose Exquisite and locally editable GAN for makeup transfer (EleGANt). It encodes facial attributes into pyramidal feature maps to preserves high-frequency information. It uses attention to extract makeup features from the reference and adapt them to the source face, and we introduce a novel Sow-Attention Module that applies attention within shifted overlapped windows to reduce the computational cost. Moreover, EleGANt is the first to achieve customized local editing within arbitrary areas by corresponding editing on the feature maps. Extensive experiments demonstrate that EleGANt generates realistic makeup faces with exquisite details and achieves state-of-the-art performance. The code is available at https://github.com/Chenyu-Yang-2000/EleGANt .
RapidFlow

Continuous subgraph matching (CSM) is an important building block in many real-time graph processing applications. Given a subgraph query Q and a data graph stream, a CSM algorithm reports the occurrences of Q in the stream. Specifically, when a new edge e arrives in the stream, existing CSM algorithms start from the inserted e in the current data graph G to search Q. However, this rigid matching order of always starting from e can lead to a massive number of partial results that will turn out futile. Also, if Q contains automorphisms, there will be a lot of redundant computation in the matching process. To address these two problems, we propose RapidFlow, an effective approach to CSM. First, we design a query reduction technique, which reduces CSM to batch subgraph matching (BSM) where we enumerate all results in a region of G that will be affected by the update. The well-established BSM techniques can determine effective matching orders, not necessarily starting from the newly inserted edge. Second, to eliminate redundant computation caused by automorphisms in Q , we propose dual matching, which leverages the duality of Q and G in the matching process. Extensive experiment results show that RapidFlow outperforms state-of-the-art algorithms, including TurboFlux and SymBi, by up to two orders of magnitude on various workloads.