Reviewerly/exHarmony huggingface dataset:
Visual Representation for Capturing Creator Theme in Brand-Creator Marketplace

Providing cold start recommendations in a brand-creator marketplace is challenging as brands’ preferences extend beyond the mere objects depicted in the creator’s content and encompass the creator’s individual theme consistently thatresonates across images shared on her social media profile. Furthermore, brands often use textual keywords to describe their campaign’s aesthetic appeal, with which creators must align. To address these challenges, we propose two methods: SAME (Same Account Media Embedding), a novel creator representation employing a Siamese network to capture the unique creator theme and OAAR (Object-Agnostic Adjective Representation), enabling filtering creators based on textual adjectives that relate to aesthetic qualities through zero-shot learning. These two methods utilize CLIP, a state-of-the-art language-image model, and improve it in addressing the aforementioned challenges.
RecAD: Towards A Unified Library for Recommender Attack and Defense

In recent years, recommender systems have become a ubiquitous part of our daily lives, while they suffer from a high risk of being attacked due to the growing commercial and social values. Despite significant research progress in recommender attack and defense, there is a lack of a widely-recognized benchmarking standard in the field, leading to unfair performance comparison and limited credibility of experiments. To address this, we propose RecAD, a unified library aiming at establishing an open benchmark for recommender attack and defense. RecAD takes an initial step to set up a unified benchmarking pipeline for reproducible research by integrating diverse datasets, standard source codes, hyper-parameter settings, running logs, attack knowledge, attack budget, and evaluation results. The benchmark is designed to be comprehensive and sustainable, covering both attack, defense, and evaluation tasks, enabling more researchers to easily follow and contribute to this promising field. RecAD will drive more solid and reproducible research on recommender systems attack and defense, reduce the redundant efforts of researchers, and ultimately increase the credibility and practical value of recommender attack and defense. The project is released at https://github.com/gusye1234/recad.
Ti-DC-GNN: Incorporating Time-Interval Dual Graphs for Recommender Systems

Recommender systems are essential for personalized content delivery and have become increasingly popular recently. However, traditional recommender systems are limited in their ability to capture complex relationships between users and items. Dynamic graph neural networks (DGNNs) have recently emerged as a promising solution for improving recommender systems by incorporating temporal and sequential information in dynamic graphs. In this paper, we propose a novel method, "Ti-DC-GNN" (Time-Interval Dual Causal Graph Neural Networks), based on an intermediate representation of graph evolution as a sequence of time-interval graphs. The main parts of the method are the novel forms of interval graphs: graph of causality and graph of consequence that explicitly preserve inter-relationships between edges (user-items interactions). The local and global message passing are developed based on edge memory to identify short-term and long-term dependencies. Experiments on several well-known datasets show that our method consistently outperforms modern temporal GNNs with node memory alone in dynamic edge prediction tasks.
Towards Self-Explaining Sequence-Aware Recommendation

Self-explaining models are becoming an important perk of recommender systems, as they help users understand the reason behind certain recommendations, which encourages them to interact more often with the platform. In order to personalize recommendations, modern approaches make the model aware of the user behavior history for interest evolution representation. However, existing explainable recommender systems do not consider the past user history to further personalize the explanation based on the user interest fluctuation. In this work, we propose a SEQuence-Aware Explainable Recommendation model (SEQUER) that is able to leverage the sequence of user-item review interactions to generate better explanations while maintaining recommendation performance. Experiments validate the effectiveness of our proposal on multiple recommendation scenarios. Our source code and preprocessed datasets are available at https://github.com/alarca94/sequer-recsys23.
RecQR: Using Recommendation Systems for Query Reformulation to correct unseen errors in spoken dialog systems

As spoken dialog systems like Siri, Alexa and Google Assistant become widespread, it becomes apparent that relying solely on global, one-size-fits-all models of Automatic Speech Recognition (ASR), Natural Language Understanding (NLU) and Entity Resolution (ER), is inadequate for delivering a friction-less customer experience. To address this issue, Query Reformulation (QR) has emerged as a crucial technique for personalizing these systems and reducing customer friction. However, existing QR models, trained on personal rephrases in history face a critical drawback - they are unable to reformulate unseen queries to unseen targets. To alleviate this, we present RecQR, a novel system based on collaborative filters, designed to reformulate unseen defective requests to target requests that a customer may never have requested for in the past. RecQR anticipates a customer’s future requests and rewrites them using state of the art, large-scale, collaborative filtering and query reformulation models. Based on experiments we find that it reduces errors by nearly 40% (relative) on the reformulated utterances.
Reproducibility Analysis of Recommender Systems relying on Visual Features: traps, pitfalls, and countermeasures

Reproducibility is an important requirement for scientific progress, and the lack of reproducibility for a large amount of published research can hinder the progress over the state-of-the-art. This concerns several research areas, and recommender systems are witnessing the same reproducibility crisis. Even solid works published at prestigious venues might not be reproducible for several reasons: data might not be public, source code for recommendation algorithms might not be available or well documented, and evaluation metrics might be computed using parameters not explicitly provided. In addition, recommendation pipelines are becoming increasingly complex due to the use of deep neural architectures or representations for multimodal side information involving text, images, audio, or video. This makes the reproducibility of experiments even more challenging.
Beyond the Sequence: Statistics-Driven Pre-training for Stabilizing Sequential Recommendation Model

The sequential recommendation task aims to predict the item that user is interested in according to his/her historical action sequence. However, inevitable random action, i.e. user randomly accesses an item among multiple candidates or clicks several items at random order, cause the sequence fails to provide stable and high-quality signals. To alleviate the issue, we propose the StatisTics-Driven Pre-traing framework (called STDP briefly). The main idea of the work lies in the exploration of utilizing the statistics information along with the pre-training paradigm to stabilize the optimization of recommendation model. Specifically, we derive two types of statistical information: item co-occurrence across sequence and attribute frequency within the sequence. And we design the following pre-training tasks: 1) The co-occurred items prediction task, which encourages the model to distribute its attention on multiple suitable targets instead of just focusing on the next item that may be unstable. 2) We generate a paired sequence by replacing items with their co-occurred items and enforce its representation close with the original one, thus enhancing the model’s robustness to the random noise. 3) To reduce the impact of random on user’s long-term preferences, we encourage the model to capture sequence-level frequent attributes. The significant improvement over six datasets demonstrates the effectiveness and superiority of the proposal, and further analysis verified the generalization of the STDP framework on other models.
A Short Text Classification Model Based on Chinese Part-of-Speech Information and Mutual Learning

Short text classification is one of the common tasks in natural language processing. Short text contains less information, and there is still much room for improvement in the performance of short text classification models. This paper proposes a new short text classification model ML-BERT based on the idea of mutual learning. ML-BERT includes a BERT that only uses word vector information and a BERT that fuses word information and part-of-speech information and introduces transmission flag to control the information transfer between the two BERTs to simulate the mutual learning process between the two models. Experimental results show that the ML-BERT model obtains a MAF1 score of 93.79% on the THUCNews dataset. Compared with the representative models Text-CNN, Text-RNN and BERT, the MAF1 score improves by 8.11%, 6.69% and 1.69%, respectively.
Distributed Implementation of SM4 Block Cipher Algorithm Based on SPDZ Secure Multi-party Computation Protocol

SM4 is a block cipher algorithm among Chinese commercial cryptographic algorithms, which is advanced in terms of efficiency and theoretical security and has become national and international standards successively. However, existing literature shows that SM4 was not designed with an emphasis on key storage, which means that in today's world where a single trusted hardware device with the built-in key faces challenges such as vulnerability, high cost, and unreliability, the usability of SM4 may be limited. Therefore, this paper proposes an implementation scheme for SM4 based on secure multi-party computation (MPC) technology. The scheme involves dispensing the key among multiple users' devices in a distributed manner, and when using the SM4 algorithm for encryption, multiple users perform joint computation without opening the full key. Specifically, this paper employs the MP-SPDZ framework, which satisfies security requirements in the presence of a dishonest majority of active adversaries. In view of the fact that this framework can only perform basic linear operations such as addition and multiplication, this paper focuses on the algebraic analysis of Sbox, which is the only non-linear component in SM4, and reconstructs it using the bit decomposition method. Furthermore, this paper demonstrates the conversion between the SM4-Sbox field $$GF(2^8)$$ and the SPDZ parameter field $$GF(2^{40})$$ through the isomorphic mapping, making it possible to perform joint calculations throughout the entire SM4 algorithm. Complexity analysis shows that this scheme has advantages in terms of data storage and communication volume, reaching a level of usability.
Research on Gesture Recognition Based on Multialgorithm Fusion

Aiming at people with hearing and speaking obstacles, this paper proposes a multialgorithm fusion for gesture recognition. This paper aims to more clearly distinguish easily confused gestures in gesture recognition and improve gesture recognition accuracy by integrating lip-reading recognition. For gesture recognition, this paper first performs skin color processing and segmentation on the hand area of the collected video sequence and detects the hand feature points by calling the hand key point model. The extracted gesture features are trained and recognized by the support vector machine algorithm. For lip reading recognition, this paper first uses the AdaBoost algorithm to detect and track key points on the collected video sequence, locate the lips, extract the key points of the lips through a convolutional neural network, and input the extracted key point feature sequence into BiLSTM to extract semantic information. The fusion of gesture recognition and lip reading recognition algorithms using the YOLOV5 model can effectively improve the accuracy of gesture recognition. Through experimental verification, the recognition rate can be increased from 89.4% to 94.3%.
A Novel Federated Learning with Bidirectional Adaptive Differential Privacy

With the explosive growth of personal data in the era of big data, federated learning has broader application prospects, in order to solve the problem of data island and preserve user data privacy, a federated learning model based on differential privacy (DP) is proposed. Participants share the parameters after adding noise to the central server for parameter aggregation by training local data. However, there are two problems in this model: on the one hand, the data information in the process of broadcasting parameters by the central server is still compromised, with the risk of user privacy leakage; on the other hand, adding too much noise to parameters will reduce the quality of parameter aggregation and affect the model accuracy of federated learning finally. Therefore, a novel federated learning approach with bidirectional adaptive differential privacy (FedBADP) is proposed, it can adaptively add noise to the gradients transmitted by participants and central server, and protects data security without affecting model accuracy. In addition, considering the performance limitations of the participants' hardware devices, this model samples their gradients to reduce communication overhead, and uses RMSprop to accelerate the convergence of the model on the participants and central server to improve the ac-curacy of the model. Experiments show that our novel model can not only obtain better results in accuracy, but also enhance user privacy preserving while reducing communication overhead.
Two-Dimensional Code Transmission System Based on Side Channel Feedback

Two-dimensional code technology is widely used in all walks of life due to its high reliability, large information capacity and relative security of information. As the carrier of data transmission, two-dimensional code has become a new form of near field communication, and mainly used for the flow of documents. The rapid spread of smart devices with cameras has further broadened the use of two-dimensional code. The combination of smartphones and two-dimensional code eliminates the dependence on special physical isolation devices and reduces transmission costs. In this paper, a two-dimensional code transmission system based on side channel is designed, which transmits confidential information through the visual main channel of two-dimensional code and retransmits feedback information through the acoustic side channel. This paper verifies the impact of QR code capacity on data integrity and improves file transfer integrity.
On Challenges of Evaluating Recommender Systems in an Offline Setting

In the past 20 years, the area of Recommender Systems (RecSys) has gained significant attention from both academia and industry. We are not in short of research papers on various RecSys models or online systems from industry players. However, in terms of model evaluation in offline settings, many researchers simply follow the commonly adopted experiment setup, and have not zoomed into the unique characteristics of the RecSys problem. In this tutorial, I will briefly review the commonly adopted evaluations in RecSys then discuss the challenges of evaluating recommender systems in an offline setting. The main emphasis is the consideration of global timeline in the evaluation, particularly when a dataset covers user-item interactions that have been collected from a long time period.
Improving Transferability Reversible Adversarial Examples Based on Flipping Transformation

Adding subtle perturbations to an image can cause the classification model to misclassify, and such images are called adversarial examples. Adversarial examples threaten the safe use of deep neural networks, but when combined with reversible data hiding (RDH) technology, they can protect images from being correctly identified by unauthorized models and recover the image lossless under authorized models. Based on this, the reversible adversarial example (RAE) is rising. However, existing RAE technology focuses on feasibility, attack success rate and image quality, but ignores transferability and time complexity. In this paper, we optimize the data hiding structure and combine data augmentation technology, which flips the input image in probability to avoid overfitting phenomenon on the dataset. On the premise of maintaining a high success rate of white-box attacks and the image's visual quality, the proposed method improves the transferability of reversible adversarial examples by approximately 16% and reduces the computational cost by approximately 43% compared to the state-of-the-art method. In addition, the appropriate flip probability can be selected for different application scenarios.
Design and Implementation of Speech Generation and Demonstration Research Based on Deep Learning

Aiming at complex and changeable factors such as speech theme and environment, which make it difficult for a speaker to prepare the speech text in a short time, this paper proposes a speech generation and demonstration system based on deep learning. This system is based on the Deep Learning Development Framework (PyTorch), trained through the theory of GPT-2 and the open source pretrained model, to generate multiple speeches according to the topics given by users, and the system generates the final speech and corresponding voice demonstration audio through text modification, speech synthesis and other technologies to help users quickly obtain the target document and audio. Experiments show that the text generated by this model is smooth and easy to use, which helps shorten the preparation time of speakers and improves the confidence of the impromptu speaker. In addition, the paper explores the application prospects of text generation and has certain reference value.
Modal Interactive Feature Encoder for Multimodal Sentiment Analysis

Multimodal Sentiment analysis refers to analyzing emotions in information carriers containing multiple modalities. To better analyze the features within and between modalities and solve the problem of incomplete multimodal feature fusion, this paper proposes a multimodal sentiment analysis model MIF (Modal Interactive Feature Encoder For Multimodal Sentiment Analysis). First, the global features of three modalities are obtained through unimodal feature extraction networks. Second, the inter-modal interactive feature encoder and the intra-modal interactive feature encoder extract similarity features between modalities and intra-modal special features separately. Finally, unimodal special features and the interaction information between modalities are decoded to get the fusion features and predict sentimental polarity results. We conduct extensive experiments on three public multimodal datasets, including one in Chinese and two in English. The results show that the performance of our approach is significantly improved compared with benchmark models.
DP-ASSGD: Differential Privacy Protection Based on Stochastic Gradient Descent Optimization

Recently, differential privacy algorithms based on deep learning have become increasingly mature. Previous studies provide privacy mostly by adding differential privacy noise to the gradient, but it will reduce the accuracy, and it is difficult to balance privacy and accuracy. In this paper, the DP-ASSGD algorithm is proposed to counterpoise privacy and accuracy. The convergence speed is improved, the number of optimized iterations is decreased, and the privacy loss is significantly reduced. On the other hand, by using the postprocessing immunity characteristics of the differential privacy model, the Laplace smoothing mechanism is added to make the training process more stable and the generalization ability stronger. The experiment uses the MNIST dataset, with the same privacy budget, and compared with the existing differential privacy algorithms, the accuracy is improved by 1.8% on average. When achieving the same accuracy, the DP-ASSGD algorithm consumes less privacy budget.
Two-Step Classification Method for Sadness and Fear Facial Expression Classification Using Facial Feature Points and FACS

Although systems have been developed to automatically estimate the evaluation of a work (e.g., comics, movie, etc.) based on the facial expressions of people viewing the work, they are insufficient for estimating the evaluation of moving or horror works, for which sadness and fear are less likely to be expressed in facial expressions and these expressions are thought to lead to evaluation. In this paper, we propose a new method to improve the accuracy of classification of facial expressions of sadness and fear. Facial feature points and Action Units (AUs) are extracted from facial images to set facial features and classify facial images into seven facial expressions: six basic facial expressions (anger, disgust, fear, happiness, sadness, and surprise) and neutral. First, the expressions of anger and sadness, which are easily confused, are merged into a single category, and similarly fear and surprise are merged to convert seven categories into five. Then, two-step classification was performed by reclassifying each of the merged facial expressions into two categories. Furthermore, the importance of each feature was compared, and the feature most suitable for each classification step was used to improve the classification accuracy. The results of a random forest model classification experiment using 490 face images from the Karolinska Directed Emotional Faces (KDEF) dataset showed that two-step classification performed better than one-step classification in the classification of sadness and fear. The accuracy was further improved by carefully selecting features.
Private Matrix Factorization with Public Item Features

We consider the problem of training private recommendation models with access to public item features. Training with Differential Privacy (DP) offers strong privacy guarantees, at the expense of loss in recommendation quality. We show that incorporating public item features during training can help mitigate this loss in quality. We propose a general approach based on collective matrix factorization (CMF), that works by simultaneously factorizing two matrices: the user feedback matrix (representing sensitive data) and an item feature matrix that encodes publicly available (non-sensitive) item information.
Co-occurrence Embedding Enhancement for Long-tail Problem in Multi-Interest Recommendation

Multi-interest recommendation methods extract multiple interest vectors to represent the user comprehensively. Despite their success in the matching stage, previous works overlook the long-tail problem. This results in the model excelling at suggesting head items, while the performance for tail items, which make up more than 70% of all items, remains suboptimal. Hence, enhancing the tail item recommendation capability holds great potential for improving the performance of the multi-interest model.