Reviewerly/exHarmony huggingface dataset:
A Multinode Collaborative Decision-Making Scheme for Privacy Protection in IIoT

Industry 5.0 is more focused on sustainability, people-centredness, and resilience. The Industrial Internet of Things (IIoT) enable real-time monitoring of equipment status through sensors deployed in dangerous environments to reduce the probability of hazards. However, the devices are vulnerable to malicious attackers or can be tampered with during the transmission of information. Security has therefore become a major concern in the IIoT. In this article, we propose a privacy-preserving multinode collaborative decision-making scheme in IIoT to ensure the validity and integrity of data under the IIoT. This collaborative decision-making solution primarily uses voting to enable accurate monitoring of sensing devices. The scheme is designed based on an efficient aggregate signature and gives a formal proof of security based on the computational Diffie–Hellman problem (CDHP). The final performance analysis phase gives a detailed presentation of the computational overhead of each phase of the protocol.
Aspect Ratio-Based Bidirectional Label Encoding for Square-Like Rotation Detection

Rotation object detection is of great importance in remote sensing imagery where the orientation is arbitrary and objects are densely distributed. However, there are several challenges that need to be overcome, such as the angular boundary problem in the regression-based methods and the square-like problem in the classification-based methods. For square-like object rotation detection, classification-based methods [e.g., circular smooth label (CSL)] suffer from inconsistencies between angular coding and evaluation mechanisms due to the variation of aspect ratio. To address the angular inconsistencies of square-like object existing in current classification methods, we design a novel angular encoding mechanism based on aspect ratio. We make optimizations and improvements in the following two aspects: 1) proposing an aspect ratio-based bidirectional coded label (AR-BCL) to replace CSL for angle coding of square-like object, which significantly improves detection accuracy for square-like object and ii) designing a cross-fusion decoupled head (CF-DH) based on angular classification to replace the existing coupled head (CH), which can help extract features that suitable for angular classification. Extensive experiments on DOTA, a large-scale public dataset for aerial images, demonstrate the effectiveness of our method for square-like object detection.
Edge Intelligence-Empowered Immersive Media

Recent years have witnessed many immersive media services and applications, ranging from 360° video streaming to augmented and virtual reality (VR) and the recent metaverse experiences. These new applications usually have common features, including high fidelity, immersive interaction, and open data exchange between people and the environment. As an emerging paradigm, edge computing has become increasingly ready to support these features. We first show that a key to unleashing the power of edge computing for immersive multimedia is handling artificial intelligence models and data. Then, we present a framework that enables joint accuracy- and latency-aware edge intelligence, with adaptive deep learning model deployment and data streaming. We show that not only conventional mechanisms such as content placement and rate adaptation but also the emerging 360° and VR streaming can benefit from such edge intelligence.
InterGrad: Energy-Efficient Training of Convolutional Neural Networks via Interleaved Gradient Scheduling

This paper addresses the design of accelerators using systolic architectures to train convolutional neural networks using a novel gradient interleaving approach. Training the neural network involves computation and backpropagation of gradients of error with respect to the activation functions and weights. It is shown that the gradient with respect to the activation function can be computed using a weight-stationary systolic array, while the gradient with respect to the weights can be computed using an output-stationary systolic array. The novelty of the proposed approach lies in interleaving the computations of these two gradients on the same configurable systolic array. This results in the reuse of the variables from one computation to the other and eliminates unnecessary memory accesses and energy consumption associated with these memory accesses. The proposed approach leads to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.4 - 2.2 \times $ </tex-math></inline-formula> savings in terms of the number of cycles and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.9 \times $ </tex-math></inline-formula> savings in terms of memory accesses in the fully-connected layer. Furthermore, the proposed method uses up to 25% fewer cycles and memory accesses, and 16% less energy than baseline implementations for state-of-the-art CNNs. Under iso-area comparisons, for Inception-v4, compared to weight-stationary (WS), Intergrad achieves 12% savings in energy, 17% savings in memory, and 4% savings in cycles. Savings for Densenet-264 are 18%, 26%, and 27% with respect to energy, memory, and cycles, respectively. Thus, the proposed novel accelerator architecture reduces the latency and energy consumption for training deep neural networks.
Verifiable Privacy-Enhanced Rotation Invariant LBP Feature Extraction in Fog Computing

Rotation invariant local binary pattern (RI-LBP) features have been applied in diverse scenarios with the advantages of gray-scale and rotation invariance. Secure fog computing has become an emerging paradigm for enterprises or individuals with a huge volume of private data, but limited computing power for feature extraction. Prior secure outsourcing protocols based on LBP and RI-LBP simply focus on local data privacy, which can only resist ciphertext-only attack, and also make extracted features exposed to the cloud. This work focuses on how to effectively ensure data confidentiality and feature integrity. We propose a verifiable privacy-enhanced protocol for RI-LBP feature extraction (VRLBP) based on the fog computing paradigm, which mitigates the aforementioned challenges by involving the proposed symmetric cryptographic scheme where local data and extracted features are proven secure against chosen plaintext attack. Meanwhile, the stage of verification can check the correctness of outsourced features with an overwhelming probability and constant computational complexity. The security analysis and computational costs demonstrate that VRLBP can reduce the computation overhead to around 30% of original feature extraction in a privacy-preserving manner. To exhibit the practical utility, VRLBP is implemented for deepfake detection on five public datasets. Extensive evaluations indicate that VRLBP achieves almost the same accuracy as the original RI-LBP algorithm and outperforms the state-of-the-art protocols.
SoCraft: Advertiser-level Predictive Scoring for Creative Performance on Meta

In this technical demonstration, we present SoCraft, a framework to build an advertiser-level multimedia ad content scoring platform for Meta Ads. The system utilizes a multimodal deep neural architecture to score and evaluate advertised content on Meta using both high- and low-level features of its contextual data such as text, image, targeting, and ad settings. In this demo, we present two deep models, SoDeep and SoWide, and validate the effectiveness of SoCraft with a successful real-world case study in Singapore.
Heterogeneous Graph-based Context-aware Document Ranking

Users' complex information needs usually require consecutive queries, which results in sessions with a series of interactions. Exploiting such contextual interactions has been proven to be favorable for result ranking. However, existing studies mainly model the contextual information independently and sequentially. They neglect the diverse information hidden in different relations and structured information of session elements as well as the valuable signals from other relevant sessions. In this paper, we propose HEXA, a heterogeneous graph-based context-aware document ranking framework. It exploits heterogeneous graphs to organize the contextual information and beneficial search logs for modeling user intents and ranking results. Specifically, we construct two heterogeneous graphs, i.e., a session graph and a query graph. The session graph is built from the current session queries and documents. Meanwhile, we sample the current query's k-layer neighbors from search logs to construct the query graph. Then, we employ heterogeneous graph neural networks and specialized readout functions on the two graphs to capture the user intents from local and global aspects. Finally, the document ranking scores are measured by how well the documents are matched with the two user intents. Results on two large-scale datasets confirm the effectiveness of our model.
CoQEx: Entity Counts Explained

For open-domain question answering, queries on entity counts, such ashow many languages are spoken in Indonesia, are challenging. Such queries can be answered through succinct contexts with counts:estimated 700 languages, and instances:Javanese and Sundanese. Answer candidates naturally give rise to a distribution, where count contexts denoting the queried entity counts and their semantic subgroups often coexist, while the instances ground the counts in their constituting entities. In this demo we showcase the CoQEx methodology (<u>Co</u>unt <u>Q</u>ueries <u>Ex</u>plained) [5,6], which aggregates and structures explanatory evidence across search snippets, for answering user queries related to entity counts [4]. Given a entity count query, our system CoQEx retrieves search-snippets and provides the user with a distribution-aware prediction prediction, categorizes the count contexts into semantic groups and ranks instances grounding the counts, all in real-time. Our demo can be accessed athttps://nlcounqer.mpi-inf.mpg.de/.
Concept-Oriented Transformers for Visual Sentiment Analysis

In the richly multimedia Web, detecting sentiment signals expressed in images would support multiple applications, e.g., measuring customer satisfaction from online reviews, analyzing trends and opinions from social media. Given an image, visual sentiment analysis aims at recognizing positive or negative sentiment, and occasionally neutral sentiment as well. A nascent yet promising direction is Transformer-based models applied to image data, whereby Vision Transformer (ViT) establishes remarkable performance on large-scale vision benchmarks. In addition to investigating the fitness of ViT for visual sentiment analysis, we further incorporate concept orientation into the self-attention mechanism, which is the core component of Transformer. The proposed model captures the relationships between image features and specific concepts. We conduct extensive experiments on Visual Sentiment Ontology (VSO) and Yelp.com online review datasets, showing that not only does the proposed model significantly improve upon the base model ViT in detecting visual sentiment but it also outperforms previous visual sentiment analysis models with narrowly-defined orientations. Additional analyses yield insightful results and better understanding of the concept-oriented self-attention mechanism.
Adversarial Autoencoder for Unsupervised Time Series Anomaly Detection and Interpretation

In many complex systems, devices are typically monitored and generating massive multivariate time series. However, due to the complex patterns and little useful labeled data, it is a great challenge to detect anomalies from these time series data. Existing methods either rely on less regularizations, or require a large number of labeled data, leading to poor accuracy in anomaly detection. To overcome the limitations, in this paper, we propose an adversarial autoencoder anomaly detection and interpretation framework named DAEMON, which performs robustly for various datasets. The key idea is to use two discriminators to adversarially train an autoencoder to learn the normal pattern of multivariate time series, and thereafter use the reconstruction error to detect anomalies. The robustness of DAEMON is guaranteed by the regularization of hidden variables and reconstructed data using the adversarial generation method. An unsupervised approach used to detect anomalies is proposed. Moreover, in order to help operators better diagnose anomalies, DAEMON provides anomaly interpretation by computing the gradients of anomalous data. An extensive empirical study on real data offers evidence that the framework is capable of outperforming state-of-the-art methods in terms of the overall F1-score and interpretation accuracy for time series anomaly detection.
ElasticDL: A Kubernetes-native Deep Learning Framework with Fault-tolerance and Elastic Scheduling

The power of artificial intelligence (AI) models originates with sophisticated model architecture as well as the sheer size of the model. These large-scale AI models impose new and challenging system requirements regarding scalability, reliability, and flexibility. One of the most promising solutions in the industry is to train these large-scale models on distributed deep-learning frameworks. With the power of all distributed computations, it is desired to achieve a training process with excellent scalability, elastic scheduling (flexibility), and fault tolerance (reliability). In this paper, we demonstrate the scalability, flexibility, and reliability of our open-source Elastic Deep Learning (ElasticDL) framework. Our ElasticDL utilizes an open-source system, i.e., Kubernetes, for automating deployment, scaling, and management of containerized application features to provide fault tolerance and support elastic scheduling for DL tasks.
Visual Matching is Enough for Scene Text Retrieval

Given a text query, the task of scene text retrieval aims at searching and localizing all the text instances that are contained in an image gallery. The state-of-the-art method learns a cross-modal similarity between the query text and the detected text regions in natural images to facilitate retrieval. However, this cross-modal approach still cannot well bridge the heterogeneity gap between the text and image modalities. In this paper, we propose a new paradigm that converts the task into a single-modality retrieval problem. Unlike previous works that rely on character recognition or embedding, we directly leverage pictorial information by rendering query text into images to learn the glyph feature of each character, which can be utilized to capture the similarity between query and scene text images. With the extracted visual features, we devise a synthetic label image guided feature alignment mechanism that is robust to different scene text styles and layouts. The modules of glyph feature learning, text instance detection, and visual matching are jointly trained in an end-to-end framework. Experimental results show that our proposed paradigm achieves the best performance in multiple benchmark datasets. As a side product, our method can also be easily generalized to support text queries with unseen characters or languages in a zero-shot manner.
"Just To See You Smile": SMILEY, a Voice-Guided &lt;strike&gt;GUY&lt;/strike&gt; GAN

In this technical demonstration, we present SMILEY, a voice-guided virtual assistant. The system utilizes a deep neural architecture ContraCLIP to manipulate facial attributes using voice instructions, allowing for deeper speaker engagement and smoother customer experience when being used in the "virtual concierge" scenario. We validate the effectiveness of SMILEY and ContraCLIP via a successful real-world case study in Singapore and a large-scale quantitative evaluation.
Self-supervised Multi-view Disentanglement for Expansion of Visual Collections

Image search engines enable the retrieval of images relevant to a query image. In this work, we consider the setting where a query for similar images is derived from a collection of images. For visual search, the similarity measurements may be made along multiple axes, or views, such as style and color. We assume access to a set of feature extractors, each of which computes representations for a specific view. Our objective is to design a retrieval algorithm that effectively combines similarities computed over representations from multiple views. To this end, we propose a self-supervised learning method for extracting disentangled view-specific representations for images such that the inter-view overlap is minimized. We show how this allows us to compute the intent of a collection as a distribution over views. Finally, we show how effective retrieval can be performed by prioritizing candidate expansion images that match the intent of a query collection.
Long-Document Cross-Lingual Summarization

Cross-Lingual Summarization (CLS) aims at generating summaries in one language for the given documents in another language. CLS has attracted wide research attention due to its practical significance in the multi-lingual world. Though great contributions have been made, existing CLS works typically focus on short documents, such as news and guides. Different from these short texts, long documents such as academic articles usually discuss complicated subjects and consist of thousands of words, making them non-trivial to process and summarize. To promote CLS research on long documents, we construct Perseus, the first long-document CLS dataset which collects about 94K Chinese scientific documents paired with English summaries. The average length of documents in Perseus is more than 2000 tokens. As a preliminary study on long-document CLS, we build and evaluate various CLS baselines, including pipeline and end-to-end methods. Experimental results on Perseus show the superiority of the end-to-end baseline, which performs the best among all methods. Furthermore, to provide a deeper understanding, we manually analyze the model outputs and discuss specific challenges faced by current approaches. We hope that our work could benchmark long-document CLS and benefit future studies.
Graphs: Privacy and Generation through ML

Graphs are ubiquitous, which makes machine learning on graphs an important research area. While there are many aspects to this field, our research is focused primarily on two aspects of it. The first research question concerns privacy in graphs, where our work primarily focuses on preserving structural privacy in graphs. The second research question is about generating graphs. With applications in various fields such as drug discovery, designing novel proteins, etc., graph generation is emerging as an essential problem. This paper briefly describes the problems and the methodology to address them.
Multimodal Pre-Training with Self-Distillation for Product Understanding in E-Commerce

Product understanding refers to a series of product-centric tasks, such as classification, alignment and attribute values prediction, which requires fine-grained fusion of various modalities of products. Excellent product modeling ability will enhance the user experience and benefit search and recommendation systems. In this paper, we propose MBSD, a pre-trained vision-and-language model which can integrate the heterogeneous information of product in a single stream BERT-style architecture. Compared with current approaches, MBSD uses a lightweight convolutional neural network instead of a heavy feature extractor for image encoding, which has lower latency. Besides, we cleverly utilize user behavior data to design a two-stage pre-training task to understand products from different perspectives. In addition, there is an underlying imbalanced problem in multimodal pre-training, which will impairs downstream tasks. To this end, we propose a novel self-distillation strategy to transfer the knowledge in dominated modality to weaker modality, so that each modality can be fully tapped during pre-training. Experimental results on several product understanding tasks demonstrate that the performance of MBSD outperforms the competitive baselines.
Enhanced Multiset Consensus Protocol Based on PBFT for Logistics Information Traceability

In the recent years, the global logistics industry has greatly driven the development of the world economy. At the same time, a large amount of data information is generated. Due to the frequent occurrence of logistics information leakage and forgery, it is necessary to find solutions that can accurately trace logistics information and ensure the security and authenticity of logistics information. The birth of blockchain technology has transformed the logistics industry from quantitative change to qualitative change. The technical characteristics of blockchain technology, such as distributed storage ideas, decentralization, immutability, and complex encryption consensus algorithm, endow it with a wide range of application prospects in the logistics industry. This paper proposes an enhanced multiset consensus algorithm based on PBFT (practical Byzantine fault tolerance) for logistics information traceability and storage on the logistics blockchain. The application of the proposed multi-set consensus algorithm in the topology structure composed of multiple sets can improve the consensus efficiency of logistics information in the blockchain. We improve consensus capability and transaction speed, avoid redundant consensus message packets occupying a large bandwidth, and efficiently process logistics information generated at any time. We ensure the traceability of logistics information and achieve efficient and accurate traceability, and the efficiency and security of the proposed algorithm are analyzed. This paper aims to solve the problems of traceability, trustworthiness, and efficient processing of blockchain applications in logistics information to operate the logistics network efficiently. This paper compares the proposed algorithm with the PBFT-related expansion algorithm regarding bandwidth occupation, delay, and throughput. The results show that the MPBFT consensus algorithm significantly improves the efficiency of the logistics blockchain network.