Reviewerly/exHarmony huggingface dataset:
Bit-aware Semantic Transformer Hashing for Multi-modal Retrieval

Multi-modal hashing learns binary hash codes with extremely low storage cost and high retrieval speed. It can support efficient multi-modal retrieval well. However, most existing methods still suffer from three important problems: 1) Limited semantic representation capability with shallow learning. 2) Mandatory feature-level multi-modal fusion ignores heterogeneous multi-modal semantic gaps. 3) Direct coarse pairwise semantic preserving cannot effectively capture the fine-grained semantic correlations. For solving these problems, in this paper, we propose a Bit-aware Semantic Transformer Hashing (BSTH) framework to excavate bit-wise semantic concepts and simultaneously align the heterogeneous modalities for multi-modal hash learning on the concept-level. Specifically, the bit-wise implicit semantic concepts are learned with the transformer in a self-attention manner, which can achieve implicit semantic alignment on the fine-grained concept-level and reduce the heterogeneous modality gaps. Then, the concept-level multi-modal fusion is performed to enhance the semantic representation capability of each implicit concept and the fused concept representations are further encoded to the corresponding hash bits via bit-wise hash functions. Further, to supervise the bit-aware transformer module, a label prototype learning module is developed to learn prototype embeddings for all categories that capture the explicit semantic correlations on the category-level by considering the co-occurrence priors. Experiments on three widely tested multi-modal retrieval datasets demonstrate the superiority of the proposed method from various aspects.
Does the User Have A Theory of the Recommender? A Grounded Theory Study

Recommender systems have gained widespread adoption in many web applications. Modern internet users experience daily interactions with recommender systems. Consequently, users, through these interactive experiences, have developed an inherent understanding of how recommender systems work, what their objectives are, and how the user might manipulate them. We describe this understanding as the Theory of the Recommender. In this paper, we explore the users' perception and understanding of the recommender system in an empirical study using a grounded theory methodology. To that end, we draw on the cognitive theory of mind to propose a comprehensive theoretical framework that explains the users' interpretation of the recommender system's knowledge, reasoning, motivation, beliefs and attitudes. Our findings, based on individual in-depth interviews, suggest that users possess a sophisticated understanding of the recommender system's behavior. Identifying the user's understanding is a necessary step in evaluating their impact and improving recommender systems accordingly. Finally, we discuss the potential implications of such user knowledge on recommendation performance.
Application of improved virtual sample and sparse representation in face recognition

Abstract Sparse representation plays an important role in the research of face recognition. As a deformable sample classification task, face recognition is often used to test the performance of classification algorithms. In face recognition, differences in expression, angle, posture, and lighting conditions have become key factors that affect recognition accuracy. Essentially, there may be significant differences between different image samples of the same face, which makes image classification very difficult. Therefore, how to build a robust virtual image representation becomes a vital issue. To solve the above problems, this paper proposes a novel image classification algorithm. First, to better retain the global features and contour information of the original sample, the algorithm uses an improved non‐linear image representation method to highlight the low‐intensity and high‐intensity pixels of the original training sample, thus generating a virtual sample. Second, by the principle of sparse representation, the linear expression coefficients of the original sample and the virtual sample can be calculated, respectively. After obtaining these two types of coefficients, calculate the distances between the original sample and the test sample and the distance between the virtual sample and the test sample. These two distances are converted into distance scores. Finally, a simple and effective weight fusion scheme is adopted to fuse the classification scores of the original image and the virtual image. The fused score will determine the final classification result. The experimental results show that the proposed method outperforms other typical sparse representation classification methods.
Cross-Probe BERT for Fast Cross-Modal Search

Owing to the effectiveness of cross-modal attentions, text-vision BERT models have achieved excellent performance in text-image retrieval. Nevertheless, cross-modal attentions in text-vision BERT models require expensive computation cost when tackling text-vision retrieval due to their pairwise input. Therefore, normally, it is impractical for deploying them for large-scale cross-modal retrieval in real applications. To address the inefficiency issue in exiting text-vision BERT models, in this work, we develop a novel architecture, cross-probe BERT. It devises a small number of text and vision probes, and the cross-modal attentions are efficiency achieved through the interactions between text and vision probes. It takes lightweight computation cost, and meanwhile effectively exploits cross-modal attention. Systematic experiments on public benchmarks demonstrate the excellent effectiveness and efficiency of our cross-probe BERT.
Knowledge Graph Question Answering Datasets and Their Generalizability

Existing approaches on Question Answering over Knowledge Graphs (KGQA) have weak generalizability. That is often due to the standard i.i.d. assumption on the underlying dataset. Recently, three levels of generalization for KGQA were defined, namely i.i.d., compositional, zero-shot. We analyze 25 well-known KGQA datasets for 5 different Knowledge Graphs (KGs). We show that according to this definition many existing and online available KGQA datasets are either not suited to train a generalizable KGQA system or that the datasets are based on discontinued and out-dated KGs. Generating new datasets is a costly process and, thus, is not an alternative to smaller research groups and companies. In this work, we propose a mitigation method for re-splitting available KGQA datasets to enable their applicability to evaluate generalization, without any cost and manual effort. We test our hypothesis on three KGQA datasets, i.e., LC-QuAD, LC-QuAD 2.0 and QALD-9). Experiments on re-splitted KGQA datasets demonstrate its effectiveness towards generalizability. The code and a unified way to access 18 available datasets is online at https://github.com/semantic-systems/KGQA-datasets as well as https://github.com/semantic-systems/KGQA-datasets-generalization.
Continual Learning Dialogue Systems - Learning during Conversation

Dialogue systems, commonly known as Chatbots, have gained escalating popularity in recent years due to their wide-spread applications in carrying out chit-chat conversations with users and accomplishing various tasks as personal assistants. However, they still have some major weaknesses. One key weakness is that they are typically trained from pre-collected and manually-labeled data and/or written with handcrafted rules. Their knowledge bases (KBs) are also fixed and pre-compiled by human experts. Due to the huge amount of manual effort involved, they are difficult to scale and also tend to produce many errors ought to their limited ability to understand natural language and the limited knowledge in their KBs. Thus, when these systems are deployed, the level of user satisfactory is often low.
Wild Networks: Exposure of 5G Network Infrastructures to Adversarial Examples

Fifth Generation (5G) networks must support billions of heterogeneous devices while guaranteeing optimal Quality of Service (QoS). Such requirements are impossible to meet with human effort alone, and Machine Learning (ML) represents a core asset in 5G. ML, however, is known to be vulnerable to adversarial examples; moreover, as our paper will show, the 5G context is exposed to a yet another type of adversarial ML attacks that cannot be formalized with existing threat models. Proactive assessment of such risks is also challenging due to the lack of ML-powered 5G equipment available for adversarial ML research. To tackle these problems, we propose a novel adversarial ML threat model that is particularly suited to 5G scenarios, and is agnostic to the precise function solved by ML. In contrast to existing ML threat models, our attacks do not require any compromise of the target 5G system while still being viable due to the QoS guarantees and the open nature of 5G networks. Furthermore, we propose an original framework for realistic ML security assessments based on public data. We proactively evaluate our threat model on 6 applications of ML envisioned in 5G. Our attacks affect both the training and the inference stages, can degrade the performance of state-of-the-art ML systems, and have a lower entry barrier than previous attacks.
A Novel and Effective Cooperative RANSAC Image Matching Method Using Geometry Histogram-Based Constructed Reduced Correspondence Set

The success of many computer vision and pattern recognition applications depends on matching local features on two or more images. Because the initial correspondence set—i.e., the set of the initial feature pairs—is often contaminated by mismatches, removing mismatches is a necessary task prior to image matching. In this paper, we first propose a fast geometry histogram-based (GH-based) mismatch removal strategy to construct a reduced correspondence set Creduced,GH from the initial correspondence set Cini. Next, we propose an effective cooperative random sample consensus (COOSAC) method for remote sensing image matching. COOSAC consists of a RANSAC, called RANSACini working on Cini, and a tiny RANSAC, called RANSACtiny,GH working on a randomly selected subset of Creduced,GH. In RANSACtiny,GH, an iterative area constraint-based sampling strategy is proposed to estimate the model solution of Ctiny,GH until the specified confidence level is reached, and then RANSACini utilizes the estimated model solution of Ctiny,GH to calculate the inlier rate of Cini. COOSAC repeats the above cooperation between RANSACtiny,GH and RANSACini until the specified confidence level is reached, reporting the resultant model solution of Cini. For convenience, our image matching method is called the GH-COOSAC method. Based on several testing datasets, thorough experimental results demonstrate that the proposed GH-COOSAC method achieves lower computational cost and higher matching accuracy benefits when compared with the state-of-the-art image matching methods.
Learned Token Pruning in Contextualized Late Interaction over BERT (ColBERT)

BERT-based rankers have been shown very effective as rerankers in information retrieval tasks. In order to extend these models to full-ranking scenarios, the ColBERT model has been recently proposed, which adopts a late interaction mechanism. This mechanism allows for the representation of documents to be precomputed in advance. However, the late-interaction mechanism leads to large index size, as one needs to save a representation for each token of every document. In this work, we focus on token pruning techniques in order to mitigate this problem. We test four methods, ranging from simpler ones to the use of a single layer of attention mechanism to select the tokens to keep at indexing time. Our experiments show that for the MS MARCO-passages collection, indexes can be pruned up to 70% of their original size, without a significant drop in performance. We also evaluate on the MS MARCO-documents collection and the BEIR benchmark, which reveals some challenges for the proposed mechanism.
Wikimarks

We provide a resource for automatically harvesting relevance benchmarks from Wikipedia -- which we refer to as "Wikimarks" to differentiate them from manually created benchmarks. Unlike simulated benchmarks, they are based on manual annotations of Wikipedia authors. Studies on the TREC Complex Answer Retrieval track demonstrated that leaderboards under Wikimarks and manually annotated benchmarks are very similar. Because of their availability, Wikimarks can fill an important need for Information Retrieval research.
A new method for task scheduling in fog‐based medical healthcare systems using a hybrid nature‐inspired algorithm

Summary The goal of the healthcare system is to offer a dependable and well‐organized solution for improving human's wellbeing. Examining a patient's history can assist clinicians in considering the patient's wants when building a healthcare system and providing service, resulting in increased patient satisfaction. Thus, healthcare is becoming a more competitive sector. Massive data volume, latency, response time, and security susceptibility are all difficulties resulting from this substantial increase in healthcare systems. As a famous distributed structure, fog computing might thus aid in the resolution of such problems. Processing parts are situated among end devices and cloud components in a fog computing infrastructure and run programs. This design is well suited to real‐time and low‐latency applications, like healthcare systems. Because task scheduling is an NP‐hard optimization issue in fog‐based medical healthcare systems, this work proposes a hybrid genetic algorithm and particle swarm optimization (GA‐PSO) strategy. A powerful single‐objective optimization technique is the GA‐PSO. Individuals in a novel generation are formed in GA‐PSO through mutation and crossover operations in GA‐PSO, which uses a redefined local optimization swarm. Hence, it may avoid local minimums and perform well in global searches. The study's goal in fog‐based medical healthcare systems is to lower the makespan and overall response time. The suggested technique is simulated in MATLAB and compared to the GA and PSO methods. The empirical findings confirmed the improved makespan, making the approach appropriate for medical and real‐time systems applications.
Cyber Fraud Economics, Scam Types, and Potential Measures to Protect U.S. Seniors: A Short Review

Cyber fraud has become increasingly common as it can be easily carried out with relative ease through multiple mediums. Particularly, the elderly population aged 60 and above seniors, are more vulnerable to such fraud/scams as they generally lack the know-how for such fraudulent activities. This paper briefly addresses the various types of fraud/scams, apparent early warning signs, and potential preventive tips before falling victim of cyber fraud. Additionally, sophisticated scam methods are also highlighted, and resources available to report/inform the general public is documented.
Perturbation-insensitive cross-domain image enhancement for low-quality face verification

Low-quality face images present two typical problems: perceptual distortion and a significant reduction in verification accuracy. Prevailing works address these two problems by face image enhancement. Generally, they require low- and high-quality image pairs or facial priors, e.g., facial landmarks, which are relatively rare in real-world scenarios. In this paper, we introduce a fresh perspective to address these problems, by hypothesizing that there is a positive correlation between cross-domain image enhancement and adversarial defense. With the incorporation of cross-domain image enhancement, paired images or facial priors are no longer necessary. Furthermore, we introduce two types of adversarial perturbations, namely appearance and semantic perturbations, and defending these perturbations aims to solve the aforementioned two typical problems. Defending appearance perturbations decreases perceptual distortion and improve image quality, while defending semantic perturbations promotes identity preservation during the image enhancement process, which improves verification accuracy. To this end, we propose a collaborative face enhancement module (COFEM) for face verification based on two types of adversarial perturbation examples. COFEM incorporates three components. First, an adversarial example generator attacks high-quality images (source domain) in two different ways to obtain appearance and semantic perturbation examples. Next, an image enhancement network denoises these perturbation examples and enhances the quality of low-quality images (target domain). Then, an image reconstruction network is utilized to preserve the identity of the enhanced image such that it is consistent with that of the corresponding input. Unlike prevailing image enhancement models which mainly focus on high perceptual quality, COFEM emphasizes identity-related feature preservation, which is vital to face verification. Combined with COFEM, we also design a face verification module to form a low-quality face verification approach. Extensive experiments demonstrate the effectiveness of our approach in improving the low-quality face image quality and verification accuracy.
Lexical ambiguity detection in professional discourse

Professional discourse is the language used by specialists, such as lawyers, doctors and academics, to communicate the knowledge and assumptions associated with their respective fields. Professional discourse can be especially difficult for non-specialists to understand due to the lexical ambiguity of commonplace words that have a different or more specific meaning within a specialist domain. This phenomena also makes it harder for specialists to communicate with the general public because they are similarly unaware of the potential for misunderstandings. In this article, we present an approach for detecting domain terms with lexical ambiguity versus everyday English. We demonstrate the efficacy of our approach with three case studies in statistics, law and biomedicine. In all case studies, we identify domain terms with a [email protected] greater than 0.9, outperforming the best performing baseline by 18.1–91.7%. Most importantly, we show this ranking is broadly consistent with semantic differences. Our results highlight the difficulties that existing semantic difference methods have in the cross-domain setting, which rank non-domain terms highly due to noise or biases in the data. We additionally show that our approach generalizes to short phrases and investigate its data efficiency by varying the number of labeled examples.
An Eyes-Based Siamese Neural Network for the Detection of GAN-Generated Face Images

Generative Adversarial Network (GAN) models are nowadays able to generate synthetic images which are visually indistinguishable from the real ones, thus raising serious concerns about the spread of fake news and the need to develop tools to distinguish fake and real images in order to preserve the trustworthiness of digital images. The most powerful current detection methods are based on Deep Learning (DL) technology. While these methods get excellent performance when tested under conditions similar to those considered for training, they often suffer from a lack of robustness and generalization ability, as they fail to detect fake images that are generated by “unseen” GAN models. A possibility to overcome this problem is to develop tools that rely on the semantic attributes of the image. In this paper, we propose a semantic-based method for distinguishing GAN-generated from real faces, that relies on the analysis of inter-eye symmetries and inconsistencies. The method resorts to the superior capabilities of similarity learning of extracting representative and robust features. More specifically, a Siamese Neural Network (SNN) is utilized to extract high-level features characterizing the inter-eye similarity, that can be used to discriminate between real and synthetic pairs of eyes. We carried out extensive experiments to assess the performance of the proposed method in both matched and mismatched conditions pertaining to the GAN type used to generate the synthetic images and the robustness of the method in presence of post-processing. The results we got are comparable, and in some cases superior, to those achieved by the best performing state-of-the-art method leveraging on the analysis of the entire face image.
Validation and calibration of models to estimate photosynthetically active radiation considering different time scales and sky conditions

Photosynthetically Active Radiation (PAR) is a fundamental parameter for developing plant productivity models. Nevertheless, instrumentation for measuring PAR and to record it is scarce at conventional meteorological stations. Several procedures have therefore been proposed for PAR estimation. In this work, 21 previously published analytical models that correlate PAR with easily available meteorological parameters are collected. Although longer time scales were considered in the original publications, a minute range was applied in this work to calibrate the PAR models. In total, more than 10 million input records were gathered from the SURFRAD station network from a 10-year long time series with data frequencies recorded every 1 min. The models were calibrated both globally, using data from all stations and locally, with data from each station. After calibration, the models were validated for minute, hourly and daily data, obtaining low fitting errors at the different stations in all cases, both when using the globally calibrated models and with the models calibrated for each location. Although the PAR results in general improved for locally calibrated models, the use of local models is not justified, since the global models presented offered very satisfactory PAR results for the different climatic conditions where the meteorological stations are located. Thus, PAR estimation model should then be selected, solely considering the meteorological variables available at the specific location. When applying the globally calibrated models to input data classified according to sky conditions (from clear to overcast), the PAR models continued to perform satisfactorily, although the error statistics of some models for overcast skies worsened.
Deepfake noise investigation and detection

The fast development of Deepfake has brought huge current and potential future negative impacts to our daily lives. As the circulating popular Deepfake videos have become difficult to be distinguished by human eyes, various Deepfake detection approaches have been attempted using deep learning models. However, even though some existing detection methods have achieved reasonable detection performance with respect to the statistical evaluation metrics, the actual underlying Deepfake forensic traces have been barely discussed. In this study, we investigate the special forensic noise traces within Deepfake image frames and propose a noise-based Deepfake detection model approach using a deep neural network. We train a Siamese noise extractor using a novel face-background strategy to investigate different forensic noise traces of a synthesized face area and an unmodified background area. A similarity matrix module is proposed to analyze the forensic noise trace difference between a cropped face square and a cropped background square from a candidate image frame for the Deepfake detection task. As a result, our proposed model trained on the high-quality Celeb-DF dataset has achieved state-of-the-art performance with 99.15% accuracy and 99.92% AUC score on the in-dataset testing set and 88.95% AUC score on the highly difficult unknown-attack Deepfake video dataset. Furthermore, the visualization of the Deepfake forensic noise traces has shown the explicit distinction between synthesized faces and any unmodified area. We believe that the visualized evidence could provide better proof of Deepfake detection results rather than simply the statistical evaluation numbers.