Reviewerly/exHarmony huggingface dataset:
Subjective Visual Vertical test with the 3D virtual reality system: effective factors and cybersickness

Background 3D Virtual Reality (VR) offers new opportunities in vestibular science. It also presents new challenges and problems.Aims/objectives The study aimed to evaluate the effective factors in the 3D VR Subjective Visual Vertical (SVV) test and the impact of cybersickness on the test results.Material and methods The effect of the foam surface, head position in the yaw axis, moving background, and arm position holding the controller was tested. Cybersickness was evaluated using the Simulator Sickness Questionnaire (SSQ).Results The head position and controller holding style significantly affected the results. The foam surface and the moving background did not have a significant effect. Although 61.4% of the patients fell into the bad category according to the symptoms of the SSQ score, cybersickness did not significantly affect the SVV results.Conclusions and significance In 3D VR SVV, additional factors should be considered: the headset’s weight, head position, and how we hold the controller. The A-effect emerged when the head was 45 degrees turned on the yaw axis. A significant shift was detected in the test, with the arm holding the controller at 90 degrees. Most subjects felt cybersickness at a considerable level. Cybersickness should always be taken into account in VR when planning new applications.
GraphNEI: A GNN-based network entity identification method for IP geolocation

Network entity geolocation technology is a technique for inferring geographic location through features such as network measurements or IP address searchable information, also known as IP geolocation. By obtaining the device type of the target can assist in target IP geolocation or IP landmark mining. Most traditional rule-based or learning-based methods identify network entities. However, due to the existence of firewalls and the limitations of feature detection, the features of individual targets are prone to be missing or spoofed, which can lead to a decrease in the accuracy of network entity identification (NEI). This paper propose a graph neural network-based network entity identification method, GraphNEI model, which embeds network entities into the graph structure and uses graph neural networks to improve the current problem of missing or spoofed features in order to improve the accuracy of current network entity identification. It mainly includes five steps: data processing, subgraph partition, weight calculation, node update and classification. First, the acquired dataset is subjected to feature extraction and anonymization; second, the target nodes are subjected to network topology graph construction and community division; third, the nodes’ self-attention, structural attention and similarity of neighbors are combined and used to calculate the nodes’ combined attention; fourth, node aggregation and update, and update the node representation based on the combined attention results; finally, the nodes are classified. We successfully identified 7 different network entities on the publicly collected dataset, and the identification accuracy of network entities is above 95.49%, improved 0.51%–10.42% compared to typical rule-based or learning-based methods. It effectively improves the effective NEI in the absence of network entity features. In addition, we conduct IP geolocation research based on NEI, and the experimental results show that the method is effective in reducing IP geolocation errors distance.
Vehicle Damage Severity Estimation for Insurance Operations Using In-The-Wild Mobile Images

Following a car accident, an insurance company must assess the level of damage to each vehicle to decide on the compensation paid to the insurance customer.This assessment is usually performed by manual inspection, which is costly and time-consuming.Automatic car damage assessment using image data is an under-addressed problem highly relevant to the insurance industry.Although there have been many attempts at solving particular aspects of this problem, we are unaware of any complete solutions available.In this work, we propose a pipeline that uses photographs of a damaged car, collected by the users from multiple angles, together with structured data about the vehicle, to estimate damage severity following an accident.Our proposed pipeline consists of several computer-vision models for the detection of damage and the determination of its extent.Unlike existing approaches in car damage assessment, we use semantic segmentation to understand which parts of the car are damaged, and to what extent.We then extract computer-vision features, indicating the location and severity of damage to each exterior panel, together with structured data, to arrive at an accurate damage cost estimation.We train and evaluate this model on a large dataset of historical insurance claims with known outcomes, all captured in the wild with mobile-phone hardware.
MABSA: A curated Malayalam aspect based sentiment analysis dataset on movie reviews

Regional languages are being used more frequently in online platforms as a result of the expanding use of digital technology. Understanding user opinions on social media platforms, forums, blogs, and other digital platforms that employ Indian regional languages has become significant due to their role in various applications. Research on sentiment analysis of Indian regional language texts suffers due to the unavailability of available regional language datasets. The curated Malayalam Aspect Based Sentiment Analysis (MABSA) dataset is a labeled dataset for Aspect Based Sentiment Analysis (ABSA) on the Indian regional language Malayalam over the movie review domain. Malayalam movie reviews, an excellent source of text data for ABSA, are collected from an online survey using Google form and manually collecting reviews from three social media platforms: IMDb, Facebook, and YouTube. Nine target aspects were identified, and three annotators annotated the dataset based on the sentiment polarity of each aspect. A total of 4000 reviews were collected, and a total of 7507 aspects are identified in the reviews. Spearman's correlation and Fleiss Kappa Index are used to analyze the annotated dataset's correlation. It has been found that the high correlation between the annotators implies that the MABSA dataset is of gold standard.
Membership Inference Attacks against Diffusion Models

Diffusion models have attracted attention in recent years as innovative generative models. In this paper, we investigate whether a diffusion model is resistant to a membership inference attack, which evaluates the privacy leakage of a machine learning model. We primarily discuss the diffusion model from the standpoints of comparison with a generative adversarial network (GAN) as conventional models and hyperparameters unique to the diffusion model, i.e., timesteps, sampling steps, and sampling variances. We conduct extensive experiments with DDIM as a diffusion model and DCGAN as a GAN on the CelebA and CIFAR-10 datasets in both white-box and black-box settings and then show that the diffusion model is comparably resistant to a membership inference attack as GAN. Next, we demonstrate that the impact of timesteps is significant and intermediate steps in a noise schedule are the most vulnerable to the attack. We also found two key insights through further analysis. First, we identify that DDIM is vulnerable to the attack for small sample sizes instead of achieving a lower FID. Second, sampling steps in hyperparameters are important for resistance to the attack, whereas the impact of sampling variances is quite limited.
The (ab)use of Open Source Code to Train Large Language Models

In recent years, Large Language Models (LLMs) have gained significant popularity due to their ability to generate human-like text and their potential applications in various fields, such as Software Engineering. LLMs for Code are commonly trained on large unsanitized corpora of source code scraped from the Internet. The content of these datasets is memorized and emitted by the models, often in a verbatim manner. In this work, we will discuss the security, privacy, and licensing implications of memorization. We argue why the use of copyleft code to train LLMs is a legal and ethical dilemma. Finally, we provide four actionable recommendations to address this issue.
SafeFL: MPC-friendly Framework for Private and Robust Federated Learning

Federated learning (FL) has gained widespread popularity in a variety of industries due to its ability to locally train models on devices while preserving privacy. However, FL systems are susceptible to i) privacy inference attacks and ii) poisoning attacks, which can compromise the system by corrupt actors. Despite a significant amount of work being done to tackle these attacks individually, the combination of these two attacks has received limited attention in the research community. To address this gap, we introduce SafeFL, a secure multiparty computation (MPC)-based framework designed to assess the efficacy of FL techniques in addressing both privacy inference and poisoning attacks. The heart of the SafeFL framework is a communicator interface that enables PyTorch-based implementations to utilize the well-established MP-SPDZ framework, which implements various MPC protocols. The goal of SafeFL is to facilitate the development of more efficient FL systems that can effectively address privacy inference and poisoning attacks.
AutoAC: Towards Automated Attribute Completion for Heterogeneous Graph Neural Network

Many real-world data can be modeled as heterogeneous graphs that contain multiple types of nodes and edges. Meanwhile, due to excellent performance, heterogeneous graph neural networks (GNNs) have received more and more attention. However, the existing work mainly focuses on the design of novel GNN models, while ignoring another important issue that also has a large impact on the model performance, namely the missing attributes of some node types. The handcrafted attribute completion requires huge expert experience and domain knowledge. Also, considering the differences in semantic characteristics between nodes, the attribute completion should be fine-grained, i.e., the attribute completion operation should be node-specific. Moreover, to improve the performance of the downstream graph learning task, attribute completion and the training of the heterogeneous GNN should be jointly optimized rather than viewed as two separate processes. To address the above challenges, we propose a differentiable attribute completion framework called AutoAC for automated completion operation search in heterogeneous GNNs. We first propose an expressive completion operation search space, including topology-dependent and topology-independent completion operations. Then, we propose a continuous relaxation schema and further propose a differentiable completion algorithm where the completion operation search is formulated as a bi-level joint optimization problem. To improve the search efficiency, we leverage two optimization techniques: discrete constraints and auxiliary unsupervised graph node clustering. Extensive experimental results on real-world datasets reveal that AutoAC outperforms the SOTA handcrafted heterogeneous GNNs and the existing attribute completion method.
Gar: A Generate-and-Rank Approach for Natural Language to SQL Translation

A Natural Language (NL) Interface to Databases (NLIDB) aims to help end-users access databases. State-of-the-art approaches primarily construct language translation models to convert NL queries to SQL queries. While these models exhibit good performance on NLIDB benchmarks, the translation accuracy seems to have stalled at between 70%-75%, and most erroneous translations happen with complex queries that require an understanding of the structure and semantics specific to a database. This paper proposes a Generate-And-Rank approach called Gar. Gar assumes that a set of sample SQL queries is given to represent the possible user-intended queries to the database. In order to provide a broad coverage, akin to avoiding over-fitting, Gar extracts the basic components from the sample set to form the basic building blocks to generate a set of generalized SQL queries. By leveraging a simple rule-based SQL to NL technique, a less natural NL expression called a dialect expression for each sample and generalized SQL query is obtained. Finally, a learning-to-rank method is used for a given NL query to retrieve the best dialect expression and hence the resulting SQL query. Extensive experiments are performed to study Gar in comparison with other approaches. The results show that Gar achieves better performance on the NLIDB benchmarks, including in particular a 78.5% translation accuracy on the popular Spider benchmark, outperforming the best reported accuracy in the literature. An extension to Gar, called Gar-j, is further introduced to aid the translation by annotating join semantics in the sample queries. The experimental results show that Gar-j can further improve translation accuracy on queries with joins. Code for Gar can be found at https://github.com/Kaimary/GAR.
ResuFormer: Semantic Structure Understanding for Resumes via Multi-Modal Pre-training

Understanding the semantic structure of resumes plays an important role for various intelligent recruitment related applications. However, due to the unique characteristics of resume documents (e.g., diverse writing styles and multi-page) and the lack of labeled data, it has been a long-standing challenge to effectively extract the structural information of resumes through machine learning models. While considerable efforts have been made in this direction, existing methods only focus on the textual information in the document where the rich multi-modal information (e.g., the visual and layout information) is largely ignored. To this end, in this paper, we propose ResuFormer for understanding the semantic structure of resumes. Specifically, ResuFormer focuses on two typical tasks in this direction, namely resume block classification and intra-block information extraction respectively. For the first task, we propose a multi-modal pre-training model with a hierarchical Transformer encoder, in which we design three self-supervised training objectives, i.e., masked layout-language model, self-supervised contrastive learning and dynamic next-sentence prediction, to pre-train the model parameters, and fine-tune the model only using a small amount of training data. For the second task, we introduce a self-distillation based self-training learning framework to make the distantly supervised model more robust to the noise data. Finally, extensive experiments conducted on real-world resume datasets have clearly validated the performance of our ResuFormer compared with state-of-the-art (SOTA) baselines.
Dynamic Shapley Value Computation

With the prevalence of data-driven research, data valuation has attracted attention from the computer science field. How to appraise a single datum becomes an imperative problem, especially in the context of machine learning. Shapley value is widely used to fairly measure the contribution of data points in machine learning since it is the unique definition that satisfies all four desired properties: balance, symmetry, additivity, and zero element. However, computing Shapley value is known to be a #P-hard problem. As data is subject to changes, dynamic data exists pervasively in real-world scenarios. Pricing such dynamic data is more challenging due to the prohibitively expensive cost of recalculation from scratch. In this paper, we study the problem of Dynamic Shapley Value Computation, which updates Shapley value when dynamically adding/deleting data points. For adding data points, to prune unnecessary computation of overlapping model utilities, we propose the pivot-based algorithm that can reduce half computation time in general. We also propose the delta-based algorithm to capture Shapley value changes, which requires a smaller sample size to converge. For deleting data points, we present the YN-NN algorithm that derives the new Shapley value from the data structure of precomputed model utilities in an efficient way. Based on Shapley value changes, we give another version of the delta-based algorithm for deleting data points. Besides, we propose heuristic algorithms to draw on experimental observations for both adding and deleting data points. Extensive experimental results demonstrate the efficiency and effectiveness of our proposed algorithms.
Marshalling Model Inference in Video Streams

Numerous cloud platforms are available to deploy and train deep models as well as process data, such as Amazon Rekognition and Azure custom Vision Service, which have made it easy for companies to adopt deep learning technologies in their operations. Commonly such services price usage per image or frame in typical applications that consume video streams and as a result the costs rapidly accumulate. In this paper we introduce a model, named EventHit, that is able to marshal model inference requests in such services by making predictions over the video stream about events of interest. As such only relevant video segments are sent for analysis to the cloud infrastructure and irrelevant parts are filtered from further processing. We introduce the architecture and fully describe its components. We present two novel optimizations in this context that aim to provide control over the trade-off between prediction accuracy (especially regarding the probability of missing an event of interest) and processing cost at the cloud infrastructure. We fully describe and analyze our proposals in the context of real datasets. We also present the results of a detailed experimental evaluation varying parameters of interest and demonstrate the practical utility of our proposals.
Firestore: The NoSQL Serverless Database for the Application Developer

The recent years have seen an explosive growth in web and mobile application development. Such applications typically have rapid development cycles, and their developers expect mobile-friendly features and serverless characteristics such as rapid deployment capabilities (with minimal initialization), scalability to handle workload spikes, and flexible pay-as-you-go billing. Google’s Firestore is a NoSQL serverless database with real-time notification capability, and together with the Firebase ecosystem greatly simplifies common app development challenges by letting application developers focus primarily on their business logic and user experience. This paper presents the Firestore architecture, how it satisfies the aforementioned requirements, and how its real-time notification system works in tandem with Firebase client libraries to allow mobile applications to provide a smooth user experience even in the presence of network connectivity issues.
Qualitative failures of image generation models and their application in detecting deepfakes

The remarkable advancement of image and video generation models has led to the creation of exceptionally realistic content, posing challenges in differentiating between genuine and fabricated instances in numerous scenarios. However, despite this progress, a gap remains between the quality of generated images and those found in the real world. To address this, we have reviewed a vast body of literature from both academic publications and social media to identify qualitative shortcomings in image generation models, which we have classified into five categories. By understanding these failures, we can identify areas where these models need improvement, as well as develop strategies for detecting generated images and deepfakes. The prevalence of deepfakes in today's society is a serious concern, and our findings can help mitigate their negative impact. In order to support research in this field, a collection of instances where models have failed is made available at here.
DaRAV: A Tool for Visualizing De-Anonymization Risks

Personal data is any information that relates to an individual. Before disclosing such data to third parties, data controllers must be aware of the de-anonymization risks associated with their datasets and take appropriate anonymization measures. To carry out such actions, data controllers require tools that can analyze the risks in their datasets while also providing the necessary anonymization methods for addressing those risks. Existing tools of this type are insufficient for handling high-dimensional data as well as visualizing their risks. In this paper, we demonstrate DaRAV (De-anonymization Risk Analysis through Visualizations), a tool that addresses these limitations by providing risk analysis methods for five types of complex, high-dimensional data through interactive visualizations, as well as anonymization methods that allow users to create anonymized versions of their data.
Lumos: Heterogeneity-aware Federated Graph Learning over Decentralized Devices

Graph neural networks (GNN) have been widely deployed in real-world networked applications and systems due to their capability to handle graph-structured data. However, the growing awareness of data privacy severely challenges the traditional centralized model training paradigm, where a server holds all the graph information. Federated learning is an emerging collaborative computing paradigm that allows model training without data centralization. Existing federated GNN studies mainly focus on systems where clients hold distinctive graphs or sub-graphs. The practical node-level federated situation, where each client is only aware of its direct neighbors, has yet to be studied. In this paper, we propose the first federated GNN framework called Lumos that supports supervised and unsupervised learning with feature and degree protection on node-level federated graphs. We first design a tree constructor to improve the representation capability given the limited structural information. We further present a Monte Carlo Markov Chain-based algorithm to mitigate the workload imbalance caused by degree heterogeneity with theoretically-guaranteed performance. Based on the constructed tree for each client, a decentralized tree-based GNN trainer is proposed to support versatile training. Extensive experiments demonstrate that Lumos outperforms the baseline with significantly higher accuracy and greatly reduced communication cost and training time.
Knowledge Graph Embedding Based on Graph Neural Network

The representation of semantic information pertaining to the real world has been active research for some time now. Among the available methods, knowledge graphs have emerged as a widely accepted approach. Meanwhile, graph neural networks (GNNs) have demonstrated excellent performance in embedding graph-based information. Given the natural graph structure of knowledge graphs, employing GNNs to embed them is expected to yield a more interpretable and trustworthy representation of the learned knowledge. In this paper, we propose three customized GNNs for different scenarios of knowledge graph representation, including traditional, multimodal, and uncertain knowledge graphs. In the traditional knowledge graph scenario, we present a graph self-supervised learning method, named deep relation graph infomax (DRGI), which incorporates both the complete graph structure information and semantic information. In the multimodal knowledge graph scenario, we introduce a novel network, named hyper-node relational graph attention network (HRGAT), which combines different modal information with graph structure information for a more precise representation of multimodal knowledge graphs. In the uncertain knowledge graph scenario, we define a novel message-passing paradigm with box embedding, named box graph neural network (BGNN). BGNN leverages both the graph structure information of uncertain knowledge graphs and the probabilistic semantics of box embedding. To validate the effectiveness of our proposed methods, we conduct a series of experiments and report the results. We also discuss possible future work in GNN-based knowledge graph embedding.