Reviewerly/exHarmony huggingface dataset:
Backward Lens: Projecting Language Model Gradients into the Vocabulary  Space

Authors:
raw_author_name: Belinkov, Yonatan
raw_author_name: Geva, Mor
raw_author_name: Wolf, Lior

Abstract: Understanding how Transformer-based Language Models (LMs) learn and recall information is a key goal of the deep learning community. Recent interpretability methods project weights and hidden states obtained from the forward pass to the models' vocabularies, helping to uncover how information flows within LMs. In this work, we extend this methodology to LMs' backward pass and gradients. We first prove that a gradient matrix can be cast as a low-rank linear combination of its forward and backward passes' inputs. We then develop methods to project these gradients into vocabulary items and explore the mechanics of how new information is stored in the LMs' neurons.
ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic

Authors:
raw_author_name: Koto, Fajri
raw_author_name: Li, Haonan
raw_author_name: Shatnawi, Sara
raw_author_name: Doughman, Jad
raw_author_name: Almubarak, Khalid
raw_author_name: Alyafeai, Zaid
raw_author_name: Sengupta, Neha
raw_author_name: Shehata, Shady
raw_author_name: Habash, Nizar
raw_author_name: Nakov, Preslav
raw_author_name: Baldwin, Timothy

Abstract: The focus of language model evaluation has transitioned towards reasoning and knowledge-intensive tasks, driven by advancements in pretraining large models. While state-of-the-art models are partially trained on large Arabic texts, evaluating their performance in Arabic remains challenging due to the limited availability of relevant datasets. To bridge this gap, we present ArabicMMLU, the first multi-task language understanding benchmark for Arabic language, sourced from school exams across diverse educational levels in different countries spanning North Africa, the Levant, and the Gulf regions. Our data comprises 40 tasks and 14,575 multiple-choice questions in Modern Standard Arabic (MSA), and is carefully constructed by collaborating with native speakers in the region. Our comprehensive evaluations of 35 models reveal substantial room for improvement, particularly among the best open-source models. Notably, BLOOMZ, mT0, LLama2, and Falcon struggle to achieve a score of 50%, while even the top-performing Arabic-centric model only achieves a score of 62.3%.
Privacy computing meets metaverse: Necessity, taxonomy and challenges

Authors:
raw_author_name: Chuan Chen
raw_affiliation_string: School of Computer Science and Engineering, Sun Yat-sen University, GuangZhou, China
raw_author_name: Yuecheng Li
raw_affiliation_string: School of Computer Science and Engineering, Sun Yat-sen University, GuangZhou, China
raw_author_name: Chengyuan Mai
raw_affiliation_string: School of Computer Science and Engineering, Sun Yat-sen University, GuangZhou, China
raw_author_name: Yanming Hu
raw_affiliation_string: School of Computer Science and Engineering, Sun Yat-sen University, GuangZhou, China
raw_author_name: Zibin Zheng
raw_affiliation_string: School of Software Engineering, Sun Yat-sen University, ZhuHai, China
raw_author_name: Jiawen Kang
raw_affiliation_string: School of Automation, Guangdong University of Technology, GuangZhou, China

Abstract: Metaverse, the core of the next-generation Internet, is a computer-generated holographic digital environment that simultaneously combines spatio-temporal, immersive, real-time, sustainable, interoperable, and data-sensitive characteristics. It cleverly blends the virtual and real worlds, allowing users to create, communicate, and transact in virtual form. With the rapid development of emerging technologies including augmented reality, virtual reality and blockchain, the metaverse system is becoming more and more sophisticated and widely used in various fields such as social, tourism, industry and economy. However, the high level of interaction with the real world also means a huge risk of privacy leakage both for individuals and enterprises, which has hindered the wide deployment of metaverse. Then, it is inevitable to apply privacy computing techniques in the framework of metaverse, which is a current research hotspot. In this paper, we conduct comprehensive research on the necessity, taxonomy and challenges when privacy computing meets metaverse. Specifically, we first introduce the underlying technologies and various applications of metaverse, on which we analyze the challenges of data usage in metaverse, especially data privacy. Next, we review and summarize state-of-the-art solutions based on federated learning, differential privacy, homomorphic encryption, and zero-knowledge proofs for different privacy problems in metaverse. Finally, we show the current security and privacy challenges in the development of metaverse and provide open directions for building a well-established privacy-preserving metaverse system.
Mask-up: Investigating Biases in Face Re-identification for Masked Faces

Authors:
raw_author_name: Jaiswal, Siddharth D
raw_author_name: Verma, Ankit Kr.
raw_author_name: Mukherjee, Animesh

Abstract: AI based Face Recognition Systems (FRSs) are now widely distributed and deployed as MLaaS solutions all over the world, moreso since the COVID-19 pandemic for tasks ranging from validating individuals' faces while buying SIM cards to surveillance of citizens. Extensive biases have been reported against marginalized groups in these systems and have led to highly discriminatory outcomes. The post-pandemic world has normalized wearing face masks but FRSs have not kept up with the changing times. As a result, these systems are susceptible to mask based face occlusion. In this study, we audit four commercial and nine open-source FRSs for the task of face re-identification between different varieties of masked and unmasked images across five benchmark datasets (total 14,722 images). These simulate a realistic validation/surveillance task as deployed in all major countries around the world. Three of the commercial and five of the open-source FRSs are highly inaccurate; they further perpetuate biases against non-White individuals, with the lowest accuracy being 0%. A survey for the same task with 85 human participants also results in a low accuracy of 40%. Thus a human-in-the-loop moderation in the pipeline does not alleviate the concerns, as has been frequently hypothesized in literature. Our large-scale study shows that developers, lawmakers and users of such services need to rethink the design principles behind FRSs, especially for the task of face re-identification, taking cognizance of observed biases.
Parsimonious Generative Machine Learning for Non-Gaussian Tail Modeling  and Risk-Neutral Distribution Extraction

Authors:
raw_author_name: Wu, Qi
raw_author_name: Xian, Zhonghao
raw_author_name: Yan, Xing
raw_author_name: Yang, Nan

Abstract: In financial modeling problems, non-Gaussian tails exist widely in many circumstances. Among them, the accurate estimation of risk-neutral distribution (RND) from option prices is of great importance for researchers and practitioners. A precise RND can provide valuable information regarding the market's expectations, and can further help empirical asset pricing studies. This paper presents a parsimonious parametric approach to extract RNDs of underlying asset returns by using a generative machine learning model. The model incorporates the asymmetric heavy tails property of returns with a clever design. To calibrate the model, we design a Monte Carlo algorithm that has good capability with the assistance of modern machine learning computing tools. Numerically, the model fits Heston option prices well and captures the main shapes of implied volatility curves. Empirically, using S\&P 500 index option prices, we demonstrate that the model outperforms some popular parametric density methods under mean absolute error. Furthermore, the skewness and kurtosis of RNDs extracted by our model are consistent with intuitive expectations.
Imbalanced Data Clustering using Equilibrium K-Means

Authors:
raw_author_name: He, Yudong

Abstract: Imbalanced data, characterized by an unequal distribution of data points across different clusters, poses a challenge for traditional hard and fuzzy clustering algorithms, such as hard K-means (HKM, or Lloyd's algorithm) and fuzzy K-means (FKM, or Bezdek's algorithm). This paper introduces equilibrium K-means (EKM), a novel and simple K-means-type algorithm that alternates between just two steps, yielding significantly improved clustering results for imbalanced data by reducing the tendency of centroids to crowd together in the center of large clusters. We also present a unifying perspective for HKM, FKM, and EKM, showing they are essentially gradient descent algorithms with an explicit relationship to Newton's method. EKM has the same time and space complexity as FKM but offers a clearer physical meaning for its membership definition. We illustrate the performance of EKM on two synthetic and ten real datasets, comparing it to various clustering algorithms, including HKM, FKM, maximum-entropy fuzzy clustering, two FKM variations designed for imbalanced data, and the Gaussian mixture model. The results demonstrate that EKM performs competitively on balanced data while significantly outperforming other techniques on imbalanced data. For high-dimensional data clustering, we demonstrate that a more discriminative representation can be obtained by mapping high-dimensional data via deep neural networks into a low-dimensional, EKM-friendly space. Deep clustering with EKM improves clustering accuracy by 35% on an imbalanced dataset derived from MNIST compared to deep clustering based on HKM.
TSFIS-GWO: Metaheuristic-Driven Takagi-Sugeno Fuzzy System for Adaptive Real-Time Routing in WBANs

Authors:
raw_author_name: Mohammad Shokouhifar
raw_affiliation_string: Department of Electrical and Computer Engineering, Shahid Beheshti University, Tehran 8174673441, Iran
raw_author_name: Seyedali Mirjalili
raw_affiliation_string: Center for Artificial Intelligence Research and Optimisation, Torrens University Australia, Brisbane, QLD 4006, Australia; University Research and Innovation Center (EKIK), Óbuda University, 1034 Budapest, Hungary

Abstract: Wireless body area network (WBAN) is an internet-of-things technology that facilitates remote patient monitoring and enables medical staff to administer timely treatments. One of the main challenges in designing WBANs is the routing problem, which is complicated due to dynamic changes in network topology and the limited resources of nodes. Several heuristic and metaheuristic methods have been presented to solve the routing problem in WBANs. Although metaheuristics outperform heuristics by producing higher-quality solutions, they cannot respond to real-time requests. This paper introduces a reactive routing protocol for WBANs that combines a fuzzy heuristic with a metaheuristic learning model. It utilizes a Takagi-Sugeno Fuzzy Inference System in conjunction with the Grey Wolf Optimizer (named TSFIS-GWO). The objective is to simultaneously benefit from the advantages of both approaches, namely, the effectiveness of metaheuristics for offline hyperparameter tuning and the quickness of fuzzy heuristics for real-time routing. At every round, the tuned fuzzy system takes multiple parameters of the current state of the nodes and links to construct the multi-hop routing tree under IEEE 802.15.6. To optimize the performance of the protocol for each WBAN, the fuzzy rules of the TSFIS model are automatically adjusted through a learning method based on GWO. This is done in accordance with the specific requirements of the application, and the tuning process takes place once before the protocol is applied. Simulation results in three applications demonstrate that the proposed TSFIS-GWO model is capable of providing real-time solutions while outperforming the existing methods in terms of application-specific performance measures.
Reconstruction techniques for quantum trees

Authors:
raw_author_name: Vladislav V. Kravchenko
raw_affiliation_string: Department of Mathematics Cinvestav  Santiago de Querétaro Mexico

Abstract: The inverse problem of recovery of a potential on a quantum tree graph from the Weyl matrix given at a number of points is considered. A method for its numerical solution is proposed. The overall approach is based on the leaf peeling method combined with Neumann series of Bessel functions (NSBF) representations for solutions of Sturm–Liouville equations. In each step, the solution of the arising inverse problems reduces to dealing with the NSBF coefficients. The leaf peeling method allows one to localize the general inverse problem to local problems on sheaves, while the approach based on the NSBF representations leads to splitting the local problems into two‐spectrum inverse problems on separate edges and reduces them to systems of linear algebraic equations for the NSBF coefficients. Moreover, the potential on each edge is recovered from the very first NSBF coefficient. The proposed method leads to an efficient numerical algorithm that is illustrated by numerical tests.
Multi Objective and Multi-Product Perishable Supply Chain with Vendor-Managed Inventory and IoT-Related Technologies

Authors:
raw_author_name: Seyed Mojtaba Sajadi
raw_affiliation_string: Operations and Information Management Department, Aston Business School, Aston University, Birmingham B4 7ET, UK
raw_author_name: Seyed Esmaeil Najafi
raw_affiliation_string: Department of Industrial Engineering, Science and Research Branch, Islamic Azad University, Tehran 1477893855, Iran
raw_author_name: Mohammadreza Taghizadeh-Yazdi
raw_affiliation_string: Faculty of Management, University of Tehran, Tehran 1417466191, Iran

Abstract: With the emergence of the fourth industrial revolution, the use of intelligent technologies in supply chains is becoming increasingly common. The aim of this research is to propose an optimal design for an intelligent supply chain of multiple perishable products under a vendor-managed inventory management policy aided by IoT-related technologies to address the challenges associated with traditional supply chains. Various levels of the intelligent supply chain employ technologies such as Wireless Sensor Networks (WSNs), Radio Frequency Identification (RFID), and Blockchain. In this paper, we develop a bi-objective nonlinear integer mathematical programming model for designing a four-level supply chain consisting of suppliers, manufacturers, retailers, and customers. The model determines the optimal network nodes, production level, product distribution and sales, and optimal choice of technology for each level. The objective functions are total cost and delivery times. The GAMS 24.2.1 optimization software is employed to solve the mathematical model in small dimensions. Considering the NP-Hard nature of the problem, the Grey Wolf Optimizer (GWO) algorithm is employed, and its performance is compared with the Multi-Objective Whale Optimization Algorithm (MOWOA) and NSGA-III. The results indicate that the adoption of these technologies in the supply chain can reduce delivery times and total supply chain costs.
Improving Sentence Embeddings with an Automatically Generated NLI  Dataset

Authors:
raw_author_name: Tsukagoshi, Hayato
raw_author_name: Sasano, Ryohei
raw_author_name: Takeda, Koichi

Abstract: Decoder-based large language models (LLMs) have shown high performance on many tasks in natural language processing. This is also true for sentence embedding learning, where a decoder-based model, PromptEOL, has achieved the best performance on semantic textual similarity (STS) tasks. However, PromptEOL makes great use of fine-tuning with a manually annotated natural language inference (NLI) dataset. We aim to improve sentence embeddings learned in an unsupervised setting by automatically generating an NLI dataset with an LLM and using it to fine-tune PromptEOL. In experiments on STS tasks, the proposed method achieved an average Spearman's rank correlation coefficient of 82.21 with respect to human evaluation, thus outperforming existing methods without using large, manually annotated datasets.
How (un)ethical are instruction-centric responses of LLMs? Unveiling the  vulnerabilities of safety guardrails to harmful queries

Authors:
raw_author_name: Banerjee, Somnath
raw_author_name: Layek, Sayan
raw_author_name: Hazra, Rima
raw_author_name: Mukherjee, Animesh

Abstract: In this study, we tackle a growing concern around the safety and ethical use of large language models (LLMs). Despite their potential, these models can be tricked into producing harmful or unethical content through various sophisticated methods, including 'jailbreaking' techniques and targeted manipulation. Our work zeroes in on a specific issue: to what extent LLMs can be led astray by asking them to generate responses that are instruction-centric such as a pseudocode, a program or a software snippet as opposed to vanilla text. To investigate this question, we introduce TechHazardQA, a dataset containing complex queries which should be answered in both text and instruction-centric formats (e.g., pseudocodes), aimed at identifying triggers for unethical responses. We query a series of LLMs -- Llama-2-13b, Llama-2-7b, Mistral-V2 and Mistral 8X7B -- and ask them to generate both text and instruction-centric responses. For evaluation we report the harmfulness score metric as well as judgements from GPT-4 and humans. Overall, we observe that asking LLMs to produce instruction-centric responses enhances the unethical response generation by ~2-38% across the models. As an additional objective, we investigate the impact of model editing using the ROME technique, which further increases the propensity for generating undesirable content. In particular, asking edited LLMs to generate instruction-centric responses further increases the unethical response generation by ~3-16% across the different models.
A Data-Centric Approach To Generate Faithful and High Quality Patient  Summaries with Large Language Models

Authors:
raw_author_name: Hegselmann, Stefan
raw_author_name: Agrawal, Monica
raw_author_name: Sontag, David
raw_author_name: Jiang, Xiaoyi

Abstract: Patients often face difficulties in understanding their hospitalizations, while healthcare workers have limited resources to provide explanations. In this work, we investigate the potential of large language models to generate patient summaries based on doctors' notes and study the effect of training data on the faithfulness and quality of the generated summaries. To this end, we develop a rigorous labeling protocol for hallucinations, and have two medical experts annotate 100 real-world summaries and 100 generated summaries. We show that fine-tuning on hallucination-free data effectively reduces hallucinations from 2.60 to 1.55 per summary for Llama 2, while preserving relevant information. Although the effect is still present, it is much smaller for GPT-4 when prompted with five examples (0.70 to 0.40). We also conduct a qualitative evaluation using hallucination-free and improved training data. GPT-4 shows very good results even in the zero-shot setting. We find that common quantitative metrics do not correlate well with faithfulness and quality. Finally, we test GPT-4 for automatic hallucination detection, which yields promising results.
Evaluating Publications' Keywords in Computer Science Education Research: A Bibliometric NLP Approach

Authors:
raw_author_name: Jia Zhu
raw_affiliation_string: Florida International University
raw_author_name: Leila Zahedi
raw_affiliation_string: Florida International University

Abstract: Abstract In the rapidly growing field of computer science education research (CSER), the collection of publication keywords represent the knowledge landscape of the research domain. This helps stakeholders to contextualize existing research topics, identify the strategic direction for future works, and shape the culture of this growing community. For the research studies published in CSER, the use of proper keywords will help to improve publication visibility in research networks. Publication visibility is defined as the share of traffic a study receives based on its ranking from Search Engine Result Pages (SERPs), which can be gained through indexing relevant keywords to help search engines to find the related studies. It is important to acknowledge that analyzing keyword selections in the context provided by the publication abstract, enables us to determine the keywords’ relevance to the research topics. However, evaluating the selection of publication keywords is an important but lesser-addressed issue. In this work, we utilized a bibliometric approach to scrutinize publication keywords in CSER publications from 2015 to 2020. We gathered the titles, abstracts, and keywords collectively from studies published in two major CS education conferences and journals. We also extracted the strategic directions from their mission statements. By applying the Natural Language Processing (NLP) technique, known as Term Frequency-Inverse Document Frequency (tf-idf), we aimed to answer: 1) What are the most prevalent research foci represented in CSER through the publication keywords? 2) Are publication keywords in alignment with the CSER publications’ stated strategic directions? 3) Do selected keywords in CSER publications adequately represent the research topics presented via publication abstracts? Our preliminary results suggest that, as expected, the most prevalent research foci represented by the most commonly used keywords of CSER were on teaching and learning and broadening participation, which is in alignment with the corresponding strategic directions. Our analysis suggests that the CSER community is putting efforts to formulate a diversified culture through teaching and learning for inclusivity. However, our results also suggest that there is a misalignment between keyword identification by authors and the topics presented in the abstracts. With our work, we hope to motivate scholars to carefully evaluate and select the keywords that will be used in the future for indexing publications to improve the research topic relevancy and publication visibility for broader impact.