Reviewerly/exHarmony huggingface dataset:
TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models

Factual consistency evaluation is often conducted using Natural Language Inference (NLI) models, yet these models exhibit limited success in evaluating summaries. Previous work improved such models with synthetic training data. However, the data is typically based on perturbed human-written summaries, which often differ in their characteristics from real model-generated summaries and have limited coverage of possible factual errors. Alternatively, large language models (LLMs) have recently shown promising results in directly evaluating generative tasks, but are too computationally expensive for practical use. Motivated by these limitations, we introduce TrueTeacher, a method for generating synthetic data by annotating diverse model-generated summaries using a LLM. Unlike prior work, TrueTeacher does not rely on human-written summaries, and is multilingual by nature. Experiments on the TRUE benchmark show that a student model trained using our data, substantially outperforms both the state-of-the-art model with similar capacity, and the LLM teacher. In a systematic study, we compare TrueTeacher to existing synthetic data generation methods and demonstrate its superiority and robustness to domain-shift. We also show that our method generalizes to multilingual scenarios. Lastly, we release our large scale synthetic dataset (1.4M examples), generated using TrueTeacher, and a checkpoint trained on this data.
On the Risk of Misinformation Pollution with Large Language Models

We investigate the potential misuse of modern Large Language Models (LLMs) for generating credible-sounding misinformation and its subsequent impact on information-intensive applications, particularly Open-Domain Question Answering (ODQA) systems. We establish a threat model and simulate potential misuse scenarios, both unintentional and intentional, to assess the extent to which LLMs can be utilized to produce misinformation. Our study reveals that LLMs can act as effective misinformation generators, leading to a significant degradation (up to 87%) in the performance of ODQA systems. Moreover, we uncover disparities in the attributes associated with persuading humans and machines, presenting an obstacle to current human-centric approaches to combat misinformation. To mitigate the harm caused by LLM-generated misinformation, we propose three defense strategies: misinformation detection, vigilant prompting, and reader ensemble. These approaches have demonstrated promising results, albeit with certain associated costs. Lastly, we discuss the practicality of utilizing LLMs as automatic misinformation generators and provide relevant resources and code to facilitate future research in this area.
Distilling ChatGPT for Explainable Automated Student Answer Assessment

Providing explainable and faithful feedback is crucial for automated student answer assessment. In this paper, we introduce a novel framework that explores using ChatGPT, a cutting-edge large language model, for the concurrent tasks of student answer scoring and rationale generation. We identify the appropriate instructions by prompting ChatGPT with different templates to collect the rationales, where inconsistent rationales are refined to align with marking standards. The refined ChatGPT outputs enable us to fine-tune a smaller language model that simultaneously assesses student answers and provides rationales. Extensive experiments on the benchmark dataset show that the proposed method improves the overall QWK score by 11% compared to ChatGPT. Furthermore, our thorough analysis and human evaluation demonstrate that the rationales generated by our proposed method are comparable to those of ChatGPT. Our approach provides a viable solution to achieve explainable automated assessment in education
IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models

The field of vision-and-language (VL) understanding has made unprecedented progress with end-to-end large pre-trained VL models (VLMs). However, they still fall short in zero-shot reasoning tasks that require multi-step inferencing. To achieve this goal, previous works resort to a divide-and-conquer pipeline. In this paper, we argue that previous efforts have several inherent shortcomings: 1) They rely on domain-specific sub-question decomposing models. 2) They force models to predict the final answer even if the sub-questions or sub-answers provide insufficient information. We address these limitations via IdealGPT, a framework that iteratively decomposes VL reasoning using large language models (LLMs). Specifically, IdealGPT utilizes an LLM to generate sub-questions, a VLM to provide corresponding sub-answers, and another LLM to reason to achieve the final answer. These three modules perform the divide-and-conquer procedure iteratively until the model is confident about the final answer to the main question. We evaluate IdealGPT on multiple challenging VL reasoning tasks under a zero-shot setting. In particular, our IdealGPT outperforms the best existing GPT-4-like models by an absolute 10% on VCR and 15% on SNLI-VE. Code is available at https://github.com/Hxyou/IdealGPT.
MEGClass: Extremely Weakly Supervised Text Classification via Mutually-Enhancing Text Granularities

Text classification is essential for organizing unstructured text. Traditional methods rely on human annotations or, more recently, a set of class seed words for supervision, which can be costly, particularly for specialized or emerging domains. To address this, using class surface names alone as extremely weak supervision has been proposed. However, existing approaches treat different levels of text granularity (documents, sentences, or words) independently, disregarding inter-granularity class disagreements and the context identifiable exclusively through joint extraction. In order to tackle these issues, we introduce MEGClass, an extremely weakly-supervised text classification method that leverages Mutually-Enhancing Text Granularities. MEGClass utilizes coarse- and fine-grained context signals obtained by jointly considering a document’s most class-indicative words and sentences. This approach enables the learning of a contextualized document representation that captures the most discriminative class indicators. By preserving the heterogeneity of potential classes, MEGClass can select the most informative class-indicative documents as iterative feedback to enhance the initial word-based class representations and ultimately fine-tune a pre-trained text classifier. Extensive experiments on seven benchmark datasets demonstrate that MEGClass outperforms other weakly and extremely weakly supervised methods.
LM vs LM: Detecting Factual Errors via Cross Examination

A prominent weakness of modern language models (LMs) is their tendency to generate factually incorrect text, which hinders their usability. A natural question is whether such factual errors can be detected automatically. Inspired by truth-seeking mechanisms in law, we propose a factuality evaluation framework for LMs that is based on cross-examination. Our key idea is that an incorrect claim is likely to result in inconsistency with other claims that the model generates. To discover such inconsistencies, we facilitate a multi-turn interaction between the LM that generated the claim and another LM (acting as an examiner) which introduces questions to discover inconsistencies. We empirically evaluate our method on factual claims made by multiple recent LMs on four benchmarks, finding that it outperforms existing methods and baselines, often by a large gap. Our results demonstrate the potential of using interacting LMs for capturing factual errors.
Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization

Federated learning (FL) is a promising paradigm to enable collaborative model training with decentralized data. However, the training process of Large Language Models (LLMs) generally incurs the update of significant parameters, which limits the applicability of FL techniques to tackle the LLMs in real scenarios. Prompt tuning can significantly reduce the number of parameters to update, but it either incurs performance degradation or low training efficiency. The straightforward utilization of prompt tuning in the FL often raises non-trivial communication costs and dramatically degrades performance. In addition, the decentralized data is generally non-Independent and Identically Distributed (non-IID), which brings client drift problems and thus poor performance. This paper proposes a Parameter-efficient prompt Tuning approach with Adaptive Optimization, i.e., FedPepTAO, to enable efficient and effective FL of LLMs. First, an efficient partial prompt tuning approach is proposed to improve performance and efficiency simultaneously. Second, a novel adaptive optimization method is developed to address the client drift problems on both the device and server sides to enhance performance further. Extensive experiments based on 10 datasets demonstrate the superb performance (up to 60.8% in terms of accuracy) and efficiency (up to 97.59% in terms of training time) of FedPepTAO compared with 9 baseline approaches. Our code is available at https://github.com/llm-eff/FedPepTAO.
SciRepEval: A Multi-Format Benchmark for Scientific Document Representations

Learned representations of scientific documents can serve as valuable input features for downstream tasks without further fine-tuning. However, existing benchmarks for evaluating these representations fail to capture the diversity of relevant tasks. In response, we introduce SciRepEval, the first comprehensive benchmark for training and evaluating scientific document representations. It includes 24 challenging and realistic tasks, 8 of which are new, across four formats: classification, regression, ranking and search. We then use this benchmark to study and improve the generalization ability of scientific document representation models. We show how state-of-the-art models like SPECTER and SciNCL struggle to generalize across the task formats, and that simple multi-task training fails to improve them. However, a new approach that learns multiple embeddings per document, each tailored to a different format, can improve performance. We experiment with task-format-specific control codes and adapters and find they outperform the existing single-embedding state-of-the-art by over 2 points absolute. We release the resulting family of multi-format models, called SPECTER2, for the community to use and build on.
Unlocking the Power of EHRs: Harnessing Unstructured Data for Machine Learning-based Outcome Predictions

The integration of Electronic Health Records (EHRs) with Machine Learning (ML) models has become imperative in examining patient outcomes due to the vast amounts of clinical data they provide. However, critical information regarding social and behavioral factors that affect health, such as social isolation, stress, and mental health complexities, is often recorded in unstructured clinical notes, hindering its accessibility. This has resulted in an over-reliance on clinical data in current EHR-based research, potentially leading to disparities in health outcomes. This study aims to evaluate the impact of incorporating patient-specific context from unstructured EHR data on the accuracy and stability of ML algorithms for predicting mortality, using the MIMIC III database. Results from the study confirmed the significance of incorporating patient-specific information into prediction models, leading to a notable improvement in the discriminatory power and robustness of the ML algorithms. Furthermore, the findings underline the importance of considering non-clinical factors related to a patient’s daily life, in addition to clinical factors, when making predictions about patient outcomes. The advent of advanced generative models, such as GPT-4, presents new opportunities for effectively extracting social and behavioral factors from unstructured clinical notes, further enhancing the accuracy and stability of ML algorithms in predicting patient outcomes. The results of our study have significant ramifications for improving ML in clinical decision support and patient outcome predictions, specifically highlighting the potential role of generative models like GPT-4 in advancing ML-based outcome predictions.
Domain-Specific Pre-training Improves Confidence in Whole Slide Image Classification

Whole Slide Images (WSIs) or histopathology images are used in digital pathology. WSIs pose great challenges to deep learning models for clinical diagnosis, owing to their size and lack of pixel-level annotations. With the recent advancements in computational pathology, newer multiple-instance learning-based models have been proposed. Multiple-instance learning for WSIs necessitates creating patches and uses the encoding of these patches for diagnosis. These models use generic pre-trained models (ResNet-50 pre-trained on ImageNet) for patch encoding. The recently proposed KimiaNet, a DenseNet121 model pre-trained on TCGA slides, is a domain-specific pre-trained model. This paper shows the effect of domain-specific pre-training on WSI classification. To investigate the effect of domain-specific pre-training, we considered the current state-of-the-art multiple-instance learning models, 1) CLAM, an attention-based model, and 2) TransMIL, a self-attention-based model, and evaluated the models’ confidence and predictive performance in detecting primary brain tumors - gliomas. Domain-specific pre-training improves the confidence of the models and also achieves a new state-of-the-art performance of WSI-based glioma subtype classification, showing a high clinical applicability in assisting glioma diagnosis. We will publicly share our code and experimental results at https://github.com/soham-chitnis10/WSI-domain-specific.
Klasifikasi Citra Tanaman Perdu Liar Berkhasiat Obat Menggunakan Jaringan Syaraf Tiruan Radial Basis Function

Wild plants or what are usually called weeds are plants that are considered harmful because they grow in unwanted places. But it turns out that some wild plants have many benefits for the health of the human body. Wild plants have many forms of vegetation, one of which is often encountered is shrubs. There are many wild herbaceous plants that are efficacious as medicine. However, most of the people who do not have knowledge about the types of wild shrubs that have medicinal properties. This study aims to implement the Radial Basis Function (RBF) algorithm for the classification of wild herbaceous plant species with medicinal properties by extracting color and texture features. The color feature extraction is based on the average RGB value, while the texture feature extraction uses a Gabor filter with the mean, entropy, and variance parameters of the magnitude image. The result of feature extraction becomes input data which will be managed by the RBF artificial neural network. RBF is a neural network that has three layers that have feedforward properties that can assist in solving classification or pattern recognition problems. Based on the test results, the precision value is 91%, recall is 89% and accuracy is 90%. These results show that the Radial Basis Function (RBF) algorithm with color and texture feature extraction can classify wild shrubs with medicinal properties well.
Phase Neural Operator for Multi‐Station Picking of Seismic Arrivals

Abstract Seismic wave arrival time measurements form the basis for numerous downstream applications. State‐of‐the‐art approaches for phase picking use deep neural networks to annotate seismograms at each station independently, yet human experts annotate seismic data by examining the whole network jointly. Here, we introduce a general‐purpose network‐wide phase picking algorithm based on a recently developed machine learning paradigm called Neural Operator. Our model, called Phase Neural Operator, leverages the spatio‐temporal contextual information to pick phases simultaneously for any seismic network geometry. This results in superior performance over leading baseline algorithms by detecting many more earthquakes, picking more phase arrivals, while also greatly improving measurement accuracy. Following similar trends being seen across the domains of artificial intelligence, our approach provides but a glimpse of the potential gains from fully‐utilizing the massive seismic data sets being collected worldwide.
NaviSTAR: Socially Aware Robot Navigation with Hybrid Spatio-Temporal Graph Transformer and Preference Learning

Developing robotic technologies for use in human society requires ensuring the safety of robots' navigation behaviors while adhering to pedestrians' expectations and social norms. However, understanding complex human-robot interactions (HRI) to infer potential cooperation and response among robots and pedestrians for cooperative collision avoid-ance is challenging. To address these challenges, we propose a novel socially-aware navigation benchmark called NaviS Tar, which utilizes a hybrid Spatio- Temporal grAph tRansformer to understand interactions in human-rich environments fusing crowd multi-modal dynamic features. We leverage an off-policy reinforcement learning algorithm with preference learning to train a policy and a reward function network with supervi-sor guidance. Additionally, we design a social score function to evaluate the overall performance of social navigation. To compare, we train and test our algorithm with other state-of-the-art methods in both simulator and real-world scenarios independently. Our results show that NaviSTAR outperforms previous methods with outstanding performance <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> The source code and experiment videos of this work are available at: https://sites.google.com/view/san-navistar
Quantum transport of high-dimensional spatial information with a nonlinear detector

Information exchange between two distant parties, where information is shared without physically transporting it, is a crucial resource in future quantum networks. Doing so with high-dimensional states offers the promise of higher information capacity and improved resilience to noise, but progress to date has been limited. Here we demonstrate how a nonlinear parametric process allows for arbitrary high-dimensional state projections in the spatial degree of freedom, where a strong coherent field enhances the probability of the process. This allows us to experimentally realise quantum transport of high-dimensional spatial information facilitated by a quantum channel with a single entangled pair and a nonlinear spatial mode detector. Using sum frequency generation we upconvert one of the photons from an entangled pair resulting in high-dimensional spatial information transported to the other. We realise a d = 15 quantum channel for arbitrary photonic spatial modes which we demonstrate by faithfully transferring information encoded into orbital angular momentum, Hermite-Gaussian and arbitrary spatial mode superpositions, without requiring knowledge of the state to be sent. Our demonstration merges the nascent fields of nonlinear control of structured light with quantum processes, offering a new approach to harnessing high-dimensional quantum states, and may be extended to other degrees of freedom too.
Distributed Broadcast Encryption from Bilinear Groups

Distributed broadcast encryption (DBE) improves on the traditional notion of broadcast encryption by eliminating the key-escrow problem: In a DBE system, users generate their own secret keys non-interactively without the help of a trusted party. Then anyone can broadcast a message for a subset S of the users, in such a way that the resulting ciphertext size is sublinear in (and, ideally, independent of) |S|. Unfortunately, the only known constructions of DBE requires heavy cryptographic machinery, such as general-purpose indistinguishability obfuscation, or come without a security proof. In this work, we formally show that obfuscation is not necessary for DBE, and we present two practical DBE schemes from standard assumptions in prime-order bilinear groups. Our constructions are conceptually simple, satisfy the strong notion of adaptive security, and are concretely efficient. In fact, their performance, in terms of number of group elements and efficiency of the algorithms, is comparable with that of traditional (non distributed) broadcast encryption schemes from bilinear groups.
Hephaestus: Modeling, Analysis, and Performance Evaluation of Cross-Chain Transactions

Ecosystems of multiple blockchains are now a reality. Multichain applications and protocols are perceived as necessary to enable scalability, privacy, and composability. Despite being a promising emerging area, we have been witnessing devastating attacks on cross-chain bridges that have caused billions of dollars in losses, and no apparent solution seems to emerge from the ongoing chaos. In this article, we present our contribution to minimizing bridge attacks, by monitoring a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">cross-chain model</i> . In particular, we aggregate <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">cross-chain events</i> into <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">cross-chain transactions</i> , and verify if they follow a set of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">cross-chain rules</i> , which then generate a model. We propose <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Hephaestus</monospace> , the first cross-chain model generator that captures the operational complexity of cross-chain applications. <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Hephaestus</monospace> can generate cross-chain models from local transactions in different ledgers, realizing arbitrary cross-chain use cases and allowing operators to monitor their applications. Monitoring helps identify outliers and malicious behavior, which can enable programmatically stopping attacks (“a circuit breaker”), including bridge hacks. We conduct a detailed evaluation of our system, where we implement a cross-chain bridge use case. Our experimental results show that <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Hephaestus</monospace> can process 600 cross-chain transactions in less than 5.5 s in an environment with two blockchains using sublinear storage, paving the way for more resilient bridge designs.
ChatGPT: More Human-Like Than Computer-Like, but Not Necessarily in a Good Way

Large language models have been shown to be useful in multiple domains including conversational agents, education, and explainable AI. ChatGPT is a large language model developed by OpenAI as a conversational agent. ChatGPT was trained on data generated by humans and by receiving human feedback. This training process results in a bias toward humans’ traits and preferences. In this paper, we stress multiple biases of ChatGPT, and show that its responses demonstrate many human traits. We begin by showing a very high correlation between the frequency of digits generated by ChatGPT and humans’ favorite numbers, with the most frequent digit generated by ChatGPT, matching humans’ most favorable number, 7. We continue by showing that ChatGPT’s responses in several social experiments are much closer to those of humans’ than to those of fully rational agents. Finally, we show that several cognitive biases, known in humans, are also present in ChatGPT’s responses.