Reviewerly/exHarmony huggingface dataset:
ROBBIE: Robust Bias Evaluation of Large Generative Language Models

As generative large language models (LLMs) grow more performant and prevalent, we must develop comprehensive enough tools to measure and improve their fairness. Different prompt-based datasets can be used to measure social bias across multiple text domains and demographic axes, meaning that testing LLMs on more datasets can potentially help us characterize their biases more fully, and better ensure equal and equitable treatment of marginalized demographic groups. In this work, our focus is two-fold: (1) Benchmarking: a comparison of 6 different prompt-based bias and toxicity metrics across 12 demographic axes and 5 families of generative LLMs. Out of those 6 metrics, AdvPromptSet and HolisticBiasR are novel datasets proposed in the paper. The comparison of those benchmarks gives us insights about the bias and toxicity of the compared models. Therefore, we explore the frequency of demographic terms in common LLM pre-training corpora and how this may relate to model biases. (2) Mitigation: we conduct a comprehensive study of how well 3 bias/toxicity mitigation techniques perform across our suite of measurements. ROBBIE aims to provide insights for practitioners while deploying a model, emphasizing the need to not only measure potential harms, but also understand how they arise by characterizing the data, mitigate harms once found, and balance any trade-offs. We open-source our analysis code in hopes of encouraging broader measurements of bias in future LLMs.
Cirquo: A Suite For Testing and Debugging Quantum Programs

This paper introduces a process framework for debugging quantum circuits, focusing on three distinct types of circuit blocks: Amplitude Permutation, Phase Modulation, and Amplitude Redistribution circuit blocks. Our research addresses the critical need for specialized debugging approaches tailored to the unique properties of each circuit type. For Amplitude Permutation Circuits, we propose techniques to correct amplitude permutations mimicking classical operations. In phase modulation circuits, our proposed strategy targets the precise calibration of phase alterations essential for quantum computations. The most complex Amplitude Redistribution Circuits demand advanced methods to adjust probability amplitudes. This research bridges a vital gap in current methodologies and lays the groundwork for future advancements in quantum circuit debugging. Our contributions are twofold: We present a comprehensive unit testing tool (Cirquo) and debugging approaches tailored to the unique demands of quantum computing, and we provide empirical evidence of its effectiveness in optimizing quantum circuit performance. This work is a crucial step toward realizing robust quantum computing systems and their applications in various domains.
Richard Feynman's talent for finding things out

This article is a summary of a talk about Richard Feynman, given at a conference Polymaths across the Eras organized in November 2023 by the St Cross Centre for the History and Philosophy of Physics (HAPP) in Oxford. It describes Feynman as an unconventional polymath, primarily concerned with his own understanding of nature, having little regard for previous authorities. His curiosity, respect for experiment, ability to calculate and notable ingenuity led to important developments in quantum mechanics, quantum electrodynamics, and strong interactions. In particle physics, Feynman diagrams have been a mainstay of calculation for the last 75 years. His relish for the spoken word led to renown as an educator and raconteur. His perceptions were influential in a wide range of other scientific fields, including weak interactions, gravity, superconductivity, biology, nanotechnology, algorithmic computation and quantum computers. In all of this, he held to the idea that nature has an intrinsic simplicity and beauty that permit us to find things out, if only we will try hard enough.
How Much Is Hidden in the NAS Benchmarks? Few-Shot Adaptation of a NAS Predictor

Neural architecture search has proven to be a powerful approach to designing and refining neural networks, often boosting their performance and efficiency over manually-designed variations, but comes with computational overhead. While there has been a considerable amount of research focused on lowering the cost of NAS for mainstream tasks, such as image classification, a lot of those improvements stem from the fact that those tasks are well-studied in the broader context. Consequently, applicability of NAS to emerging and under-represented domains is still associated with a relatively high cost and/or uncertainty about the achievable gains. To address this issue, we turn our focus towards the recent growth of publicly available NAS benchmarks in an attempt to extract general NAS knowledge, transferable across different tasks and search spaces. We borrow from the rich field of meta-learning for few-shot adaptation and carefully study applicability of those methods to NAS, with a special focus on the relationship between task-level correlation (domain shift) and predictor transferability; which we deem critical for improving NAS on diverse tasks. In our experiments, we use 6 NAS benchmarks in conjunction, spanning in total 16 NAS settings -- our meta-learning approach not only shows superior (or matching) performance in the cross-validation experiments but also successful extrapolation to a new search space and tasks.
Nonnegative Preserving Projection for Dimensionality Reduction

Nonnegative matrix factorization (NMF) is a method that represent classical data based on matrix decomposition. NMF is more compatible with real-world applications and improves readability because the decomposed elements must be non-negative. The original data, which is very redundant and noisy, is directly decomposed using NMF. It is generally high-dimensional. This will waste a lot of time and the obtained result of decomposition is also not optimal. To obtain a low-dimensional compact representation of observational data, in which the hidden semantic information is uncovered and the partial structure information is simultaneously preserved, a novel dimensionality reduction framework named nonnegative preserving projection (NPP) is proposed. In NPP, we firstly construct an affinity graph for encoding the geometrical information needed to capture the data's local manifold. Then, the projection matrix is introduced into NMF. Meanwhile, we apply a PCA-based constraint to the objective function to ensure that the bulk of the energy from the original data is preserved. We examined the proposed NPP with existing approaches in different picture databases to verify the performance of NPP in dimensionality reduction. According to the findings of the experiments, the procedure is both feasible and effective.
GFN-SR: Symbolic Regression with Generative Flow Networks

Symbolic regression (SR) is an area of interpretable machine learning that aims to identify mathematical expressions, often composed of simple functions, that best fit in a given set of covariates $X$ and response $y$. In recent years, deep symbolic regression (DSR) has emerged as a popular method in the field by leveraging deep reinforcement learning to solve the complicated combinatorial search problem. In this work, we propose an alternative framework (GFN-SR) to approach SR with deep learning. We model the construction of an expression tree as traversing through a directed acyclic graph (DAG) so that GFlowNet can learn a stochastic policy to generate such trees sequentially. Enhanced with an adaptive reward baseline, our method is capable of generating a diverse set of best-fitting expressions. Notably, we observe that GFN-SR outperforms other SR algorithms in noisy data regimes, owing to its ability to learn a distribution of rewards over a space of candidate solutions.
ENHANCING PEDESTRIAN TARGET RECOGNITION IN OPEN COMMUNITY MULTI-SCENE SPACES USING THE YOLO-STP NETWORK

Abstract. Addressing the challenge of quantitatively analyzing and presenting pedestrian elements within open community spaces is of significant importance. Focusing on the indoor scene spaces of open communities, this study introduces the TJ-Person pedestrian target recognition image dataset. Furthermore, we design a deep learning-based community pedestrian activity analysis network model and incorporate various attention mechanisms, such as SA, CA, CBAM, SE, and SK, into the YOLO v5s deep learning target recognition network framework for comparative evaluation of pedestrian target recognition in open communities. Utilizing the optimized YOLO Swin Transformer Person (YOLO-STP) network, precise identification of pedestrian targets across multiple scenarios was achieved. We conducted experimental verification using four typical scenarios within Shanghai's NICE2035 open community as case studies. The results demonstrated that the proposed YOLO-STP community pedestrian activity analysis network model achieved an optimal detection accuracy of up to 98.47%. In all four tested scenarios, the YOLO-STP method consistently exhibited competitive performance. Moreover, in the COCO-2017 open-source dataset testing, the YOLO-STP method outperformed other networks of the same type, showcasing its significant advantages. Overall, the research presented in this study provides a crucial technical foundation for the analysis and recognition of pedestrian targets in future community scenarios.
SequencePAR: Understanding Pedestrian Attributes via A Sequence Generation Paradigm

Current pedestrian attribute recognition (PAR) algorithms are developed based on multi-label or multi-task learning frameworks, which aim to discriminate the attributes using specific classification heads. However, these discriminative models are easily influenced by imbalanced data or noisy samples. Inspired by the success of generative models, we rethink the pedestrian attribute recognition scheme and believe the generative models may perform better on modeling dependencies and complexity between human attributes. In this paper, we propose a novel sequence generation paradigm for pedestrian attribute recognition, termed SequencePAR. It extracts the pedestrian features using a pre-trained CLIP model and embeds the attribute set into query tokens under the guidance of text prompts. Then, a Transformer decoder is proposed to generate the human attributes by incorporating the visual features and attribute query tokens. The masked multi-head attention layer is introduced into the decoder module to prevent the model from remembering the next attribute while making attribute predictions during training. Extensive experiments on multiple widely used pedestrian attribute recognition datasets fully validated the effectiveness of our proposed SequencePAR. The source code and pre-trained models will be released at https://github.com/Event-AHU/OpenPAR.
Towards Learning a Generalist Model for Embodied Navigation

Building a generalist agent that can interact with the world is the intriguing target of AI systems, thus spurring the research for embodied navigation, where an agent is required to navigate according to instructions or respond to queries. Despite the major progress attained, previous works primarily focus on task-specific agents and lack generalizability to unseen scenarios. Recently, LLMs have presented remarkable capabilities across various fields, and provided a promising opportunity for embodied navigation. Drawing on this, we propose the first generalist model for embodied navigation, NaviLLM. It adapts LLMs to embodied navigation by introducing schema-based instruction. The schema-based instruction flexibly casts various tasks into generation problems, thereby unifying a wide range of tasks. This approach allows us to integrate diverse data sources from various datasets into the training, equipping NaviLLM with a wide range of capabilities required by embodied navigation. We conduct extensive experiments to evaluate the performance and generalizability of our model. The experimental results demonstrate that our unified model achieves state-of-the-art performance on CVDN, SOON, and ScanQA. Specifically, it surpasses the previous stats-of-the-art method by a significant margin of 29% in goal progress on CVDN. Moreover, our model also demonstrates strong generalizability and presents impressive results on unseen tasks, e.g., embodied question answering and 3D captioning.
User Persona Mapping Technique for Human Interaction with Augmented and Virtual Reality Environments

An in-depth analysis of user-centered design's function in the context of augmented and virtual reality demonstrates how “User Persona Mapping” can help satisfy the target audience's requirements. As shown by this study, it is crucial to consider user needs and preferences throughout the design phase of augmented and virtual reality (AR/VR) interfaces. Essential to this analysis is the “User Persona Mapping” technique, which entails the construction of detailed user personas based on extensive user research. By synthesizing user attributes, behaviors, and preferences, designers can better meet the expectations of their end users if they first construct a concrete representation of that demographic. Through real-world examples, this article demonstrates how this strategy can be used to improve the usability of augmented and virtual reality interfaces. This study contends that in the future of interactive technology, empathic design will be essential for ensuring that extended and virtual reality experiences are not only innovative but also authentically user-centric and engaging.
Think Twice Before Selection: Federated Evidential Active Learning for Medical Image Analysis with Domain Shifts

Federated learning facilitates the collaborative learning of a global model across multiple distributed medical institutions without centralizing data. Nevertheless, the expensive cost of annotation on local clients remains an obstacle to effectively utilizing local data. To mitigate this issue, federated active learning methods suggest leveraging local and global model predictions to select a relatively small amount of informative local data for annotation. However, existing methods mainly focus on all local data sampled from the same domain, making them unreliable in realistic medical scenarios with domain shifts among different clients. In this paper, we make the first attempt to assess the informativeness of local data derived from diverse domains and propose a novel methodology termed Federated Evidential Active Learning (FEAL) to calibrate the data evaluation under domain shift. Specifically, we introduce a Dirichlet prior distribution in both local and global models to treat the prediction as a distribution over the probability simplex and capture both aleatoric and epistemic uncertainties by using the Dirichlet-based evidential model. Then we employ the epistemic uncertainty to calibrate the aleatoric uncertainty. Afterward, we design a diversity relaxation strategy to reduce data redundancy and maintain data diversity. Extensive experiments and analyses are conducted to show the superiority of FEAL over the state-of-the-art active learning methods and the efficiency of FEAL under the federated active learning framework.
A comparative study between SA and GA in solving MTSP

The multiple traveling salesmen problems (MTSP) is a combinatorial optimization and np-hard problem. In practice, the computational resource required to solve such problems is usually prohibitive, and, in most cases, using heuristic algorithms is the only practical option. This paper implements genetic algorithms (GA) and simulated annealing (SA) to solve the MTSP and does an experimental study based on a benchmark from the TSPLIB instance to compare the performance of two algorithms in reality. The results show that GA can achieve an acceptable solution in a shorter time for any of the MTSP cases and is more accurate when the data size is small. Meanwhile, SA is more robust and achieves a better solution than GA for complex MTSP cases, but it takes more time to converge. Therefore, the result indicates that it is hard to identify which algorithm is comprehensively superior to the other one. However, It also provides an essential reference to developers who want to choose algorithms to solve MTSP in real life, facilitating them to balance the algorithms performance on different metrics they value.
Utility of Rapid Diagnostic Test for the Diagnosis of Malaria in Children: A Cross-sectional Study

Introduction: Malaria is one of the primary causes of paediatric deaths. Rapid and accurate diagnosis, followed by effective treatment, is the main strategy for malaria control. Microscopy has been the gold standard test for diagnosing malaria for over a century. It offers good sensitivity, identifies various species, and measures parasitaemia levels. However, microscopy is time-consuming, labour-intensive, and requires technical expertise. Therefore, there is a need for an easily performed test that is more sensitive and reliable, especially in resource- limited settings. Rapid Diagnostic Tests (RDTs) are a viable alternative due to their simplicity, ease of use, accuracy, and reproducibility. Aim: To compare RDT with microscopy for diagnosis of malaria in children. Materials and Methods: This cross-sectional study was conducted in the Paediatric Ward of Government Stanley Medical College and Hospital, Chennai, Tamil Nadu, India, from September 2020 to February 2021. The study included children under 12 years of age with acute febrile illness, hepatosplenomegaly/ splenomegaly, or anaemia and thrombocytopenia. Thick and thin blood smears, as well as RDT, were performed using venous samples. The collected data were analysed using the Statistical Package for the Social Sciences (SPSS) software version 23.0. To determine significance in categorical data, the Chi-square test/Fisher’s exact test was used. The efficacy was evaluated using the Receiver Operating Characteristic (ROC) curve, which provided sensitivity, specificity, positive predictive value, negative predictive value, and accuracy. Results: Among the 71 participants, 49 (69%) were males and 22 (31%) were females. Malaria presence, as determined by microscopy and RDT, was observed in 20 (28.2%) and 47 (66.2%) children, respectively, out of the total 71 enrolled in this study. The sensitivity, specificity, positive predictive value, negative predictive value, and accuracy of RDT compared to microscopy were 100%, 48%, 44.7%, 100%, and 63.4%, respectively. The Area under the ROC (AUC) curve of RDT for diagnosing malaria was 0.740 (95% CI, 0.628 to 0.852; p-value=0.001). There was a statistically significant association between clinical response to antimalarials in microscopy-negative but RDT-positive subjects (p-value=0.0005). Conclusion: The present study demonstrates that RDT could be used as an alternative to microscopy in the diagnosis of malaria.
GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions

There is growing interest in systems that generate captions for scientific figures. However, assessing these systems’ output poses a significant challenge. Human evaluation requires academic expertise and is costly, while automatic evaluation depends on often low-quality author-written captions. This paper investigates using large language models (LLMs) as a cost-effective, reference-free method for evaluating figure captions. We first constructed SCICAP-EVAL, a human evaluation dataset that contains human judgments for 3,600 scientific figure captions, both original and machine-made, for 600 arXiv figures. We then prompted LLMs like GPT-4 and GPT-3 to score (1-6) each caption based on its potential to aid reader understanding, given relevant context such as figure-mentioning paragraphs. Results show that GPT-4, used as a zero-shot evaluator, outperformed all other models and even surpassed assessments made by computer science undergraduates, achieving a Kendall correlation score of 0.401 with Ph.D. students’ rankings.
MUST&amp;P-SRL: Multi-lingual and Unified Syllabification in Text and Phonetic Domains for Speech Representation Learning

In this paper, we present a methodology for linguistic feature extraction, focusing particularly on automatically syllabifying words in multiple languages, with a design to be compatible with a forced-alignment tool, the Montreal Forced Aligner (MFA). In both the textual and phonetic domains, our method focuses on the extraction of phonetic transcriptions from text, stress marks, and a unified automatic syllabification (in text and phonetic domains). The system was built with open-source components and resources. Through an ablation study, we demonstrate the efficacy of our approach in automatically syllabifying words from several languages (English, French and Spanish). Additionally, we apply the technique to the transcriptions of the CMU ARCTIC dataset, generating valuable annotations available online (https://github.com/noetits/MUST_P-SRL) that are ideal for speech representation learning, speech unit discovery, and disentanglement of speech factors in several speech-related fields.
Improving Contextual Query Rewrite for Conversational AI Agents through User-preference Feedback Learning

Zhongkai Sun, Yingxue Zhou, Jie Hao, Xing Fan, Yanbin Lu, Chengyuan Ma, Wei Shen, Chenlei Guo. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track. 2023.
Enhancing Argument Structure Extraction with Efficient Leverage of Contextual Information

Argument structure extraction (ASE) aims to identify the discourse structure of arguments within documents. Previous research has demonstrated that contextual information is crucial for developing an effective ASE model. However, we observe that merely concatenating sentences in a contextual window does not fully utilize contextual information and can sometimes lead to excessive attention on less informative sentences. To tackle this challenge, we propose an Efficient Context-aware ASE model (ECASE) that fully exploits contextual information by enhancing modeling capacity and augmenting training data. Specifically, we introduce a sequence-attention module and distance-weighted similarity loss to aggregate contextual information and argumentative information. Additionally, we augment the training data by randomly masking discourse markers and sentences, which reduces the model’s reliance on specific words or less informative sentences. Our experiments on five datasets from various domains demonstrate that our model achieves state-of-the-art performance. Furthermore, ablation studies confirm the effectiveness of each module in our model.
Towards large language model-based personal agents in the enterprise: Current trends and open problems

There is an emerging trend to use large language models (LLMs) to reason about complex goals and orchestrate a set of pluggable tools or APIs to accomplish a goal. This functionality could, among other use cases, be used to build personal assistants for knowledge workers. While there are impressive demos of LLMs being used as autonomous agents or for tool composition, these solutions are not ready mission-critical enterprise settings. For example, they are brittle to input changes, and can produce inconsistent results for the same inputs. These use cases have many open problems in an exciting area of NLP research, such as trust and explainability, consistency and reproducibility, adherence to guardrails and policies, best practices for composable tool design, and the need for new metrics and benchmarks. This vision paper illustrates some examples of LLM-based autonomous agents that reason and compose tools, highlights cases where they fail, surveys some of the recent efforts in this space, and lays out the research challenges to make these solutions viable for enterprises.