OpenRaiser/Intern-Atlas huggingface dataset:
Exploring patient trust in clinical advice from AI-driven LLMs like ChatGPT for self-diagnosis
Venue: arXiv cs.HC 2024
Year: 2024

Trustworthy clinical advice is crucial but burdensome when seeking health support from professionals. Inaccessibility and financial burdens present obstacles to obtaining professional clinical advice, even when healthcare is available. Consequently, individuals often resort to self-diagnosis, utilizing medical materials to validate the health conditions of their families and friends. However, the convenient method of self-diagnosis requires a commitment to learning and is often not effective, presenting risks when individuals seek self-care approaches or treatment strategies without professional guidance. Artificial Intelligence (AI), supported by Large Language Models (LLM), may become a powerful yet risky self-diagnosis tool for clinical advice due to the hallucination of LLM, where it produces inaccurate yet deceiving information. Thus, can we trust the clinical advice from AI-driven LLMs like ChatGPT like ChatGPT4 for self-diagnosis? We examined this issue through a think-aloud observation: a patient uses GPT4 for self-diagnosis and clinical advice while a doctor assesses ChatGPT responses with their own expertise. After that, we conducted a semi-structured interview with the patient to understand their trust in AI-driven LLMs for clinical advice. we have concluded that the confounding factors influencing a patient's trust revolve around their competency-evaluation. Essentially, trust is equated with efficacy, which is determined by whether decisions made based on the AI agent's clinical advice and suggestion will effectively achieve the patient health goals. Patients tend to trust doctors more than AI agents due to this strategy, believing that educated, authorized doctors can provide effective medical guidance. This competency-based trust also explains why patients often perceive more experienced doctors as more trustworthy compared to less experienced ones.
Point and Instruct: Enabling Precise Image Editing by Unifying Direct Manipulation and Text Instructions
Venue: arXiv cs.AI 2024
Year: 2024

Machine learning has enabled the development of powerful systems capable of editing images from natural language instructions. However, in many common scenarios it is difficult for users to specify precise image transformations with text alone. For example, in an image with several dogs, it is difficult to select a particular dog and move it to a precise location. Doing this with text alone would require a complex prompt that disambiguates the target dog and describes the destination. However, direct manipulation is well suited to visual tasks like selecting objects and specifying locations. We introduce Point and Instruct, a system for seamlessly combining familiar direct manipulation and textual instructions to enable precise image manipulation. With our system, a user can visually mark objects and locations, and reference them in textual instructions. This allows users to benefit from both the visual descriptiveness of natural language and the spatial precision of direct manipulation.
Sentinels of the Stream: Unleashing Large Language Models for Dynamic Packet Classification in Software Defined Networks -- Position Paper
Venue: arXiv cs.CR 2024
Year: 2024

With the release of OpenAI's ChatGPT, the field of large language models (LLM) saw an increase of academic interest in GPT based chat assistants. In the next few months multiple accesible large language models were released that included Meta's LLama models and Mistral AI's Mistral and Mixtral MoE models. These models are available openly for a wide array of purposes with a wide spectrum of licenses. These LLMs have found their use in a different number of fields like code development, SQL generation etc. In this work we propose our plan to explore the applicability of large language model in the domain of network security. We plan to create Sentinel, a LLM, to analyse network packet contents and pass a judgment on it's threat level. This work is a preliminary report that will lay our plan for our future endeavors.
Volitional Control of the Paretic Hand Post-Stroke Increases Finger Stiffness and Resistance to Robot-Assisted Movement
Venue: arXiv cs.RO 2024
Year: 2024

Increased effort during use of the paretic arm and hand can provoke involuntary abnormal synergy patterns and amplify stiffness effects of muscle tone for individuals after stroke, which can add difficulty for user-controlled devices to assist hand movement during functional tasks. We study how volitional effort, exerted in an attempt to open or close the hand, affects resistance to robot-assisted movement at the finger level. We perform experiments with three chronic stroke survivors to measure changes in stiffness when the user is actively exerting effort to activate ipsilateral EMG-controlled robot-assisted hand movements, compared with when the fingers are passively stretched, as well as overall effects from sustained active engagement and use. Our results suggest that active engagement of the upper extremity increases muscle tone in the finger to a much greater degree than through passive-stretch or sustained exertion over time. Potential design implications of this work suggest that developers should anticipate higher levels of finger stiffness when relying on user-driven ipsilateral control methods for assistive or rehabilitative devices for stroke.
Multi-Attribute Vision Transformers are Efficient and Robust Learners
Venue: arXiv cs.CV 2024
Year: 2024

Since their inception, Vision Transformers (ViTs) have emerged as a compelling alternative to Convolutional Neural Networks (CNNs) across a wide spectrum of tasks. ViTs exhibit notable characteristics, including global attention, resilience against occlusions, and adaptability to distribution shifts. One underexplored aspect of ViTs is their potential for multi-attribute learning, referring to their ability to simultaneously grasp multiple attribute-related tasks. In this paper, we delve into the multi-attribute learning capability of ViTs, presenting a straightforward yet effective strategy for training various attributes through a single ViT network as distinct tasks. We assess the resilience of multi-attribute ViTs against adversarial attacks and compare their performance against ViTs designed for single attributes. Moreover, we further evaluate the robustness of multi-attribute ViTs against a recent transformer based attack called Patch-Fool. Our empirical findings on the CelebA dataset provide validation for our assertion. Our code is available at https://github.com/hananshafi/MTL-ViT
Enhancing Programming Error Messages in Real Time with Generative AI
Venue: arXiv cs.HC 2024
Year: 2024

Generative AI is changing the way that many disciplines are taught, including computer science. Researchers have shown that generative AI tools are capable of solving programming problems, writing extensive blocks of code, and explaining complex code in simple terms. Particular promise has been shown in using generative AI to enhance programming error messages. Both students and instructors have complained for decades that these messages are often cryptic and difficult to understand. Yet recent work has shown that students make fewer repeated errors when enhanced via GPT-4. We extend this work by implementing feedback from ChatGPT for all programs submitted to our automated assessment tool, Athene, providing help for compiler, run-time, and logic errors. Our results indicate that adding generative AI to an automated assessment tool does not necessarily make it better and that design of the interface matters greatly to the usability of the feedback that GPT-4 provided.
Grounding Data Science Code Generation with Input-Output Specifications
Venue: arXiv cs.LG 2024
Year: 2024

Large language models (LLMs) have recently demonstrated a remarkable ability to generate code from natural language (NL) prompts. However, in the real world, NL is often too ambiguous to capture the true intent behind programming problems, requiring additional input-output (I/O) specifications. Unfortunately, LLMs can have difficulty aligning their outputs with both the NL prompt and the I/O specification. In this paper, we give a way to mitigate this issue in the context of data science programming, where tasks require explicit I/O specifications for clarity. Specifically, we propose GIFT4Code, a novel approach for the instruction fine-tuning of LLMs with respect to I/O specifications. Our method leverages synthetic data produced by the LLM itself and utilizes execution-derived feedback as a key learning signal. This feedback, in the form of program I/O specifications, is provided to the LLM to facilitate instruction fine-tuning. We evaluated our approach on two challenging data science benchmarks, Arcade and DS-1000. The results demonstrate a significant improvement in the LLM's ability to generate code that is not only executable but also accurately aligned with user specifications, substantially improving the quality of code generation for complex data science tasks.
Text-centric Alignment for Multi-Modality Learning
Venue: arXiv cs.LG 2024
Year: 2024

This research paper addresses the challenge of modality mismatch in multimodal learning, where the modalities available during inference differ from those available at training. We propose the Text-centric Alignment for Multi-Modality Learning (TAMML) approach, an innovative method that utilizes Large Language Models (LLMs) with in-context learning and foundation models to enhance the generalizability of multimodal systems under these conditions. By leveraging the unique properties of text as a unified semantic space, TAMML demonstrates significant improvements in handling unseen, diverse, and unpredictable modality combinations. TAMML not only adapts to varying modalities but also maintains robust performance, showcasing the potential of foundation models in overcoming the limitations of traditional fixed-modality frameworks in embedding representations. This study contributes to the field by offering a flexible, effective solution for real-world applications where modality availability is dynamic and uncertain.
Out-of-Distribution Detection and Data Drift Monitoring using Statistical Process Control
Venue: arXiv cs.AI 2024
Year: 2024

Background: Machine learning (ML) methods often fail with data that deviates from their training distribution. This is a significant concern for ML-enabled devices in clinical settings, where data drift may cause unexpected performance that jeopardizes patient safety. Method: We propose a ML-enabled Statistical Process Control (SPC) framework for out-of-distribution (OOD) detection and drift monitoring. SPC is advantageous as it visually and statistically highlights deviations from the expected distribution. To demonstrate the utility of the proposed framework for monitoring data drift in radiological images, we investigated different design choices, including methods for extracting feature representations, drift quantification, and SPC parameter selection. Results: We demonstrate the effectiveness of our framework for two tasks: 1) differentiating axial vs. non-axial computed tomography (CT) images and 2) separating chest x-ray (CXR) from other modalities. For both tasks, we achieved high accuracy in detecting OOD inputs, with 0.913 in CT and 0.995 in CXR, and sensitivity of 0.980 in CT and 0.984 in CXR. Our framework was also adept at monitoring data streams and identifying the time a drift occurred. In a simulation with 100 daily CXR cases, we detected a drift in OOD input percentage from 0-1% to 3-5% within two days, maintaining a low false-positive rate. Through additional experimental results, we demonstrate the framework's data-agnostic nature and independence from the underlying model's structure. Conclusion: We propose a framework for OOD detection and drift monitoring that is agnostic to data, modality, and model. The framework is customizable and can be adapted for specific applications.
On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks
Venue: arXiv cs.AI 2024
Year: 2024

There has been considerable divergence of opinion on the reasoning abilities of Large Language Models (LLMs). While the initial optimism that reasoning might emerge automatically with scale has been tempered thanks to a slew of counterexamples--ranging from multiplication to simple planning--there persists a wide spread belief that LLMs can self-critique and improve their own solutions in an iterative fashion. This belief seemingly rests on the assumption that verification of correctness should be easier than generation--a rather classical argument from computational complexity--which should be irrelevant to LLMs to the extent that what they are doing is approximate retrieval. In this paper, we set out to systematically investigate the effectiveness of iterative prompting in the context of reasoning and planning. We present a principled empirical study of the performance of GPT-4 in three domains: Game of 24, Graph Coloring, and STRIPS planning. We experiment both with the model critiquing its own answers and with an external correct reasoner verifying proposed solutions. In each case, we analyze whether the content of criticisms actually affects bottom line performance, and whether we can ablate elements of the augmented system without losing performance. We observe significant performance collapse with self-critique and significant performance gains with sound external verification. We also note that merely re-prompting with a sound verifier maintains most of the benefits of more involved setups.
Unmasking honey adulteration : a breakthrough in quality assurance through cutting-edge convolutional neural network analysis of thermal images
Venue: arXiv cs.CV 2024
Year: 2024

Honey, a natural product generated from organic sources, is widely recognized for its revered reputation. Nevertheless, honey is susceptible to adulteration, a situation that has substantial consequences for both the well-being of the general population and the financial well-being of a country. Conventional approaches for detecting honey adulteration are often associated with extensive time requirements and restricted sensitivity. This paper presents a novel approach to address the aforementioned issue by employing Convolutional Neural Networks (CNNs) for the classification of honey samples based on thermal images. The use of thermal imaging technique offers a significant advantage in detecting adulterants, as it can reveal differences in temperature in honey samples caused by variations in sugar composition, moisture levels, and other substances used for adulteration. To establish a meticulous approach to categorizing honey, a thorough dataset comprising thermal images of authentic and tainted honey samples was collected. Several state-of-the-art Convolutional Neural Network (CNN) models were trained and optimized using the dataset that was gathered. Within this set of models, there exist pre-trained models such as InceptionV3, Xception, VGG19, and ResNet that have exhibited exceptional performance, achieving classification accuracies ranging from 88% to 98%. Furthermore, we have implemented a more streamlined and less complex convolutional neural network (CNN) model, outperforming comparable models with an outstanding accuracy rate of 99%. This simplification offers not only the sole advantage of the model, but it also concurrently offers a more efficient solution in terms of resources and time. This approach offers a viable way to implement quality control measures in the honey business, so guaranteeing the genuineness and safety of this valuable organic commodity.
Randomized Algorithms for Symmetric Nonnegative Matrix Factorization
Venue: arXiv cs.LG 2024
Year: 2024

Symmetric Nonnegative Matrix Factorization (SymNMF) is a technique in data analysis and machine learning that approximates a symmetric matrix with a product of a nonnegative, low-rank matrix and its transpose. To design faster and more scalable algorithms for SymNMF we develop two randomized algorithms for its computation. The first algorithm uses randomized matrix sketching to compute an initial low-rank approximation to the input matrix and proceeds to rapidly compute a SymNMF of the approximation. The second algorithm uses randomized leverage score sampling to approximately solve constrained least squares problems. Many successful methods for SymNMF rely on (approximately) solving sequences of constrained least squares problems. We prove theoretically that leverage score sampling can approximately solve nonnegative least squares problems to a chosen accuracy with high probability. Additionally, we prove sampling complexity results for previously proposed hybrid sampling techniques which deterministically include high leverage score rows. This hybrid scheme is crucial for obtaining speeds ups in practice. Finally we demonstrate that both methods work well in practice by applying them to graph clustering tasks on large real world data sets. These experiments show that our methods approximately maintain solution quality and achieve significant speed ups for both large dense and large sparse problems.
Epistemic Exploration for Generalizable Planning and Learning in Non-Stationary Settings
Venue: arXiv cs.AI 2024
Year: 2024

This paper introduces a new approach for continual planning and model learning in relational, non-stationary stochastic environments. Such capabilities are essential for the deployment of sequential decision-making systems in the uncertain and constantly evolving real world. Working in such practical settings with unknown (and non-stationary) transition systems and changing tasks, the proposed framework models gaps in the agent's current state of knowledge and uses them to conduct focused, investigative explorations. Data collected using these explorations is used for learning generalizable probabilistic models for solving the current task despite continual changes in the environment dynamics. Empirical evaluations on several non-stationary benchmark domains show that this approach significantly outperforms planning and RL baselines in terms of sample complexity. Theoretical results show that the system exhibits desirable convergence properties when stationarity holds.
A Projection-Based Time-Segmented Reduced Order Model for Fluid-Structure Interactions
Venue: arXiv cs.CE 2024
Year: 2024

In this paper, a type of novel projection-based, time-segmented reduced order model (ROM) is proposed for dynamic fluid-structure interaction (FSI) problems based upon the arbitrary Lagrangian--Eulerian (ALE)-finite element method (FEM) in a monolithic frame, where spatially, each variable is separated from others in terms of their attribution (fluid/structure), category (velocity/pressure) and component (horizontal/vertical) while temporally, the proper orthogonal decomposition (POD) bases are constructed in some deliberately partitioned time segments tailored through extensive numerical trials. By the combination of spatial and temporal decompositions, the developed ROM approach enables prolonged simulations under prescribed accuracy thresholds. Numerical experiments are carried out to compare numerical performances of the proposed ROM with corresponding full-order model (FOM) by solving a two-dimensional FSI benchmark problem that involves a vibrating elastic beam in the fluid, where the performance of offline ROM on perturbed physical parameters in the online phase is investigated as well. Extensive numerical results demonstrate that the proposed ROM has a comparable accuracy to while much higher efficiency than the FOM. The developed ROM approach is dimension-independent and can be seamlessly extended to solve high dimensional FSI problems.
Learning time-dependent PDE via graph neural networks and deep operator network for robust accuracy on irregular grids
Venue: arXiv cs.LG 2024
Year: 2024

Scientific computing using deep learning has seen significant advancements in recent years. There has been growing interest in models that learn the operator from the parameters of a partial differential equation (PDE) to the corresponding solutions. Deep Operator Network (DeepONet) and Fourier Neural operator, among other models, have been designed with structures suitable for handling functions as inputs and outputs, enabling real-time predictions as surrogate models for solution operators. There has also been significant progress in the research on surrogate models based on graph neural networks (GNNs), specifically targeting the dynamics in time-dependent PDEs. In this paper, we propose GraphDeepONet, an autoregressive model based on GNNs, to effectively adapt DeepONet, which is well-known for successful operator learning. GraphDeepONet exhibits robust accuracy in predicting solutions compared to existing GNN-based PDE solver models. It maintains consistent performance even on irregular grids, leveraging the advantages inherited from DeepONet and enabling predictions on arbitrary grids. Additionally, unlike traditional DeepONet and its variants, GraphDeepONet enables time extrapolation for time-dependent PDE solutions. We also provide theoretical analysis of the universal approximation capability of GraphDeepONet in approximating continuous operators across arbitrary time intervals.
BBSEA: An Exploration of Brain-Body Synchronization for Embodied Agents
Venue: arXiv cs.RO 2024
Year: 2024

Embodied agents capable of complex physical skills can improve productivity, elevate life quality, and reshape human-machine collaboration. We aim at autonomous training of embodied agents for various tasks involving mainly large foundation models. It is believed that these models could act as a brain for embodied agents; however, existing methods heavily rely on humans for task proposal and scene customization, limiting the learning autonomy, training efficiency, and generalization of the learned policies. In contrast, we introduce a brain-body synchronization ({\it BBSEA}) scheme to promote embodied learning in unknown environments without human involvement. The proposed combines the wisdom of foundation models (``brain'') with the physical capabilities of embodied agents (``body''). Specifically, it leverages the ``brain'' to propose learnable physical tasks and success metrics, enabling the ``body'' to automatically acquire various skills by continuously interacting with the scene. We carry out an exploration of the proposed autonomous learning scheme in a table-top setting, and we demonstrate that the proposed synchronization can generate diverse tasks and develop multi-task policies with promising adaptability to new tasks and configurations. We will release our data, code, and trained models to facilitate future studies in building autonomously learning agents with large foundation models in more complex scenarios. More visualizations are available at \href{https://bbsea-embodied-ai.github.io}{https://bbsea-embodied-ai.github.io}