Miracle-dz/UniFAR-FTUs huggingface dataset:
Dietary Assessment with Multimodal ChatGPT: A Systematic Analysis

Conventional approaches to dietary assessment are primarily grounded in self-reporting methods or structured interviews conducted under the supervision of dietitians. These methods, however, are often subjective, potentially inaccurate, and time-intensive. Although artificial intelligence (AI)-based solutions have been devised to automate the dietary assessment process, these prior AI methodologies encounter challenges in their ability to generalize across a diverse range of food types, dietary behaviors, and cultural contexts. This results in AI applications in the dietary field that possess a narrow specialization and limited accuracy. Recently, the emergence of multimodal foundation models such as GPT-4V powering the latest ChatGPT has exhibited transformative potential across a wide range of tasks (e.g., Scene understanding and image captioning) in numerous research domains. These models have demonstrated remarkable generalist intelligence and accuracy, capable of processing various data modalities. In this study, we explore the application of multimodal ChatGPT within the realm of dietary assessment. Our findings reveal that GPT-4V excels in food detection under challenging conditions with accuracy up to 87.5% without any fine-tuning or adaptation using food-specific datasets. By guiding the model with specific language prompts (e.g., African cuisine), it shifts from recognizing common staples like rice and bread to accurately identifying regional dishes like banku and ugali. Another GPT-4V's standout feature is its contextual awareness. GPT-4V can leverage surrounding objects as scale references to deduce the portion sizes of food items, further enhancing its accuracy in translating food weight into nutritional content. This alignment with the USDA National Nutrient Database underscores GPT-4V's potential to advance nutritional science and dietary assessment techniques.
CAT: A Causally Graph Attention Network for Trimming Heterophilic Graph

Local Attention-guided Message Passing Mechanism (LAMP) adopted in Graph Attention Networks (GATs) is designed to adaptively learn the importance of neighboring nodes for better local aggregation on the graph, which can bring the representations of similar neighbors closer effectively, thus showing stronger discrimination ability. However, existing GATs suffer from a significant discrimination ability decline in heterophilic graphs because the high proportion of dissimilar neighbors can weaken the self-attention of the central node, jointly resulting in the deviation of the central node from similar nodes in the representation space. This kind of effect generated by neighboring nodes is called the Distraction Effect (DE) in this paper. To estimate and weaken the DE of neighboring nodes, we propose a Causally graph Attention network for Trimming heterophilic graph (CAT). To estimate the DE, since the DE are generated through two paths (grab the attention assigned to neighbors and reduce the self-attention of the central node), we use Total Effect to model DE, which is a kind of causal estimand and can be estimated from intervened data; To weaken the DE, we identify the neighbors with the highest DE (we call them Distraction Neighbors) and remove them. We adopt three representative GATs as the base model within the proposed CAT framework and conduct experiments on seven heterophilic datasets in three different sizes. Comparative experiments show that CAT can improve the node classification accuracy of all base GAT models. Ablation experiments and visualization further validate the enhancement of discrimination ability brought by CAT. The source code is available at https://github.com/GeoX-Lab/CAT.
DiffusionLight: Light Probes for Free by Painting a Chrome Ball

We present a simple yet effective technique to estimate lighting in a single input image. Current techniques rely heavily on HDR panorama datasets to train neural networks to regress an input with limited field-of-view to a full environment map. However, these approaches often struggle with real-world, uncontrolled settings due to the limited diversity and size of their datasets. To address this problem, we leverage diffusion models trained on billions of standard images to render a chrome ball into the input image. Despite its simplicity, this task remains challenging: the diffusion models often insert incorrect or inconsistent objects and cannot readily generate images in HDR format. Our research uncovers a surprising relationship between the appearance of chrome balls and the initial diffusion noise map, which we utilize to consistently generate high-quality chrome balls. We further fine-tune an LDR difusion model (Stable Diffusion XL) with LoRA, enabling it to perform exposure bracketing for HDR light estimation. Our method produces convincing light estimates across diverse settings and demonstrates superior generalization to in-the-wild scenarios.
Design, construction and evaluation of emotional multimodal pathological speech database

The lack of an available emotion pathology database is one of the key obstacles in studying the emotion expression status of patients with dysarthria. The first Chinese multimodal emotional pathological speech database containing multi-perspective information is constructed in this paper. It includes 29 controls and 39 patients with different degrees of motor dysarthria, expressing happy, sad, angry and neutral emotions. All emotional speech was labeled for intelligibility, types and discrete dimensional emotions by developed WeChat mini-program. The subjective analysis justifies from emotion discrimination accuracy, speech intelligibility, valence-arousal spatial distribution, and correlation between SCL-90 and disease severity. The automatic recognition tested on speech and glottal data, with average accuracy of 78% for controls and 60% for patients in audio, while 51% for controls and 38% for patients in glottal data, indicating an influence of the disease on emotional expression.
Connectivity Oracles for Predictable Vertex Failures

The problem of designing connectivity oracles supporting vertex failures is one of the basic data structures problems for undirected graphs. It is already well understood: previous works [Duan--Pettie STOC'10; Long--Saranurak FOCS'22] achieve query time linear in the number of failed vertices, and it is conditionally optimal as long as we require preprocessing time polynomial in the size of the graph and update time polynomial in the number of failed vertices. We revisit this problem in the paradigm of algorithms with predictions: we ask if the query time can be improved if the set of failed vertices can be predicted beforehand up to a small number of errors. More specifically, we design a data structure that, given a graph $G=(V,E)$ and a set of vertices predicted to fail $\widehat{D} \subseteq V$ of size $d=|\widehat{D}|$, preprocesses it in time $\tilde{O}(d|E|)$ and then can receive an update given as the symmetric difference between the predicted and the actual set of failed vertices $\widehat{D} \triangle D = (\widehat{D} \setminus D) \cup (D \setminus \widehat{D})$ of size $\eta = |\widehat{D} \triangle D|$, process it in time $\tilde{O}(\eta^4)$, and after that answer connectivity queries in $G \setminus D$ in time $O(\eta)$. Viewed from another perspective, our data structure provides an improvement over the state of the art for the \emph{fully dynamic subgraph connectivity problem} in the \emph{sensitivity setting} [Henzinger--Neumann ESA'16]. We argue that the preprocessing time and query time of our data structure are conditionally optimal under standard fine-grained complexity assumptions.
Unveiling Diversity: Empowering OSS Project Leaders with Community Diversity and Turnover Dashboards

Managing open-source software (OSS) projects requires managing communities of contributors. In particular, it is essential for project leaders to understand their community's diversity and turnover. We present CommunityTapestry, a dynamic real-time community dashboard, which presents key diversity and turnover signals that we identified from the literature and through participatory design sessions with stakeholders. We evaluated CommunityTapestry with an OSS project's contributors and Project Management Committee members, who explored the dashboard using their own project data. Our study results demonstrate that CommunityTapestry increased participants' awareness of their community composition and the diversity and turnover rates in the project. It helped them identify areas of improvement and gave them actionable information.
Topic Bias in Emotion Classification

Emotion corpora are typically sampled based on keyword/hashtag search or by asking study participants to generate textual instances. In any case, these corpora are not uniform samples representing the entirety of a domain. We hypothesize that this practice of data acquision leads to unrealistic correlations between overrepresented topics in these corpora that harm the generalizability of models. Such topic bias could lead to wrong predictions for instances like “I organized the service for my aunt’s funeral. ” when funeral events are overpresented for instances labeled with sadness, despite the emotion of pride being more appropriate here. In this paper, we study this topic bias both from the data and the modeling perspective. We first label a set of emotion corpora automatically via topic modeling and show that emotions in fact correlate with specific topics. Further, we see that emotion classifiers are confounded by such topics. Finally, we show that the established debiasing method of adversarial correction via gradient reversal mitigates the issue. Our work points out issues with existing emotion corpora and that more representative resources are required for fair evaluation of models predicting affective concepts from text.
Labels Need Prompts Too: Mask Matching for Natural Language Understanding Tasks

Textual label names (descriptions) are typically semantically rich in many natural language understanding (NLU) tasks. In this paper, we incorporate the prompting methodology, which is widely used to enrich model input, into the label side for the first time. Specifically, we propose a Mask Matching method, which equips an input with a prompt and its label with another, and then makes predictions by matching their mask representations. We evaluate our method extensively on 8 NLU tasks with 14 datasets. The experimental results show that Mask Matching significantly outperforms its counterparts of fine-tuning and conventional prompt-tuning, setting up state-of-the-art performances in several datasets. Mask Matching is particularly good at handling NLU tasks with large label counts and informative label names. As pioneering efforts that investigate the label-side prompt, we also discuss open issues for future study.
ExROPPP: Fast, Accurate and Spin-Pure Calculation of the Electronically Excited States of Organic Hydrocarbon Radicals

Recent years have seen an explosion of interest in organic radicals due to their promise for highly efficient organic light-emitting diodes (OLEDs) and molecular qubits. However, accurately and inexpensively computing their electronic structure has been challenging, especially for excited states, due to the spin-contamination problem. Furthermore, while alternacy or `pseudoparity' rules have guided the interpretation and prediction of the excited states of closed-shell hydrocarbons since the 1950s, similarly general rules for hydrocarbon radicals have not to our knowledge yet been found. In this article we present solutions to both of these challenges. We firstly combine the extended configuration interaction singles (XCIS) method with Pariser-Parr-Pople (PPP) theory to obtain a method which we call ExROPPP (Extended Restricted Open-shell PPP theory). We find that ExROPPP computes spin-pure excited states of hydrocarbon radicals with comparable accuracy to experiment as high-level GMC-QDPT calculations but at a computational cost that is at least two orders of magnitude lower. We then use ExROPPP to derive widely-applicable rules for the spectra of alternant hydrocarbon radicals which are completely consistent with our computed results. These findings pave the way for the highly accurate and efficient computation and prediction of the excited states of organic radicals.
Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis

Building general-purpose robots that can operate seamlessly, in any environment, with any object, and utilizing various skills to complete diverse tasks has been a long-standing goal in Artificial Intelligence. Unfortunately, however, most existing robotic systems have been constrained - having been designed for specific tasks, trained on specific datasets, and deployed within specific environments. These systems usually require extensively-labeled data, rely on task-specific models, have numerous generalization issues when deployed in real-world scenarios, and struggle to remain robust to distribution shifts. Motivated by the impressive open-set performance and content generation capabilities of web-scale, large-capacity pre-trained models (i.e., foundation models) in research fields such as Natural Language Processing (NLP) and Computer Vision (CV), we devote this survey to exploring (i) how these existing foundation models from NLP and CV can be applied to the field of robotics, and also exploring (ii) what a robotics-specific foundation model would look like. We begin by providing an overview of what constitutes a conventional robotic system and the fundamental barriers to making it universally applicable. Next, we establish a taxonomy to discuss current work exploring ways to leverage existing foundation models for robotics and develop ones catered to robotics. Finally, we discuss key challenges and promising future directions in using foundation models for enabling general-purpose robotic systems. We encourage readers to view our living GitHub repository of resources, including papers reviewed in this survey as well as related projects and repositories for developing foundation models for robotics.
Design Space Exploration of Low-Bit Quantized Neural Networks for Visual Place Recognition

Visual Place Recognition (VPR) is a critical task for performing global re-localization in visual perception systems. It requires the ability to accurately recognize a previously visited location under variations such as illumination, occlusion, appearance and viewpoint. In the case of robotic systems and augmented reality, the target devices for deployment are battery powered edge devices. Therefore whilst the accuracy of VPR methods is important so too is memory consumption and latency. Recently new works have focused on the recall@1 metric as a performance measure with limited focus on resource utilization. This has resulted in methods that use deep learning models too large to deploy on low powered edge devices. We hypothesize that these large models are highly over-parameterized and can be optimized to satisfy the constraints of a low powered embedded system whilst maintaining high recall performance. Our work studies the impact of compact convolutional network architecture design in combination with full-precision and mixed-precision post-training quantization on VPR performance. Importantly we not only measure performance via the recall@1 score but also measure memory consumption and latency. We characterize the design implications on memory, latency and recall scores and provide a number of design recommendations for VPR systems under these resource limitations.
Scene 3-D Reconstruction System in Scattering Medium

The research on neural radiance fields for new view synthesis has experienced explosive growth with the development of new models and extensions. The NERF algorithm, suitable for underwater scenes or scattering media, is also evolving. Existing underwater 3D reconstruction systems still face challenges such as extensive training time and low rendering efficiency. This paper proposes an improved underwater 3D reconstruction system to address these issues and achieve rapid, high-quality 3D reconstruction.To begin with, we enhance underwater videos captured by a monocular camera to correct the poor image quality caused by the physical properties of the water medium while ensuring consistency in enhancement across adjacent frames. Subsequently, we perform keyframe selection on the video frames to optimize resource utilization and eliminate the impact of dynamic objects on the reconstruction results. The selected keyframes, after pose estimation using COLMAP, undergo a three-dimensional reconstruction improvement process using neural radiance fields based on multi-resolution hash coding for model construction and rendering.
A Comprehensive Approach to Ensuring Quality in Spreadsheet-Based Metadata

While scientists increasingly recognize the importance of metadata in describing their data, spreadsheets remain the preferred tool for supplying this information despite their limitations in ensuring compliance and quality. Various tools have been developed to address these limitations, but they suffer from their own shortcomings, such as steep learning curves and limited customization. In this paper, we describe an end-to-end approach that supports spreadsheet-based entry of metadata while providing rigorous compliance and quality control. Our approach employs several key strategies, including customizable templates for defining metadata, integral support for the use of controlled terminologies when defining these templates, and an interactive Web-based tool that allows users to rapidly identify and fix errors in the spreadsheet-based metadata they supply. We demonstrate how this approach is being deployed in a biomedical consortium to define and collect metadata about scientific experiments.
PerfectTailor: Scale-Preserving 2D Pattern Adjustment Driven by 3D Garment Editing

We address the problem of modifying a given well-designed 2D sewing pattern to accommodate garment edits in the 3D space. Existing methods usually adjust the sewing pattern by applying uniform flattening to the 3D garment. The problems are twofold: first, it ignores local scaling of the 2D sewing pattern such as shrinking ribs of cuffs; second, it does not respect the implicit design rules and conventions of the industry, such as the use of straight edges for simplicity and precision in sewing. To address those problems, we present a pattern adjustment method that considers the non-uniform local scaling of the 2D sewing pattern by utilizing the intrinsic scale matrix. In addition, we preserve the original boundary shape by an as-original-as-possible geometric constraint when desirable. We build a prototype with a set of commonly used alteration operations and showcase the capability of our method via a number of alteration examples throughout the paper.
Integrating Particle Flavor into Deep Learning Models for Hadronization

Hadronization models used in event generators are physics-inspired functions with many tunable parameters. Since we do not understand hadronization from first principles, there have been multiple proposals to improve the accuracy of hadronization models by utilizing more flexible parameterizations based on neural networks. These recent proposals have focused on the kinematic properties of hadrons, but a full model must also include particle flavor. In this paper, we show how to build a deep learning-based hadronization model that includes both kinematic (continuous) and flavor (discrete) degrees of freedom. Our approach is based on Generative Adversarial Networks and we show the performance within the context of the cluster hadronization model within the Herwig event generator.
Enabling End-to-End Secure Federated Learning in Biomedical Research on Heterogeneous Computing Environments with APPFLx

Facilitating large-scale, cross-institutional collaboration in biomedical machine learning projects requires a trustworthy and resilient federated learning (FL) environment to ensure that sensitive information such as protected health information is kept confidential. In this work, we introduce APPFLx, a low-code FL framework that enables the easy setup, configuration, and running of FL experiments across organizational and administrative boundaries while providing secure end-to-end communication, privacy-preserving functionality, and identity management. APPFLx is completely agnostic to the underlying computational infrastructure of participating clients. We demonstrate the capability of APPFLx as an easy-to-use framework for accelerating biomedical studies across institutions and healthcare systems while maintaining the protection of private medical data in two case studies: (1) predicting participant age from electrocardiogram (ECG) waveforms, and (2) detecting COVID-19 disease from chest radiographs. These experiments were performed securely across heterogeneous compute resources, including a mixture of on-premise high-performance computing and cloud computing, and highlight the role of federated learning in improving model generalizability and performance when aggregating data from multiple healthcare systems. Finally, we demonstrate that APPFLx serves as a convenient and easy-to-use framework for accelerating biomedical studies across institutions and healthcare system while maintaining the protection of private medical data.
Χ-binding Functions for Squares of Bipartite Graphs and Its Subclasses

A class of graphs $\mathcal{G}$ is $\chi$-bounded if there exists a function $f$ such that $\chi(G) \leq f(\omega(G))$ for each graph $G \in \mathcal{G}$, where $\chi(G)$ and $\omega(G)$ are the chromatic and clique number of $G$, respectively. The square of a graph $G$, denoted as $G^2$, is the graph with the same vertex set as $G$ in which two vertices are adjacent when they are at a distance at most two in $G$. In this paper, we study the $\chi$-boundedness of squares of bipartite graphs and its subclasses. Note that the class of squares of graphs, in general, admit a quadratic $\chi$-binding function. Moreover there exist bipartite graphs $B$ for which $\chi\left(B^2\right)$ is $\Omega\left(\frac{\left(\omega\left(B^2\right)\right)^2 }{\log \omega\left(B^2\right)}\right)$. We first ask the following question:"What sub-classes of bipartite graphs have a linear $\chi$-binding function? "We focus on the class of convex bipartite graphs and prove the following result: for any convex bipartite graph $G$, $\chi\left(G^2\right) \leq \frac{3 \omega\left(G^2\right)}{2}$. Our proof also yields a polynomial-time $3/2$-approximation algorithm for coloring squares of convex bipartite graphs. We then introduce a notion called"partite testable properties"for the squares of bipartite graphs. We say that a graph property $P$ is partite testable for the squares of bipartite graphs if for a bipartite graph $G=(A,B,E)$, whenever the induced subgraphs $G^2[A]$ and $G^2[B]$ satisfies the property $P$ then $G^2$ also satisfies the property $P$. Here, we discuss whether some of the well-known graph properties like perfectness, chordality, (anti-hole)-freeness, etc. are partite testable or not. As a consequence, we prove that the squares of biconvex bipartite graphs are perfect.