OpenRaiser/Intern-Atlas huggingface dataset:
"The struggle is a part of the experience": Engaging Discontents in the Design of Family Meal Technologies
Venue: arXiv cs.HC 2024
Year: 2024

Meals are a central (and messy) part of family life. Previous design framings for mealtime technologies have focused on supporting dietary needs or social and celebratory interactions at the dinner table; however, family meals involve the coordination of many activities and complicated family dynamics. In this paper, we report on findings from interviews and design sessions with 18 families from the Midwestern United States (including both partners/parents and children) to uncover important family differences and tensions that arise around domestic meal experiences. Drawing on feminist theory, we unpack the work of feeding a family as a form of care, drawing attention to the social and emotional complexity of family meals. Critically situating our data within current design narratives, we propose the sensitizing concepts of generative and systemic discontents as a productive way towards troubling the design space of family-food interaction to contend with the struggles that are a part of everyday family meal experiences.
HybridFC: A Hybrid Fact-Checking Approach for Knowledge Graphs
Venue: arXiv cs.LG 2024
Year: 2024

We consider fact-checking approaches that aim to predict the veracity of assertions in knowledge graphs. Five main categories of fact-checking approaches for knowledge graphs have been proposed in the recent literature, of which each is subject to partially overlapping limitations. In particular, current text-based approaches are limited by manual feature engineering. Path-based and rule-based approaches are limited by their exclusive use of knowledge graphs as background knowledge, and embedding-based approaches suffer from low accuracy scores on current fact-checking tasks. We propose a hybrid approach -- dubbed HybridFC -- that exploits the diversity of existing categories of fact-checking approaches within an ensemble learning setting to achieve a significantly better prediction performance. In particular, our approach outperforms the state of the art by 0.14 to 0.27 in terms of Area Under the Receiver Operating Characteristic curve on the FactBench dataset. Our code is open-source and can be found at https://github.com/dice-group/HybridFC.
Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Driving
Venue: arXiv cs.CV 2024
Year: 2024

End-to-end architectures in autonomous driving (AD) face a significant challenge in interpretability, impeding human-AI trust. Human-friendly natural language has been explored for tasks such as driving explanation and 3D captioning. However, previous works primarily focused on the paradigm of declarative interpretability, where the natural language interpretations are not grounded in the intermediate outputs of AD systems, making the interpretations only declarative. In contrast, aligned interpretability establishes a connection between language and the intermediate outputs of AD systems. Here we introduce Hint-AD, an integrated AD-language system that generates language aligned with the holistic perception-prediction-planning outputs of the AD model. By incorporating the intermediate outputs and a holistic token mixer sub-network for effective feature adaptation, Hint-AD achieves desirable accuracy, achieving state-of-the-art results in driving language tasks including driving explanation, 3D dense captioning, and command prediction. To facilitate further study on driving explanation task on nuScenes, we also introduce a human-labeled dataset, Nu-X. Codes, dataset, and models will be publicly available.
Elementary School Students' and Teachers' Perceptions Towards Creative Mathematical Writing with Generative AI
Venue: arXiv cs.CY 2024
Year: 2024

While mathematical creative writing can potentially engage students in expressing mathematical ideas in an imaginative way, some elementary school-age students struggle in this process. Generative AI (GenAI) offers possibilities for supporting creative writing activities, such as providing story generation. However, the design of GenAI-powered learning technologies requires careful consideration of the technology reception in the actual classrooms. This study explores students' and teachers' perceptions of creative mathematical writing with the developed GenAI-powered technology. The study adopted a qualitative thematic analysis of the interviews, triangulated with open-ended survey responses and classroom observation of 79 elementary school students, resulting in six themes and 19 subthemes. This study contributes by investigating the lived experience of GenAI-supported learning and the design considerations for GenAI-powered learning technologies and instructions.
Feedback-based Modal Mutual Search for Attacking Vision-Language Pre-training Models
Venue: arXiv cs.CV 2024
Year: 2024

Although vision-language pre-training (VLP) models have achieved remarkable progress on cross-modal tasks, they remain vulnerable to adversarial attacks. Using data augmentation and cross-modal interactions to generate transferable adversarial examples on surrogate models, transfer-based black-box attacks have become the mainstream methods in attacking VLP models, as they are more practical in real-world scenarios. However, their transferability may be limited due to the differences on feature representation across different models. To this end, we propose a new attack paradigm called Feedback-based Modal Mutual Search (FMMS). FMMS introduces a novel modal mutual loss (MML), aiming to push away the matched image-text pairs while randomly drawing mismatched pairs closer in feature space, guiding the update directions of the adversarial examples. Additionally, FMMS leverages the target model feedback to iteratively refine adversarial examples, driving them into the adversarial region. To our knowledge, this is the first work to exploit target model feedback to explore multi-modality adversarial boundaries. Extensive empirical evaluations on Flickr30K and MSCOCO datasets for image-text matching tasks show that FMMS significantly outperforms the state-of-the-art baselines.
Adaptive Meta-Domain Transfer Learning (AMDTL): A Novel Approach for Knowledge Transfer in AI
Venue: arXiv cs.LG 2024
Year: 2024

This paper presents Adaptive Meta-Domain Transfer Learning (AMDTL), a novel methodology that combines principles of meta-learning with domain-specific adaptations to enhance the transferability of artificial intelligence models across diverse and unknown domains. AMDTL aims to address the main challenges of transfer learning, such as domain misalignment, negative transfer, and catastrophic forgetting, through a hybrid framework that emphasizes both generalization and contextual specialization. The framework integrates a meta-learner trained on a diverse distribution of tasks, adversarial training techniques for aligning domain feature distributions, and dynamic feature regulation mechanisms based on contextual domain embeddings. Experimental results on benchmark datasets demonstrate that AMDTL outperforms existing transfer learning methodologies in terms of accuracy, adaptation efficiency, and robustness. This research provides a solid theoretical and practical foundation for the application of AMDTL in various fields, opening new perspectives for the development of more adaptable and inclusive AI systems.
Formative Study for AI-assisted Data Visualization
Venue: arXiv cs.HC 2024
Year: 2024

This formative study investigates the impact of data quality on AI-assisted data visualizations, focusing on how uncleaned datasets influence the outcomes of these tools. By generating visualizations from datasets with inherent quality issues, the research aims to identify and categorize the specific visualization problems that arise. The study further explores potential methods and tools to address these visualization challenges efficiently and effectively. Although tool development has not yet been undertaken, the findings emphasize enhancing AI visualization tools to handle flawed data better. This research underscores the critical need for more robust, user-friendly solutions that facilitate quicker and easier correction of data and visualization errors, thereby improving the overall reliability and usability of AI-assisted data visualization processes.
Inefficient Alliance Formation in Coalitional Blotto Games
Venue: arXiv cs.GT 2024
Year: 2024

When multiple agents are engaged in a network of conflict, some can advance their competitive positions by forming alliances with each other. However, the costs associated with establishing an alliance may outweigh the potential benefits. This study investigates costly alliance formation in the framework of coalitional Blotto games, in which two players compete separately against a common adversary, and are able to collude by exchanging resources with one another. Previous work has shown that both players in the alliance can mutually benefit if one player unilaterally donates, or transfers, a portion of their budget to the other. In this letter, we consider a variation where the transfer of resources is inherently inefficient, meaning that the recipient of the transfer only receives a fraction of the donation. Our findings reveal that even in the presence of inefficiencies, mutually beneficial transfers are still possible. More formally, our main result provides necessary and sufficient conditions for the existence of such transfers, offering insights into the robustness of alliance formation in competitive environments with resource constraints.
Semi-Supervised Reward Modeling via Iterative Self-Training
Venue: arXiv cs.LG 2024
Year: 2024

Reward models (RM) capture the values and preferences of humans and play a central role in Reinforcement Learning with Human Feedback (RLHF) to align pretrained large language models (LLMs). Traditionally, training these models relies on extensive human-annotated preference data, which poses significant challenges in terms of scalability and cost. To overcome these limitations, we propose Semi-Supervised Reward Modeling (SSRM), an approach that enhances RM training using unlabeled data. Given an unlabeled dataset, SSRM involves three key iterative steps: pseudo-labeling unlabeled examples, selecting high-confidence examples through a confidence threshold, and supervised finetuning on the refined dataset. Across extensive experiments on various model configurations, we demonstrate that SSRM significantly improves reward models without incurring additional labeling costs. Notably, SSRM can achieve performance comparable to models trained entirely on labeled data of equivalent volumes. Overall, SSRM substantially reduces the dependency on large volumes of human-annotated data, thereby decreasing the overall cost and time involved in training effective reward models.
FSMDet: Vision-guided feature diffusion for fully sparse 3D detector
Venue: arXiv cs.CV 2024
Year: 2024

Fully sparse 3D detection has attracted an increasing interest in the recent years. However, the sparsity of the features in these frameworks challenges the generation of proposals because of the limited diffusion process. In addition, the quest for efficiency has led to only few work on vision-assisted fully sparse models. In this paper, we propose FSMDet (Fully Sparse Multi-modal Detection), which use visual information to guide the LiDAR feature diffusion process while still maintaining the efficiency of the pipeline. Specifically, most of fully sparse works focus on complex customized center fusion diffusion/regression operators. However, we observed that if the adequate object completion is performed, even the simplest interpolation operator leads to satisfactory results. Inspired by this observation, we split the vision-guided diffusion process into two modules: a Shape Recover Layer (SRLayer) and a Self Diffusion Layer (SDLayer). The former uses RGB information to recover the shape of the visible part of an object, and the latter uses a visual prior to further spread the features to the center region. Experiments demonstrate that our approach successfully improves the performance of previous fully sparse models that use LiDAR only and reaches SOTA performance in multimodal models. At the same time, thanks to the sparse architecture, our method can be up to 5 times more efficient than previous SOTA methods in the inference process.
A Comparative Study of Table Sized Physicalization and Digital Visualization
Venue: arXiv cs.HC 2024
Year: 2024

Data physicalization is gaining popularity in public and educational contexts due to its potential to make abstract data more tangible and understandable. Despite its growing use, there remains a significant gap in our understanding of how large-size physical visualizations compare to their digital counterparts in terms of user comprehension and memory retention. This study aims to bridge this knowledge gap by comparing the effectiveness of visualizing school building history data on large digital screens versus large physical models. Our experimental approach involved 32 participants who were exposed to one of the visualization mediums. We assessed their user experience and immediate understanding of the content, measured through tests after exposure, and evaluated memory retention with follow-up tests seven days later. The results revealed notable differences between the two forms of visualization: physicalization not only facilitated better initial comprehension but also significantly enhanced long-term memory retention. Furthermore, user feedback on usability was also higher on physicalization. These findings underscore the substantial impact of physicalization in improving information comprehension and retention. This study contributes crucial insights into future visualization media selection in educational and public settings.
A New Notion of Regularity: Finite State Automata Accepting Graphs
Venue: arXiv cs.FL 2024
Year: 2024

Analogous to regular string and tree languages, regular languages of directed acyclic graphs (DAGs) are defined in the literature. Although called regular, those DAG-languages are more powerful and, consequently, standard problems have a higher complexity than in the string case. Top-down as well as bottom-up deterministic DAG languages are subclasses of the regular DAG languages. We refine this hierarchy by providing a weaker subclass of the deterministic DAG languages. For a DAG grammar generating a language in this new DAG language class, or, equivalently, a DAG-automaton recognizing it, a classical deterministic finite state automaton (DFA) can be constructed. As the main result, we provide a characterization of this class. The motivation behind this is the transfer of techniques for regular string languages to graphs. Trivially, our restricted DAG language class is closed under union and intersection. This permits the application of minimization and hyper-minimization algorithms known for DFAs. This alternative notion of regularity coins at the existence of a DFA for recognizing a DAG language.
Non-Global Parikh Tree Automata
Venue: arXiv cs.FL 2024
Year: 2024

Parikh (tree) automata are an expressive and yet computationally well-behaved extension of finite automata -- they allow to increment a number of counters during their computations, which are finally tested by a semilinear constraint. In this work, we introduce and investigate a new perspective on Parikh tree automata (PTA): instead of testing one counter configuration that results from the whole input tree, we implement a non-global automaton model. Here, we copy and distribute the current configuration at each node to all its children, incrementing the counters pathwise, and check the arithmetic constraint at each leaf. We obtain that the classes of tree languages recognizable by global PTA and non-global PTA are incomparable. In contrast to global PTA, the non-emptiness problem is undecidable for non-global PTA if we allow the automata to work with at least three counters, whereas the membership problem stays decidable. However, for a restriction of the model, where counter configurations are passed in a linear fashion to at most one child node, we can prove decidability of the non-emptiness problem.
E-commerce Webpage Recommendation Scheme Base on Semantic Mining and Neural Networks
Venue: arXiv cs.IR 2024
Year: 2024

In e-commerce websites, web mining web page recommendation technology has been widely used. However, recommendation solutions often cannot meet the actual application needs of online shopping users. To address this problem, this paper proposes an e-commerce web page recommendation solution that combines semantic web mining and BP neural networks. First, the web logs of user searches are processed, and 5 features are extracted: content priority, time consumption priority, online shopping users' explicit/implicit feedback on the website, recommendation semantics and input deviation amount. Then, these features are used as input features of the BP neural network to classify and identify the priority of the final output web page. Finally, the web pages are sorted according to priority and recommended to users. This project uses book sales webpages as samples for experiments. The results show that this solution can quickly and accurately identify the webpages required by users.
Pushing the Limits of Vision-Language Models in Remote Sensing without Human Annotations
Venue: arXiv cs.CV 2024
Year: 2024

The prominence of generalized foundation models in vision-language integration has witnessed a surge, given their multifarious applications. Within the natural domain, the procurement of vision-language datasets to construct these foundation models is facilitated by their abundant availability and the ease of web crawling. Conversely, in the remote sensing domain, although vision-language datasets exist, their volume is suboptimal for constructing robust foundation models. This study introduces an approach to curate vision-language datasets by employing an image decoding machine learning model, negating the need for human-annotated labels. Utilizing this methodology, we amassed approximately 9.6 million vision-language paired datasets in VHR imagery. The resultant model outperformed counterparts that did not leverage publicly available vision-language datasets, particularly in downstream tasks such as zero-shot classification, semantic localization, and image-text retrieval. Moreover, in tasks exclusively employing vision encoders, such as linear probing and k-NN classification, our model demonstrated superior efficacy compared to those relying on domain-specific vision-language datasets.
Invariant filtering for wheeled vehicle localization with unknown wheel radius and unknown GNSS lever arm
Venue: arXiv cs.RO 2024
Year: 2024

We consider the problem of observer design for a nonholonomic car (more generally a wheeled robot) equipped with wheel speeds with unknown wheel radius, and whose position is measured via a GNSS antenna placed at an unknown position in the car. In a tutorial and unified exposition, we recall the recent theory of two-frame systems within the field of invariant Kalman filtering. We then show how to adapt it geometrically to address the considered problem, although it seems at first sight out of its scope. This yields an invariant extended Kalman filter having autonomous error equations, and state-independent Jacobians, which is shown to work remarkably well in simulations. The proposed novel construction thus extends the application scope of invariant filtering.
Automated Speaking Assessment of Conversation Tests with Novel Graph-based Modeling on Spoken Response Coherence
Venue: arXiv cs.CL 2024
Year: 2024

Automated speaking assessment in conversation tests (ASAC) aims to evaluate the overall speaking proficiency of an L2 (second-language) speaker in a setting where an interlocutor interacts with one or more candidates. Although prior ASAC approaches have shown promising performance on their respective datasets, there is still a dearth of research specifically focused on incorporating the coherence of the logical flow within a conversation into the grading model. To address this critical challenge, we propose a hierarchical graph model that aptly incorporates both broad inter-response interactions (e.g., discourse relations) and nuanced semantic information (e.g., semantic words and speaker intents), which is subsequently fused with contextual information for the final prediction. Extensive experimental results on the NICT-JLE benchmark dataset suggest that our proposed modeling approach can yield considerable improvements in prediction accuracy with respect to various assessment metrics, as compared to some strong baselines. This also sheds light on the importance of investigating coherence-related facets of spoken responses in ASAC.
Multimodal Emotion Recognition with Vision-language Prompting and Modality Dropout
Venue: arXiv cs.CV 2024
Year: 2024

In this paper, we present our solution for the Second Multimodal Emotion Recognition Challenge Track 1(MER2024-SEMI). To enhance the accuracy and generalization performance of emotion recognition, we propose several methods for Multimodal Emotion Recognition. Firstly, we introduce EmoVCLIP, a model fine-tuned based on CLIP using vision-language prompt learning, designed for video-based emotion recognition tasks. By leveraging prompt learning on CLIP, EmoVCLIP improves the performance of pre-trained CLIP on emotional videos. Additionally, to address the issue of modality dependence in multimodal fusion, we employ modality dropout for robust information fusion. Furthermore, to aid Baichuan in better extracting emotional information, we suggest using GPT-4 as the prompt for Baichuan. Lastly, we utilize a self-training strategy to leverage unlabeled videos. In this process, we use unlabeled videos with high-confidence pseudo-labels generated by our model and incorporate them into the training set. Experimental results demonstrate that our model ranks 1st in the MER2024-SEMI track, achieving an accuracy of 90.15% on the test set.
Scenario Execution for Robotics: A generic, backend-agnostic library for running reproducible robotics experiments and tests
Venue: arXiv cs.RO 2024
Year: 2024

Testing and evaluation of robotics systems is a difficult and oftentimes tedious task due to the systems' complexity and a lack of tools to conduct reproducible robotics experiments. Additionally, almost all available tools are either tailored towards a specific application domain, simulator or middleware. Particularly scenario-based testing, a common practice in the domain of automated driving, is not sufficiently covered in the robotics domain. In this paper, we propose a novel backend- and middleware-agnostic approach for conducting systematic, reproducible and automatable robotics experiments called Scenario Execution for Robotics. Our approach is implemented as a Python library built on top of the generic scenario description language OpenSCENARIO 2 and Behavior Trees and is made publicly available on GitHub. In extensive experiments, we demonstrate that our approach supports multiple simulators as backend and can be used as a standalone Python-library or as part of the ROS2 ecosystem. Furthermore, we demonstrate how our approach enables testing over ranges of varying values. Finally, we show how Scenario Execution for Robotics allows to move from simulation-based to real-world experiments with minimal adaptations to the scenario description file.