Reviewerly/exHarmony huggingface dataset:
Learnable Cross-modal Knowledge Distillation for Multi-modal Learning with Missing Modality

The problem of missing modalities is both critical and non-trivial to be handled in multi-modal models. It is common for multi-modal tasks that certain modalities contribute more compared to other modalities, and if those important modalities are missing, the model performance drops significantly. Such fact remains unexplored by current multi-modal approaches that recover the representation from missing modalities by feature reconstruction or blind feature aggregation from other modalities, instead of extracting useful information from the best performing modalities. In this paper, we propose a Learnable Cross-modal Knowledge Distillation (LCKD) model to adaptively identify important modalities and distil knowledge from them to help other modalities from the cross-modal perspective for solving the missing modality issue. Our approach introduces a teacher election procedure to select the most ``qualified'' teachers based on their single modality performance on certain tasks. Then, cross-modal knowledge distillation is performed between teacher and student modalities for each task to push the model parameters to a point that is beneficial for all tasks. Hence, even if the teacher modalities for certain tasks are missing during testing, the available student modalities can accomplish the task well enough based on the learned knowledge from their automatically elected teacher modalities. Experiments on the Brain Tumour Segmentation Dataset 2018 (BraTS2018) shows that LCKD outperforms other methods by a considerable margin, improving the state-of-the-art performance by 3.61% for enhancing tumour, 5.99% for tumour core, and 3.76% for whole tumour in terms of segmentation Dice score.
Optimization or Architecture: How to Hack Kalman Filtering

In non-linear filtering, it is traditional to compare non-linear architectures such as neural networks to the standard linear Kalman Filter (KF). We observe that this mixes the evaluation of two separate components: the non-linear architecture, and the parameters optimization method. In particular, the non-linear model is often optimized, whereas the reference KF model is not. We argue that both should be optimized similarly, and to that end present the Optimized KF (OKF). We demonstrate that the KF may become competitive to neural models - if optimized using OKF. This implies that experimental conclusions of certain previous studies were derived from a flawed process. The advantage of OKF over the standard KF is further studied theoretically and empirically, in a variety of problems. Conveniently, OKF can replace the KF in real-world systems by merely updating the parameters.
Revolutionizing Mobile Interaction: Enabling a 3 Billion Parameter GPT LLM on Mobile

The field of Artificial Intelligence has witnessed remarkable progress in recent years, especially with the emergence of powerful large language models (LLMs) based on the transformer architecture. Cloud-based LLMs, such as OpenAI's ChatGPT, offer impressive capabilities but come with concerns regarding latency and privacy due to network dependencies. This article presents an innovative approach to LLM inference, envisioning a future where LLMs with billions of parameters can be executed directly on mobile devices without network connectivity. The article showcases a fine-tuned GPT LLM with 3 billion parameters that can operate smoothly on devices with as low as 4GB of memory. Through the integration of native code and model quantization techniques, the application not only serves as a general-purpose assistant but also facilitates seamless mobile interactions with text-to-actions features. The article provides insights into the training pipeline, implementation details, test results, and future directions of on-device LLM inference. This breakthrough technology opens up possibilities for empowering users with sophisticated AI capabilities while preserving their privacy and eliminating latency concerns.
You Only Look at Once for Real-time and Generic Multi-Task

High precision, lightweight, and real-time responsiveness are three essential requirements for implementing autonomous driving. In this study, we present an adaptive, real-time, and lightweight multi-task model designed to concurrently address object detection, drivable area segmentation, and lane line segmentation tasks. Specifically, we developed an end-to-end multi-task model with a unified and streamlined segmentation structure. We introduced a learnable parameter that adaptively concatenate features in segmentation necks, using the same loss function for all segmentation tasks. This eliminates the need for customizations and enhances the model's generalization capabilities. We also introduced a segmentation head composed only of a series of convolutional layers, which reduces the inference time. We achieved competitive results on the BDD100k dataset, particularly in visualization outcomes. The performance results show a mAP50 of 81.1% for object detection, a mIoU of 91.0% for drivable area segmentation, and an IoU of 28.8% for lane line segmentation. Additionally, we introduced real-world scenarios to evaluate our model's performance in a real scene, which significantly outperforms competitors. This demonstrates that our model not only exhibits competitive performance but is also more flexible and faster than existing multi-task models. The source codes and pre-trained models are released at https://github.com/JiayuanWang-JW/YOLOv8-multi-task
On Representation Complexity of Model-based and Model-free Reinforcement Learning

We study the representation complexity of model-based and model-free reinforcement learning (RL) in the context of circuit complexity. We prove theoretically that there exists a broad class of MDPs such that their underlying transition and reward functions can be represented by constant depth circuits with polynomial size, while the optimal $Q$-function suffers an exponential circuit complexity in constant-depth circuits. By drawing attention to the approximation errors and building connections to complexity theory, our theory provides unique insights into why model-based algorithms usually enjoy better sample complexity than model-free algorithms from a novel representation complexity perspective: in some cases, the ground-truth rule (model) of the environment is simple to represent, while other quantities, such as $Q$-function, appear complex. We empirically corroborate our theory by comparing the approximation error of the transition kernel, reward function, and optimal $Q$-function in various Mujoco environments, which demonstrates that the approximation errors of the transition kernel and reward function are consistently lower than those of the optimal $Q$-function. To the best of our knowledge, this work is the first to study the circuit complexity of RL, which also provides a rigorous framework for future research.
Can GPT-4 Replicate Empirical Software Engineering Research?

Empirical software engineering research on production systems has brought forth a better understanding of the software engineering process for practitioners and researchers alike. However, only a small subset of production systems is studied, limiting the impact of this research. While software engineering practitioners benefit from replicating research on their own data, this poses its own set of challenges, since performing replications requires a deep understanding of research methodologies and subtle nuances in software engineering data. Given that large language models (LLMs), such as GPT-4, show promise in tackling both software engineering- and science-related tasks, these models could help democratize empirical software engineering research. In this paper, we examine LLMs' abilities to perform replications of empirical software engineering research on new data. We specifically study their ability to surface assumptions made in empirical software engineering research methodologies, as well as their ability to plan and generate code for analysis pipelines on seven empirical software engineering papers. We perform a user study with 14 participants with software engineering research expertise, who evaluate GPT-4-generated assumptions and analysis plans (i.e., a list of module specifications) from the papers. We find that GPT-4 is able to surface correct assumptions, but struggle to generate ones that reflect common knowledge about software engineering data. In a manual analysis of the generated code, we find that the GPT-4-generated code contains the correct high-level logic, given a subset of the methodology. However, the code contains many small implementation-level errors, reflecting a lack of software engineering knowledge. Our findings have implications for leveraging LLMs for software engineering research as well as practitioner data scientists in software teams.
SYRAC: Synthesize, Rank, and Count

Crowd counting is a critical task in computer vision, with several important applications. However, existing counting methods rely on labor-intensive density map annotations, necessitating the manual localization of each individual pedestrian. While recent efforts have attempted to alleviate the annotation burden through weakly or semi-supervised learning, these approaches fall short of significantly reducing the workload. We propose a novel approach to eliminate the annotation burden by leveraging latent diffusion models to generate synthetic data. However, these models struggle to reliably understand object quantities, leading to noisy annotations when prompted to produce images with a specific quantity of objects. To address this, we use latent diffusion models to create two types of synthetic data: one by removing pedestrians from real images, which generates ranked image pairs with a weak but reliable object quantity signal, and the other by generating synthetic images with a predetermined number of objects, offering a strong but noisy counting signal. Our method utilizes the ranking image pairs for pre-training and then fits a linear layer to the noisy synthetic images using these crowd quantity features. We report state-of-the-art results for unsupervised crowd counting.
Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond

In this study, we explore the potential of Multimodal Large Language Models (MLLMs) in improving embodied decision-making processes for agents. While Large Language Models (LLMs) have been widely used due to their advanced reasoning skills and vast world knowledge, MLLMs like GPT4-Vision offer enhanced visual understanding and reasoning capabilities. We investigate whether state-of-the-art MLLMs can handle embodied decision-making in an end-to-end manner and whether collaborations between LLMs and MLLMs can enhance decision-making. To address these questions, we introduce a new benchmark called PCA-EVAL, which evaluates embodied decision-making from the perspectives of Perception, Cognition, and Action. Additionally, we propose HOLMES, a multi-agent cooperation framework that allows LLMs to leverage MLLMs and APIs to gather multimodal information for informed decision-making. We compare end-to-end embodied decision-making and HOLMES on our benchmark and find that the GPT4-Vision model demonstrates strong end-to-end embodied decision-making abilities, outperforming GPT4-HOLMES in terms of average decision accuracy (+3%). However, this performance is exclusive to the latest GPT4-Vision model, surpassing the open-source state-of-the-art MLLM by 26%. Our results indicate that powerful MLLMs like GPT4-Vision hold promise for decision-making in embodied agents, offering new avenues for MLLM research. Code and data are open at https://github.com/pkunlp-icler/PCA-EVAL/.
Design and Performance Test of Multi Intake Foods Product Cutting Machine

Appropriate technology is one of the answers to support small and medium enterprises in the sustainability of their business production. This research aims to design and test the performance of multi-intake food product cutting machines, this cutting machine can be applied to cutting food products such as tempeh, crackers, fruits, or tubers. This machine is called multi intake because it can cut more than one material at the same time and produce more cuts in a relatively short time. This machine is equipped with an automatic propulsion system that can produce pieces of uniform thickness. The results showed that the construction of a multi-intake cutting machine for cutting food ingredients in the form of tempeh has a capacity of 120 kg h -1 to 173.5 kg h -1 , the most optimal performance of the cutting machine for foodstuffs and can increase the capacity and quality of the stripping results achieved in the use of an electric motor with power 375 Watt with engine shaft rotation (n) = 327 rad s -1 and a thrust speed of 0.370 cm s -1 which can produce a cutting capacity of 173.5 kg h -1 with intact, not defective, and flat cut quality.
A Multitask Framework for Label Refinement and Lesion Segmentation in Clinical Brain Imaging

Supervised deep learning methods offer the potential for automating lesion segmentation in routine clinical brain imaging, but performance is dependent on label quality. In practice, obtaining high-quality labels from experienced annotators for large-scale datasets is not always feasible, while noisy labels from less experienced annotators are often available. Prior studies focus on either label refinement methods or on learning to segment with noisy labels, but there has been little work on integrating these approaches within a unified framework. To address this gap, we propose a novel multitask framework for end-to-end noisy-label refinement and lesion segmentation. Our approach minimizes the discrepancy between the refined label and the predicted segmentation mask, and is highly customizable for scenarios with multiple sets of noisy labels, incomplete ground truth coverage and/or 2D/3D scans. In extensive experiments on both proprietary and public clinical brain imaging datasets, we demonstrate that our end-to-end framework offers strong performance improvements over prevailing baselines on both label refinement and lesion segmentation. Our proposed framework maintains performance gains over baselines even when ground truth labels are available for only 25–50% of the dataset. Our approach has implications for effective medical image segmentation in settings that are replete with noisy labels but sparse on ground truth annotation.
A New Encoder Using Character and Word Feature Fusion for Chinese Math Word Problem Solving

Automatically solving math word problems (MWPs) aims at generating corresponding math expressions based on the problem text. Since pre-trained language models can provide rich semantic representation for decoders, they can be used as the encoders for MWP models. However, most Chinese version of pre-trained language models use characters as the basic unit for downstream tasks, thus using them as encoders will lose the information carried by words. In addition, most of current methods ignore the difference between the operators +, $$\times $$ , and -, $${\div }$$ , $$\wedge $$ . They neglect to make the model understand the position order of the left and right subtree of operators -, $${\div }$$ , and $$\wedge $$ is important. To address the above issues, we propose a Character and Word Fusion Encoder (CWFE) and a negative label loss function in this paper. CWFE can fuse character and word feature information in problem text. Negative label loss prevents the model from incorrectly resolving the positions of operands. Using GTS as the decoder, our model called CWFE-GTS-N outperforms the models HGEN, RPKHS, and E-GTS in the experiments. The accuracy of CWFE-GTS-N reaches 85.9% and 86.4% on two datasets, Math23K and Ape-clean $$^\sim $$ , respectively. Code is available at https://github.com/SCNU203/CWFE-GTS-N
Fine-Grained Question-Answer Matching via Sentence-Aware Contrastive Self-supervised Transfer

Previous studies always consider the question-answer (QA) matching task as a one-to-one text matching problem. This study builds upon existing research by expanding the scope to a many-to-many mapping scenario and proposes a new fine-grained QA matching (FQAM) task, which aims to accurately predict the many-to-many matching relationship between all sub-questions and sub-answers within each QA pair. Particularly, a meticulously annotated corpus of high quality is constructed specifically for FQAM to facilitate this research. On this basis, owing to the challenge of expensive data annotation associated with FQAM, we propose a sentence-aware contrastive self-supervised transfer (SCST) approach to transfer the sentence alignment information pre-trained with massive unannotated QA pairs to assist in FQAM. Experimental evaluations conducted on our annotated corpus demonstrate the importance of utilizing sentence alignment information from unannotated QA pairs in FQAM and justify the effectiveness of our approach in capturing such information.
Between Batik and Fiberglass: The Hybrid-Technology of Boat Construction

Batik is one of the cultural elements in handicrafts for the Malay community in Malaysia. The production of batik is usually done on fabrics made from materials such as cotton to produce clothes for men and women. In Malaysia, batik is not only worn by Malays but also by non-Malays because of its attractive pattern and suitable to be worn when attending formal or informal events. However, the prominence of Malay cultural identity through the art of batik production is seen only as limited to clothing, jewelry, and accessories. Recognizing this fact, a group of lecturers and students from Universiti Kuala Lumpur Malaysian Institute of Marine Engineering Technology (UniKL MIMET) sought to explore the application of batik in the construction of fiberglass boats as an initiative and alternative to highlight and maintain Malay cultural identity based on batik. The research method used is experimental through the construction of a prototype of a fiberglass boat and the method is done by combining fabrics that contain batik patterns through certain processes. As a result, the external shape of the fiberglass boat will be covered with batik patterns and make this boat more beautiful and attractive. Batik-based fiberglass boats have the potential to be used in the tourism sector as a cruise boat to attract domestic and foreign tourists due to its attractiveness and uniqueness. In addition, the boat can also maintain and promote the Malay cultural heritage through batik handicrafts applied to fiberglass boats in Malaysia.
GLORIA: A Graph Convolutional Network-Based Approach for Review Spam Detection

Spam reviews contain untruthful content created with malevolent intent, to affect the overall reputation of a product, service or company. This content is commonly made by malicious users or automated programs (i.e., bots) that mimic human behaviour. With the recent boom of online review systems, performing accurate review spam detection has become of primary importance for a review platform, to mitigate the effect of malicious users responsible for untruthful content. In this work, we propose a review spam classification approach, named GLORIA, that adopts a graph representation of review data and trains a graph convolutional neural network for edge classification as a review spam detection model. In particular, GLORIA represents both users (i.e., authors of reviews) and products (i.e., reviewed items) as nodes of a heterogeneous graph, while it represents reviews as graph edges that connect each author of a review to the reviewed item. Features of users, products and reviews are associated with nodes and edges, respectively. Experiments performed on publicly available review datasets prove the effectiveness of the proposed approach compared with some state-of-the-art approaches.
Adaptive Container Orchestration Mechanism on Electric Power Supercomputing Clouds

Electric power supercomputing clouds bring huge computing capacity to various scientific, engineering and commercial applications. In this context, container as a flexible, efficient and scalable virtualization technology has become a key means to improve the resource utilization and operational efficiency of electric power supercomputing clouds. However, existing container orchestration algorithms tend to be static and unable to adapt to dynamically changing loads and environments, resulting in significant resource consumption and failure to meet user quality of service. To this end, we propose an adaptive container orchestration mechanism for power supercomputing clouds that captures the periodic pattern of loads through a time-series model, and design a waiting window approach to mitigate the oscillation phenomenon in resource scaling. Experimental results on two application loads show that our container orchestration mechanism achieves the best service performance and the lowest system resource overhead compared to strong baselines.
A Review of Knowledge-Grounded Dialogue Systems

With the rapid development of computer science and natural language processing technologies, the development of dialogue systems attracts researchers' wide interest, and how to build natural and engaging conversations becomes an urgent problem in great demand. Traditional dialogue systems that solely rely on dialogue context have exposed a series of severe shortages such as tend to generate numerously uninformative responses and meaningless replies. To this end, the principle of knowledge-grounded dialogue systems is proposed, which provides new views on the evolution and research of dialogue systems. However, it's challenging for knowledge-grounded dialogue systems to select appropriate and accurate knowledge information from the knowledge base and integrate the selected knowledge information and the dialogue history into the responses correctly. This paper introduces the definition and development of knowledge-grounded dialogue systems, lists several common datasets, summarizes current main methods and model structures based on the division of the main structures of knowledge-grounded dialogue systems, finally discusses the shortcomings of previous work, and provides views on future research trends.
Identification of Abnormal Behavior in Electric Power Operations Using the Human Skeleton's Key Features

At present, the safety monitoring of electric power sites mainly monitors the monitoring video through personnel, which has problems such as heavy workload, missed detection, and false alarms. In order to realize the intelligent recognition of the behavior of personnel at the electric power operation site, an abnormal behavior recognition method based on the key points of human skeleton is proposed, which can realize the real-time detection of the behavior of personnel at the operation site. First, the backbone network of YOLOv5s is improved for target detection, and the trajectory of the target is obtained by combining the DeepSort target tracking algorithm; then, the AlphaPose attitude estimation method is used to extract the target skeleton sequence from the trajectory; finally, the multi-partition spatiotemporal graph convolutional network is used to track the personnel Abnormal behavior identification. The experimental results show that the recognition accuracy rate of the algorithm reaches 93.5%, and the processing speed reaches 18.32 fps, which can identify abnormal behaviors of electric power workers in real time and accurately.