OpenRaiser/Intern-Atlas huggingface dataset:
Fed-MUnet: Multi-modal Federated Unet for Brain Tumor Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

Deep learning-based techniques have been widely utilized for brain tumor segmentation using both single and multi-modal Magnetic Resonance Imaging (MRI) images. Most current studies focus on centralized training due to the intrinsic challenge of data sharing across clinics. To mitigate privacy concerns, researchers have introduced Federated Learning (FL) methods to brain tumor segmentation tasks. However, currently such methods are focusing on single modal MRI, with limited study on multi-modal MRI. The challenges include complex structure, large-scale parameters, and overfitting issues of the FL based methods using multi-modal MRI. To address the above challenges, we propose a novel multi-modal FL framework for brain tumor segmentation (Fed-MUnet) that is suitable for FL training. We evaluate our approach with the BraTS2022 datasets, which are publicly available. The experimental results demonstrate that our framework achieves FL nature of distributed learning and privacy preserving. For the enhancing tumor, tumor core and whole tumor, the mean of five major metrics were 87.5%, 90.6% and 92.2%, respectively, which were higher than SOTA methods while preserving privacy. In terms of parameters count, quantity of floating-point operations (FLOPs) and inference, Fed-MUnet is Pareto optimal compared with the state-of-the-art segmentation backbone while achieves higher performance and tackles privacy issue. Our codes are open-sourced at https://github.com/Arnold-Jun/Fed-MUnet.
Mindscape: Research of high-information density street environments based on electroencephalogram recording and virtual reality head-mounted simulation
Venue: arXiv cs.HC 2024
Year: 2024

This study aims to investigate, through neuroscientific methods, the effects of particular architectural elements on pedestrian spatial cognition and experience in the analysis and design of walking street spaces. More precisely, this paper will describe the impact of the density variation of storefront signs on the brainwaves of passersby in East Asian city walking streets, providing strategies and guidelines for urban development and renewal. Firstly, the paper summarizes the research method through the review of research questions and related literature; secondly, the paper establishes experiments via this path, analyzing results and indicators through data processing; finally, suggestions for future pedestrian street design are proposed based on research and analysis results.
Upgrading Pepper Robot s Social Interaction with Advanced Hardware and Perception Enhancements
Venue: arXiv cs.RO 2024
Year: 2024

In this paper, we propose hardware and software enhancements for the Pepper robot to improve its human-robot interaction capabilities. This includes the integration of an NVIDIA Jetson GPU to enhance computational capabilities and execute real time algorithms, and a RealSense D435i camera to capture depth images, as well as the computer vision algorithms to detect and localize the humans around the robot and estimate their body orientation and gaze direction. The new stack is implemented on ROS and is running on the extended Pepper hardware, and the communication with the robot s firmware is done through the NAOqi ROS driver API. We have also collected a MoCap dataset of human activities in a controlled environment, together with the corresponding RGB-D data, to validate the proposed perception algorithms.
NYK-MS: A Well-annotated Multi-modal Metaphor and Sarcasm Understanding Benchmark on Cartoon-Caption Dataset
Venue: arXiv cs.CL 2024
Year: 2024

Metaphor and sarcasm are common figurative expressions in people's communication, especially on the Internet or the memes popular among teenagers. We create a new benchmark named NYK-MS (NewYorKer for Metaphor and Sarcasm), which contains 1,583 samples for metaphor understanding tasks and 1,578 samples for sarcasm understanding tasks. These tasks include whether it contains metaphor/sarcasm, which word or object contains metaphor/sarcasm, what does it satirize and why does it contains metaphor/sarcasm, all of the 7 tasks are well-annotated by at least 3 annotators. We annotate the dataset for several rounds to improve the consistency and quality, and use GUI and GPT-4V to raise our efficiency. Based on the benchmark, we conduct plenty of experiments. In the zero-shot experiments, we show that Large Language Models (LLM) and Large Multi-modal Models (LMM) can't do classification task well, and as the scale increases, the performance on other 5 tasks improves. In the experiments on traditional pre-train models, we show the enhancement with augment and alignment methods, which prove our benchmark is consistent with previous dataset and requires the model to understand both of the two modalities.
Robust Vehicle Localization and Tracking in Rain using Street Maps
Venue: arXiv cs.RO 2024
Year: 2024

GPS-based vehicle localization and tracking suffers from unstable positional information commonly experienced in tunnel segments and in dense urban areas. Also, both Visual Odometry (VO) and Visual Inertial Odometry (VIO) are susceptible to adverse weather conditions that causes occlusions or blur on the visual input. In this paper, we propose a novel approach for vehicle localization that uses street network based map information to correct drifting odometry estimates and intermittent GPS measurements especially, in adversarial scenarios such as driving in rain and tunnels. Specifically, our approach is a flexible fusion algorithm that integrates intermittent GPS, drifting IMU and VO estimates together with 2D map information for robust vehicle localization and tracking. We refer to our approach as Map-Fusion. We robustly evaluate our proposed approach on four geographically diverse datasets from different countries ranging across clear and rain weather conditions. These datasets also include challenging visual segments in tunnels and underpasses. We show that with the integration of the map information, our Map-Fusion algorithm reduces the error of the state-of-the-art VO and VIO approaches across all datasets. We also validate our proposed algorithm in a real-world environment and in real-time on a hardware constrained mobile robot. Map-Fusion achieved 2.46m error in clear weather and 6.05m error in rain weather for a 150m route.
Follow-Your-Canvas: Higher-Resolution Video Outpainting with Extensive Content Generation
Venue: arXiv cs.CV 2024
Year: 2024

This paper explores higher-resolution video outpainting with extensive content generation. We point out common issues faced by existing methods when attempting to largely outpaint videos: the generation of low-quality content and limitations imposed by GPU memory. To address these challenges, we propose a diffusion-based method called \textit{Follow-Your-Canvas}. It builds upon two core designs. First, instead of employing the common practice of "single-shot" outpainting, we distribute the task across spatial windows and seamlessly merge them. It allows us to outpaint videos of any size and resolution without being constrained by GPU memory. Second, the source video and its relative positional relation are injected into the generation process of each window. It makes the generated spatial layout within each window harmonize with the source video. Coupling with these two designs enables us to generate higher-resolution outpainting videos with rich content while keeping spatial and temporal consistency. Follow-Your-Canvas excels in large-scale video outpainting, e.g., from 512X512 to 1152X2048 (9X), while producing high-quality and aesthetically pleasing results. It achieves the best quantitative results across various resolution and scale setups. The code is released on https://github.com/mayuelala/FollowYourCanvas
Progressive Retinal Image Registration via Global and Local Deformable Transformations
Venue: arXiv cs.CV 2024
Year: 2024

Retinal image registration plays an important role in the ophthalmological diagnosis process. Since there exist variances in viewing angles and anatomical structures across different retinal images, keypoint-based approaches become the mainstream methods for retinal image registration thanks to their robustness and low latency. These methods typically assume the retinal surfaces are planar, and adopt feature matching to obtain the homography matrix that represents the global transformation between images. Yet, such a planar hypothesis inevitably introduces registration errors since retinal surface is approximately curved. This limitation is more prominent when registering image pairs with significant differences in viewing angles. To address this problem, we propose a hybrid registration framework called HybridRetina, which progressively registers retinal images with global and local deformable transformations. For that, we use a keypoint detector and a deformation network called GAMorph to estimate the global transformation and local deformable transformation, respectively. Specifically, we integrate multi-level pixel relation knowledge to guide the training of GAMorph. Additionally, we utilize an edge attention module that includes the geometric priors of the images, ensuring the deformation field focuses more on the vascular regions of clinical interest. Experiments on two widely-used datasets, FIRE and FLoRI21, show that our proposed HybridRetina significantly outperforms some state-of-the-art methods. The code is available at https://github.com/lyp-deeplearning/awesome-retinal-registration.
Soft-lockins in Public Sector Acquisitions of Open Source Software-solutions: A Case Study on a Municipal E-Service Platform
Venue: arXiv cs.SE 2024
Year: 2024

Background: Open Source Software (OSS) is often seen as an option to mitigate risks of lock-ins. Yet, single-vendor OSS can still result in soft lock-ins due to knowledge asymmetries and technical barriers. Aim: This study explores actors that render such soft lock-ins. Research design: We conduct a qualitative case study of an E-service Platform (ESP) used by over 190+ municipalities. Results: User-driven lock-in factors emerged as a significant category, including limited and non-transparent communication, restrictive qualification requirements in procurement, confusion on maintainership, and comfort in the status quo. Technical lock-in factors include inadequate documentation, dependency management issues, and limited test coverage. Conclusions: Strong leadership and continuous training is needed to address presence of comfort and conservative culture among municipalities. Open Source Stewards, i.e., neutral hosts for OSS projects, can support municipalities in these tasks while also helping to foster an open, competitive collaboration that can enable a broader supplier ecosystem.
Understanding Multimodal Hallucination with Parameter-Free Representation Alignment
Venue: arXiv cs.CV 2024
Year: 2024

Hallucination is a common issue in Multimodal Large Language Models (MLLMs), yet the underlying principles remain poorly understood. In this paper, we investigate which components of MLLMs contribute to object hallucinations. To analyze image representations while completely avoiding the influence of all other factors other than the image representation itself, we propose a parametric-free representation alignment metric (Pfram) that can measure the similarities between any two representation systems without requiring additional training parameters. Notably, Pfram can also assess the alignment of a neural representation system with the human representation system, represented by ground-truth annotations of images. By evaluating the alignment with object annotations, we demonstrate that this metric shows strong and consistent correlations with object hallucination across a wide range of state-of-the-art MLLMs, spanning various model architectures and sizes. Furthermore, using this metric, we explore other key issues related to image representations in MLLMs, such as the role of different modules, the impact of textual instructions, and potential improvements including the use of alternative visual encoders. Our code is available at: https://github.com/yellow-binary-tree/Pfram.
Forecasting infectious disease prevalence with associated uncertainty using neural networks
Venue: arXiv cs.LG 2024
Year: 2024

Infectious diseases pose significant human and economic burdens. Accurately forecasting disease incidence can enable public health agencies to respond effectively to existing or emerging diseases. Despite progress in the field, developing accurate forecasting models remains a significant challenge. This thesis proposes two methodological frameworks using neural networks (NNs) with associated uncertainty estimates - a critical component limiting the application of NNs to epidemic forecasting thus far. We develop our frameworks by forecasting influenza-like illness (ILI) in the United States. Our first proposed method uses Web search activity data in conjunction with historical ILI rates as observations for training NN architectures. Our models incorporate Bayesian layers to produce uncertainty intervals, positioning themselves as legitimate alternatives to more conventional approaches. The best performing architecture: iterative recurrent neural network (IRNN), reduces mean absolute error by 10.3% and improves Skill by 17.1% on average in forecasting tasks across four flu seasons compared to the state-of-the-art. We build on this method by introducing IRNNs, an architecture which changes the sampling procedure in the IRNN to improve the uncertainty estimation. Our second framework uses neural ordinary differential equations to bridge the gap between mechanistic compartmental models and NNs; benefiting from the physical constraints that compartmental models provide. We evaluate eight neural ODE models utilising a mixture of ILI rates and Web search activity data to provide forecasts. These are compared with the IRNN and IRNN0 - the IRNN using only ILI rates. Models trained without Web search activity data outperform the IRNN0 by 16% in terms of Skill. Future work should focus on more effectively using neural ODEs with Web search data to compete with the best performing IRNN.
PACSBO: Probably approximately correct safe Bayesian optimization
Venue: arXiv cs.LG 2024
Year: 2024

Safe Bayesian optimization (BO) algorithms promise to find optimal control policies without knowing the system dynamics while at the same time guaranteeing safety with high probability. In exchange for those guarantees, popular algorithms require a smoothness assumption: a known upper bound on a norm in a reproducing kernel Hilbert space (RKHS). The RKHS is a potentially infinite-dimensional space, and it is unclear how to, in practice, obtain an upper bound of an unknown function in its corresponding RKHS. In response, we propose an algorithm that estimates an upper bound on the RKHS norm of an unknown function from data and investigate its theoretical properties. Moreover, akin to Lipschitz-based methods, we treat the RKHS norm as a local rather than a global object, and thus reduce conservatism. Integrating the RKHS norm estimation and the local interpretation of the RKHS norm into a safe BO algorithm yields PACSBO, an algorithm for probably approximately correct safe Bayesian optimization, for which we provide numerical and hardware experiments that demonstrate its applicability and benefits over popular safe BO algorithms.
Variation of Camera Parameters due to Common Physical Changes in Focal Length and Camera Pose
Venue: arXiv cs.CV 2024
Year: 2024

Accurate calibration of camera intrinsic parameters is crucial to various computer vision-based applications in the fields of intelligent systems, autonomous vehicles, etc. However, existing calibration schemes are incompetent for finding general trend of the variation of camera parameters due to common physical changes. In this paper, it is demonstrated that major and minor variations due to changes in focal length and camera pose, respectively, can be identified with a recently proposed calibration method. It is readily observable from the experimental results that the former variations have different trends (directions) of principal point deviation for different types of camera, possibly due to different internal lens configurations, while the latter have very similar trends in the deviation which is most likely due to direction of gravity. Finally, to confirm the validity of such unprecedented findings, 3D to 2D reprojection errors are compared for different methods of camera calibration.
Space module with gyroscope and accelerometer integration
Venue: arXiv cs.HC 2024
Year: 2024

MEIGA is a module specially designed for people with tetraplegia or anyone who has very limited movement capacity in their upper limbs. MEIGA converts the user's head movements into mouse movements. To simulate keystrokes, it uses blinking, reading the movement of the cheek that occurs with it. The performance, speed of movement of the mouse and its precision are practically equivalent to their respective measurements using the hand.
MobileIQA: Exploiting Mobile-level Diverse Opinion Network For No-Reference Image Quality Assessment Using Knowledge Distillation
Venue: arXiv cs.CV 2024
Year: 2024

With the rising demand for high-resolution (HR) images, No-Reference Image Quality Assessment (NR-IQA) gains more attention, as it can ecaluate image quality in real-time on mobile devices and enhance user experience. However, existing NR-IQA methods often resize or crop the HR images into small resolution, which leads to a loss of important details. And most of them are of high computational complexity, which hinders their application on mobile devices due to limited computational resources. To address these challenges, we propose MobileIQA, a novel approach that utilizes lightweight backbones to efficiently assess image quality while preserving image details through high-resolution input. MobileIQA employs the proposed multi-view attention learning (MAL) module to capture diverse opinions, simulating subjective opinions provided by different annotators during the dataset annotation process. The model uses a teacher model to guide the learning of a student model through knowledge distillation. This method significantly reduces computational complexity while maintaining high performance. Experiments demonstrate that MobileIQA outperforms novel IQA methods on evaluation metrics and computational efficiency. The code is available at https://github.com/chencn2020/MobileIQA.
Spatial-Aware Conformal Prediction for Trustworthy Hyperspectral Image Classification
Venue: arXiv cs.CV 2024
Year: 2024

Hyperspectral image (HSI) classification involves assigning unique labels to each pixel to identify various land cover categories. While deep classifiers have achieved high predictive accuracy in this field, they lack the ability to rigorously quantify confidence in their predictions. Quantifying the certainty of model predictions is crucial for the safe usage of predictive models, and this limitation restricts their application in critical contexts where the cost of prediction errors is significant. To support the safe deployment of HSI classifiers, we first provide a theoretical proof establishing the validity of the emerging uncertainty quantification technique, conformal prediction, in the context of HSI classification. We then propose a conformal procedure that equips any trained HSI classifier with trustworthy prediction sets, ensuring that these sets include the true labels with a user-specified probability (e.g., 95\%). Building on this foundation, we introduce Spatial-Aware Conformal Prediction (\texttt{SACP}), a conformal prediction framework specifically designed for HSI data. This method integrates essential spatial information inherent in HSIs by aggregating the non-conformity scores of pixels with high spatial correlation, which effectively enhances the efficiency of prediction sets. Both theoretical and empirical results validate the effectiveness of our proposed approach. The source code is available at \url{https://github.com/J4ckLiu/SACP}.
Contaminant Dispersion Simulation in a Digital Twin Framework for Critical Infrastructure Protection
Venue: arXiv cs.CE 2024
Year: 2024

A digital twin framework for rapid predictions of atmospheric contaminant dispersion is developed to support informed decision making in emergency situations. In an offline preparation phase, the geometry of a built environment is discretized with a finite element (FEM) mesh and a reduced-order model (ROM) of the steady-state incompressible Navier-Stokes equations is constructed for various wind conditions. Subsequently, the ROM provides a fast wind field estimate based on the current wind speed during the online phase. To support crisis management, several methodological building blocks are combined. Automatic FEM meshing of built environments and numerical flow solver capabilities enable fast forward-simulations of contaminant dispersion using the advection-diffusion equation as transport model. Further methods are integrated in the framework to address inverse problems such as contaminant source localization based on sparse concentration measurements. Additionally, the contaminant dispersion model is coupled with a continuum-based pedestrian crowd model to derive fast and safe evacuation routes for people seeking protection during contaminant dispersion emergencies. The interplay of these methods is demonstrated in two critical infrastructure protection (CIP) test cases. Based on simulated real world interaction (measurements, communication), this article demonstrates a full Measurement-Inversion-Prediction-Steering (MIPS) cycle including a Bayesian formulation of the inverse problem.
Generating Packet-Level Header Traces Using GNN-powered GAN
Venue: arXiv cs.NI 2024
Year: 2024

This study presents a novel method combining Graph Neural Networks (GNNs) and Generative Adversarial Networks (GANs) for generating packet-level header traces. By incorporating word2vec embeddings, this work significantly mitigates the dimensionality curse often associated with traditional one-hot encoding, thereby enhancing the training effectiveness of the model. Experimental results demonstrate that word2vec encoding captures semantic relationships between field values more effectively than one-hot encoding, improving the accuracy and naturalness of the generated data. Additionally, the introduction of GNNs further boosts the discriminator's ability to distinguish between real and synthetic data, leading to more realistic and diverse generated samples. The findings not only provide a new theoretical approach for network traffic data generation but also offer practical insights into improving data synthesis quality through enhanced feature representation and model architecture. Future research could focus on optimizing the integration of GNNs and GANs, reducing computational costs, and validating the model's generalizability on larger datasets. Exploring other encoding methods and model structure improvements may also yield new possibilities for network data generation. This research advances the field of data synthesis, with potential applications in network security and traffic analysis.