OpenRaiser/Intern-Atlas huggingface dataset:
Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Manner
Venue: arXiv cs.CV 2024
Year: 2024

Advancements in Large Language Models (LLMs) inspire various strategies for integrating video modalities. A key approach is Video-LLMs, which incorporate an optimizable interface linking sophisticated video encoders to LLMs. However, due to computation and data limitations, these Video-LLMs are typically pre-trained to process only short videos, limiting their broader application for understanding longer video content. Additionally, fine-tuning Video-LLMs to handle longer videos is cost-prohibitive. Consequently, it becomes essential to explore the interpolation of Video-LLMs under a completely training-free setting. In this paper, we first identify the primary challenges in interpolating Video-LLMs: (1) the video encoder and modality alignment projector are fixed, preventing the integration of additional frames into Video-LLMs, and (2) the LLM backbone is limited in its content length capabilities, which complicates the processing of an increased number of video tokens. To address these challenges, we propose a specific INTerPolation method for Video-LLMs (INTP-Video-LLMs). We introduce an alternative video token rearrangement technique that circumvents limitations imposed by the fixed video encoder and alignment projector. Furthermore, we introduce a training-free LLM context window extension method to enable Video-LLMs to understand a correspondingly increased number of visual tokens.
The Era of Foundation Models in Medical Imaging is Approaching : A Scoping Review of the Clinical Value of Large-Scale Generative AI Applications in Radiology
Venue: arXiv cs.CV 2024
Year: 2024

Social problems stemming from the shortage of radiologists are intensifying, and artificial intelligence is being highlighted as a potential solution. Recently emerging large-scale generative AI has expanded from large language models (LLMs) to multi-modal models, showing potential to revolutionize the entire process of medical imaging. However, comprehensive reviews on their development status and future challenges are currently lacking. This scoping review systematically organizes existing literature on the clinical value of large-scale generative AI applications by following PCC guidelines. A systematic search was conducted across four databases: PubMed, EMbase, IEEE-Xplore, and Google Scholar, and 15 studies meeting the inclusion/exclusion criteria set by the researchers were reviewed. Most of these studies focused on improving the efficiency of report generation in specific parts of the interpretation process or on translating reports to aid patient understanding, with the latest studies extending to AI applications performing direct interpretations. All studies were quantitatively evaluated by clinicians, with most utilizing LLMs and only three employing multi-modal models. Both LLMs and multi-modal models showed excellent results in specific areas, but none yet outperformed radiologists in diagnostic performance. Most studies utilized GPT, with few using models specialized for the medical imaging domain. This study provides insights into the current state and limitations of large-scale generative AI-based applications in the medical imaging field, offering foundational data and suggesting that the era of medical imaging foundation models is on the horizon, which may fundamentally transform clinical practice in the near future.
Subject integration with spreadsheets -- Ignoring education is the greatest risk ever
Venue: arXiv cs.HC 2024
Year: 2024

Within the framework of Technological Pedagogical and Content Knowledge, subject integration is one possible solution for the introduction of meaningful digitalization and digitization in schools. This process incorporates that any school subject can be taught with digital support, informatics (computer) classes can be contextualized, and the gap between 'serious informatics' and 'digital literacy' can be minimized. The present paper details how three traditional Grade 3 tasks can be solved in spreadsheets, what skills, competencies, and computer science knowledge of both teachers and students can be developed. The solutions also reveal that analysing, understanding, planning, and discussing tasks is as important as the activity in the spreadsheets, which process plays a crucial role in the preparation of students for their future jobs.
Excel: Automated Ledger or Analytics IDE?
Venue: arXiv cs.CY 2024
Year: 2024

Since the inception of VisiCalc over four decades ago, spreadsheets have undergone a gradual transformation, evolving from simple ledger automation tools to the current state of Excel, which can be described as an Integrated Development Environment (IDE) for analytics. The slow evolution of Excel from an automation tool for ledgers to an IDE for analytics explains why many people have not noticed that Excel includes a fully functional database, an OLAP Engine, multiple statistical programming languages, multiple third-party software libraries, dynamic charts, and real time data connectors. The simplicity of accessing these multiple tools is a low-code framework controlled from the Excel tool that is effectively an IDE. Once we acknowledge Excel's shift from a desk top application to an IDE for analytics, the importance of establishing a comprehensive risk framework for managing this distinctive development environment becomes clear. In this paper we will explain how the current risk framework for spreadsheets needs to be expanded to manage the growing risks of using Excel as an IDE for analytics.
Semantic Meta-Split Learning: A TinyML Scheme for Few-Shot Wireless Image Classification
Venue: arXiv cs.CV 2024
Year: 2024

Semantic and goal-oriented (SGO) communication is an emerging technology that only transmits significant information for a given task. Semantic communication encounters many challenges, such as computational complexity at end users, availability of data, and privacy-preserving. This work presents a TinyML-based semantic communication framework for few-shot wireless image classification that integrates split-learning and meta-learning. We exploit split-learning to limit the computations performed by the end-users while ensuring privacy-preserving. In addition, meta-learning overcomes data availability concerns and speeds up training by utilizing similarly trained tasks. The proposed algorithm is tested using a data set of images of hand-written letters. In addition, we present an uncertainty analysis of the predictions using conformal prediction (CP) techniques. Simulation results show that the proposed Semantic-MSL outperforms conventional schemes by achieving 20 % gain on classification accuracy using fewer data points, yet less training energy consumption.
Large Language Model-Enhanced Interactive Agent for Public Education on Newborn Auricular Deformities
Venue: arXiv cs.CY 2024
Year: 2024

Auricular deformities are quite common in newborns with potential long-term negative effects of mental and even hearing problems.Early diagnosis and subsequent treatment are critical for the illness; yet they are missing most of the time due to lack of knowledge among parents. With the help of large language model of Ernie of Baidu Inc., we derive a realization of interactive agent. Firstly, it is intelligent enough to detect which type of auricular deformity corresponding to uploaded images, which is accomplished by PaddleDetection, with precision rate 75\%. Secondly, in terms of popularizing the knowledge of auricular deformities, the agent can give professional suggestions of the illness to parents. The above two effects are evaluated via tests on volunteers with control groups in the paper. The agent can reach parents with newborns as well as their pediatrician remotely via Internet in vast, rural areas with quality medical diagnosis capabilities and professional query-answering functions, which is good news for newborn auricular deformity and other illness that requires early intervention for better treatment.
pyrtklib: An open-source package for tightly coupled deep learning and GNSS integration for positioning in urban canyons
Venue: arXiv cs.LG 2024
Year: 2024

Artificial intelligence (AI) is revolutionizing numerous fields, with increasing applications in Global Navigation Satellite Systems (GNSS) positioning algorithms in intelligent transportation systems (ITS) via deep learning. However, a significant technological disparity exists as traditional GNSS algorithms are often developed in Fortran or C, contrasting with the Python-based implementation prevalent in deep learning tools. To address this discrepancy, this paper introduces pyrtklib, a Python binding for the widely utilized open-source GNSS tool, RTKLIB. This binding makes all RTKLIB functionalities accessible in Python, facilitating seamless integration. Moreover, we present a deep learning subsystem under pyrtklib, which is a novel deep learning framework that leverages pyrtklib to accurately predict weights and biases within the GNSS positioning process. The use of pyrtklib enables developers to easily and quickly prototype and implement deep learning-aided GNSS algorithms, showcasing its potential to enhance positioning accuracy significantly.
iCost: A Novel Instance Complexity Based Cost-Sensitive Learning Framework
Venue: arXiv cs.LG 2024
Year: 2024

Class imbalance in data presents significant challenges for classification tasks. It is fairly common and requires careful handling to obtain desirable performance. Traditional classification algorithms become biased toward the majority class. One way to alleviate the scenario is to make the classifiers cost-sensitive. This is achieved by assigning a higher misclassification cost to minority-class instances. One issue with this implementation is that all the minority-class instances are treated equally, and assigned with the same penalty value. However, the learning difficulties of all the instances are not the same. Instances that are located in the overlapping region or near the decision boundary are harder to classify, whereas those further away are easier. Without taking into consideration the instance complexity and naively weighting all the minority-class samples uniformly, results in an unwarranted bias and consequently, a higher number of misclassifications of the majority-class instances. This is undesirable and to overcome the situation, we propose a novel instance complexity-based cost-sensitive approach (termed 'iCost') in this study. We first categorize all the minority-class instances based on their difficulty level and then the instances are penalized accordingly. This ensures a more equitable instance weighting and prevents excessive penalization. The performance of the proposed approach is tested on 65 binary and 10 multiclass imbalanced datasets against the traditional cost-sensitive learning frameworks. A significant improvement in performance has been observed, demonstrating the effectiveness of the proposed strategy.
Nesting of Touching Polygons
Venue: arXiv cs.CG 2024
Year: 2024

Polygons are cycles embedded into the plane; their vertices are associated with $x$- and $y$-coordinates and the edges are straight lines. Here, we consider a set of polygons with pairwise non-overlapping interior that may touch along their boundaries. Ideas of the sweep line algorithm by Bajaj and Dey for non-touching polygons are adapted to accommodate polygons that share boundary points. The algorithms established here achieves a running time of $\mathcal{O}(n+N\log N)$, where $n$ is the total number of vertices and $N<n$ is the total number of "maximal outstretched segments" of all polygons. It is asymptotically optimal if the number of maximal outstretched segments per polygon is bounded. In particular, this is the case for convex polygons.
FedAT: Federated Adversarial Training for Distributed Insider Threat Detection
Venue: arXiv cs.CR 2024
Year: 2024

Insider threats usually occur from within the workplace, where the attacker is an entity closely associated with the organization. The sequence of actions the entities take on the resources to which they have access rights allows us to identify the insiders. Insider Threat Detection (ITD) using Machine Learning (ML)-based approaches gained attention in the last few years. However, most techniques employed centralized ML methods to perform such an ITD. Organizations operating from multiple locations cannot contribute to the centralized models as the data is generated from various locations. In particular, the user behavior data, which is the primary source of ITD, cannot be shared among the locations due to privacy concerns. Additionally, the data distributed across various locations result in extreme class imbalance due to the rarity of attacks. Federated Learning (FL), a distributed data modeling paradigm, gained much interest recently. However, FL-enabled ITD is not yet explored, and it still needs research to study the significant issues of its implementation in practical settings. As such, our work investigates an FL-enabled multiclass ITD paradigm that considers non-Independent and Identically Distributed (non-IID) data distribution to detect insider threats from different locations (clients) of an organization. Specifically, we propose a Federated Adversarial Training (FedAT) approach using a generative model to alleviate the extreme data skewness arising from the non-IID data distribution among the clients. Besides, we propose to utilize a Self-normalized Neural Network-based Multi-Layer Perceptron (SNN-MLP) model to improve ITD. We perform comprehensive experiments and compare the results with the benchmarks to manifest the enhanced performance of the proposed FedATdriven ITD scheme.
Usage of Virtual Reality in Combating Social Anxiety Disorders in Non-native English Speakers: A Survey
Venue: arXiv cs.HC 2024
Year: 2024

Social Anxiety Disorder (SAD) is a common yet underestimated mental health disorder. While non-native English speaker (NNES) students face public speaking, they are more likely to suffer some public speaking anxiety (PSA) due to linguistic and sociocultural differences \cite{cite1}. Virtual Reality (VR) technology has already benefitted social-emotional training. The core objective is to summarise the benefits and limitations of using VR technology to help NNES students practice and improve their public speaking skills. This is not a comprehensive survey of the literature. Instead, the selected papers are intended to reflect the current knowledge across various broad topics. Virtual Reality, Social Anxiety Disorder, Public Speaking Anxiety, English as a Second Language, and Non native English speakers are the keywords used for searching mainly in the Academic Search Complete (ASC) database. Compared with native English speaker (NES) students, NNES students have the potential to achieve better results when using VR technology for PSA social-emotional training.
Some new techniques to use in serial sparse Cholesky factorization algorithms
Venue: arXiv cs.MS 2024
Year: 2024

We present a new variant of serial right-looking supernodal sparse Cholesky factorization (RL). Our comparison of RL with the multifrontal method confirms that RL is simpler, slightly faster, and requires slightly less storage. The key to the rest of the work in this paper is recent work on reordering columns within supernodes so that the dense off-diagonal blocks in the factor matrix joining pairs of supernodes are fewer and larger. We present a second new variant of serial right-looking supernodal sparse Cholesky factorization (RLB), where this one is specifically designed to exploit fewer and larger off-diagonal blocks in the factor matrix obtained by reordering within supernodes. A key distinction found in RLB is that it uses no floating-point working storage and performs no assembly operations. Our key finding is that RLB is unequivocally faster than its competitors. Indeed, RLB is consistently, but modestly, faster than its competitors whenever Intel's MKL sequential BLAS are used. More importantly, RLB is substantially faster than its competitors whenever Intel's MKL multithreaded BLAS are used. Finally, RLB using the multithreaded BLAS achieves impressive speedups over RLB using the sequential BLAS.
BGDB: Bernoulli-Gaussian Decision Block with Improved Denoising Diffusion Probabilistic Models
Venue: arXiv cs.CV 2024
Year: 2024

Generative models can enhance discriminative classifiers by constructing complex feature spaces, thereby improving performance on intricate datasets. Conventional methods typically augment datasets with more detailed feature representations or increase dimensionality to make nonlinear data linearly separable. Utilizing a generative model solely for feature space processing falls short of unlocking its full potential within a classifier and typically lacks a solid theoretical foundation. We base our approach on a novel hypothesis: the probability information (logit) derived from a single model training can be used to generate the equivalent of multiple training sessions. Leveraging the central limit theorem, this synthesized probability information is anticipated to converge toward the true probability more accurately. To achieve this goal, we propose the Bernoulli-Gaussian Decision Block (BGDB), a novel module inspired by the Central Limit Theorem and the concept that the mean of multiple Bernoulli trials approximates the probability of success in a single trial. Specifically, we utilize Improved Denoising Diffusion Probabilistic Models (IDDPM) to model the probability of Bernoulli Trials. Our approach shifts the focus from reconstructing features to reconstructing logits, transforming the logit from a single iteration into logits analogous to those from multiple experiments. We provide the theoretical foundations of our approach through mathematical analysis and validate its effectiveness through experimental evaluation using various datasets for multiple imaging tasks, including both classification and segmentation.
Are Large Language Models Good Essay Graders?
Venue: arXiv cs.CL 2024
Year: 2024

We evaluate the effectiveness of Large Language Models (LLMs) in assessing essay quality, focusing on their alignment with human grading. More precisely, we evaluate ChatGPT and Llama in the Automated Essay Scoring (AES) task, a crucial natural language processing (NLP) application in Education. We consider both zero-shot and few-shot learning and different prompting approaches. We compare the numeric grade provided by the LLMs to human rater-provided scores utilizing the ASAP dataset, a well-known benchmark for the AES task. Our research reveals that both LLMs generally assign lower scores compared to those provided by the human raters; moreover, those scores do not correlate well with those provided by the humans. In particular, ChatGPT tends to be harsher and further misaligned with human evaluations than Llama. We also experiment with a number of essay features commonly used by previous AES methods, related to length, usage of connectives and transition words, and readability metrics, including the number of spelling and grammar mistakes. We find that, generally, none of these features correlates strongly with human or LLM scores. Finally, we report results on Llama 3, which are generally better across the board, as expected. Overall, while LLMs do not seem an adequate replacement for human grading, our results are somewhat encouraging for their use as a tool to assist humans in the grading of written essays in the future.
UniTabNet: Bridging Vision and Language Models for Enhanced Table Structure Recognition
Venue: arXiv cs.CV 2024
Year: 2024

In the digital era, table structure recognition technology is a critical tool for processing and analyzing large volumes of tabular data. Previous methods primarily focus on visual aspects of table structure recovery but often fail to effectively comprehend the textual semantics within tables, particularly for descriptive textual cells. In this paper, we introduce UniTabNet, a novel framework for table structure parsing based on the image-to-text model. UniTabNet employs a ``divide-and-conquer'' strategy, utilizing an image-to-text model to decouple table cells and integrating both physical and logical decoders to reconstruct the complete table structure. We further enhance our framework with the Vision Guider, which directs the model's focus towards pertinent areas, thereby boosting prediction accuracy. Additionally, we introduce the Language Guider to refine the model's capability to understand textual semantics in table images. Evaluated on prominent table structure datasets such as PubTabNet, PubTables1M, WTW, and iFLYTAB, UniTabNet achieves a new state-of-the-art performance, demonstrating the efficacy of our approach. The code will also be made publicly available.
Obstacle-Free Path Planning for Autonomous Drones Using Floyd Algorithm
Venue: arXiv cs.DS 2024
Year: 2024

This research investigates the efficiency of Floyd algorithm for obstacle-free path planning for autonomous aerial vehicles (UAVs) or drones. Floyd algorithm is used to generate the shortest paths for UAVs to fly from any place to the destination in a large-scale field with obstacles which UAVs cannot fly over. The simulation results demonstrated that Floyd algorithm effectively plans the shortest obstacle-free paths for UAVs to fly to a destination. It is verified that Floyd algorithm holds a time complexity of O(n3). This research revealed a correlation of a cubic polynomial relationship between the time cost and the size of the field, no correlation between the time cost and the number of obstacles, and no correlation between the time cost and the number of UAVs in the tested field. The applications of the research results are discussed in the paper as well.
Learning Visual Information Utility with PIXER
Venue: arXiv cs.CV 2024
Year: 2024

Accurate feature detection is fundamental for various computer vision tasks, including autonomous robotics, 3D reconstruction, medical imaging, and remote sensing. Despite advancements in enhancing the robustness of visual features, no existing method measures the utility of visual information before processing by specific feature-type algorithms. To address this gap, we introduce PIXER and the concept of "Featureness," which reflects the inherent interest and reliability of visual information for robust recognition, independent of any specific feature type. Leveraging a generalization on Bayesian learning, our approach quantifies both the probability and uncertainty of a pixel's contribution to robust visual utility in a single-shot process, avoiding costly operations such as Monte Carlo sampling and permitting customizable featureness definitions adaptable to a wide range of applications. We evaluate PIXER on visual odometry with featureness selectivity, achieving an average of 31% improvement in RMSE trajectory with 49% fewer features.
Beyond Skip Connection: Pooling and Unpooling Design for Elimination Singularities
Venue: arXiv cs.CV 2024
Year: 2024

Training deep Convolutional Neural Networks (CNNs) presents unique challenges, including the pervasive issue of elimination singularities, consistent deactivation of nodes leading to degenerate manifolds within the loss landscape. These singularities impede efficient learning by disrupting feature propagation. To mitigate this, we introduce Pool Skip, an architectural enhancement that strategically combines a Max Pooling, a Max Unpooling, a 3 times 3 convolution, and a skip connection. This configuration helps stabilize the training process and maintain feature integrity across layers. We also propose the Weight Inertia hypothesis, which underpins the development of Pool Skip, providing theoretical insights into mitigating degradation caused by elimination singularities through dimensional and affine compensation. We evaluate our method on a variety of benchmarks, focusing on both 2D natural and 3D medical imaging applications, including tasks such as classification and segmentation. Our findings highlight Pool Skip's effectiveness in facilitating more robust CNN training and improving model performance.