OpenRaiser/Intern-Atlas huggingface dataset:
Learning to Adapt Foundation Model DINOv2 for Capsule Endoscopy Diagnosis
Venue: arXiv cs.CV 2024
Year: 2024

Foundation models have become prominent in computer vision, achieving notable success in various tasks. However, their effectiveness largely depends on pre-training with extensive datasets. Applying foundation models directly to small datasets of capsule endoscopy images from scratch is challenging. Pre-training on broad, general vision datasets is crucial for successfully fine-tuning our model for specific tasks. In this work, we introduce a simplified approach called Adapt foundation models with a low-rank adaptation (LoRA) technique for easier customization. Our method, inspired by the DINOv2 foundation model, applies low-rank adaptation learning to tailor foundation models for capsule endoscopy diagnosis effectively. Unlike traditional fine-tuning methods, our strategy includes LoRA layers designed to absorb specific surgical domain knowledge. During the training process, we keep the main model (the backbone encoder) fixed and focus on optimizing the LoRA layers and the disease classification component. We tested our method on two publicly available datasets for capsule endoscopy disease classification. The results were impressive, with our model achieving 97.75% accuracy on the Kvasir-Capsule dataset and 98.81% on the Kvasirv2 dataset. Our solution demonstrates that foundation models can be adeptly adapted for capsule endoscopy diagnosis, highlighting that mere reliance on straightforward fine-tuning or pre-trained models from general computer vision tasks is inadequate for such specific applications.
Full reference point cloud quality assessment using support vector regression
Venue: arXiv cs.CV 2024
Year: 2024

Point clouds are a general format for representing realistic 3D objects in diverse 3D applications. Since point clouds have large data sizes, developing efficient point cloud compression methods is crucial. However, excessive compression leads to various distortions, which deteriorates the point cloud quality perceived by end users. Thus, establishing reliable point cloud quality assessment (PCQA) methods is essential as a benchmark to develop efficient compression methods. This paper presents an accurate full-reference point cloud quality assessment (FR-PCQA) method called full-reference quality assessment using support vector regression (FRSVR) for various types of degradations such as compression distortion, Gaussian noise, and down-sampling. The proposed method demonstrates accurate PCQA by integrating five FR-based metrics covering various types of errors (e.g., considering geometric distortion, color distortion, and point count) using support vector regression (SVR). Moreover, the proposed method achieves a superior trade-off between accuracy and calculation speed because it includes only the calculation of these five simple metrics and SVR, which can perform fast prediction. Experimental results with three types of open datasets show that the proposed method is more accurate than conventional FR-PCQA methods. In addition, the proposed method is faster than state-of-the-art methods that utilize complicated features such as curvature and multi-scale features. Thus, the proposed method provides excellent performance in terms of the accuracy of PCQA and processing speed. Our method is available from https://github.com/STAC-USC/FRSVR-PCQA.
Reward Schemes and Committee Sizes in Proof of Stake Governance
Venue: arXiv cs.GT 2024
Year: 2024

In this paper, we investigate the impact of reward schemes and committee sizes motivated by governance systems over blockchain communities. We introduce a model for elections with a binary outcome space where there is a ground truth (i.e., a "correct" outcome), and where stakeholders can only choose to delegate their voting power to a set of delegation representatives (DReps). Moreover, the effort (cost) invested by each DRep positively influences both (i) her ability to vote correctly and (ii) the total delegation that she attracts, thereby increasing her voting power. This model constitutes the natural counterpart of delegated proof-of-stake (PoS) protocols, where delegated stakes are used to elect the block builders. As a way to motivate the representatives to exert effort, a reward scheme can be used based on the delegation attracted by each DRep. We analyze both the game-theoretic aspects and the optimization counterpart of this model. Our primary focus is on selecting a committee that maximizes the probability of reaching the correct outcome, given a fixed monetary budget allocated for rewarding the delegates. Our findings provide insights into the design of effective reward mechanisms and optimal committee structures (i.e., how many DReps are enough) in these PoS-like governance systems.
A Theory of Interpretable Approximations
Venue: arXiv cs.LG 2024
Year: 2024

Can a deep neural network be approximated by a small decision tree based on simple features? This question and its variants are behind the growing demand for machine learning models that are *interpretable* by humans. In this work we study such questions by introducing *interpretable approximations*, a notion that captures the idea of approximating a target concept $c$ by a small aggregation of concepts from some base class $\mathcal{H}$. In particular, we consider the approximation of a binary concept $c$ by decision trees based on a simple class $\mathcal{H}$ (e.g., of bounded VC dimension), and use the tree depth as a measure of complexity. Our primary contribution is the following remarkable trichotomy. For any given pair of $\mathcal{H}$ and $c$, exactly one of these cases holds: (i) $c$ cannot be approximated by $\mathcal{H}$ with arbitrary accuracy; (ii) $c$ can be approximated by $\mathcal{H}$ with arbitrary accuracy, but there exists no universal rate that bounds the complexity of the approximations as a function of the accuracy; or (iii) there exists a constant $κ$ that depends only on $\mathcal{H}$ and $c$ such that, for *any* data distribution and *any* desired accuracy level, $c$ can be approximated by $\mathcal{H}$ with a complexity not exceeding $κ$. This taxonomy stands in stark contrast to the landscape of supervised classification, which offers a complex array of distribution-free and universally learnable scenarios. We show that, in the case of interpretable approximations, even a slightly nontrivial a-priori guarantee on the complexity of approximations implies approximations with constant (distribution-free and accuracy-free) complexity. We extend our trichotomy to classes $\mathcal{H}$ of unbounded VC dimension and give characterizations of interpretability based on the algebra generated by $\mathcal{H}$.
Finite-difference-informed graph network for solving steady-state incompressible flows on block-structured grids
Venue: arXiv cs.LG 2024
Year: 2024

Advances in deep learning have enabled physics-informed neural networks to solve partial differential equations. Numerical differentiation using the finite-difference (FD) method is efficient in physics-constrained designs, even in parameterized settings. In traditional computational fluid dynamics(CFD), body-fitted block-structured grids are often employed for complex flow cases when obtaining FD solutions. However, convolution operators in convolutional neural networks for FD are typically limited to single-block grids. To address this issue, \blueText{graphs and graph networks are used} to learn flow representations across multi-block-structured grids. \blueText{A graph convolution-based FD method (GC-FDM) is proposed} to train graph networks in a label-free physics-constrained manner, enabling differentiable FD operations on unstructured graph outputs. To demonstrate model performance from single- to multi-block-structured grids, \blueText{the parameterized steady incompressible Navier-Stokes equations are solved} for a lid-driven cavity flow and the flows around single and double circular cylinder configurations. When compared to a CFD solver under various boundary conditions, the proposed method achieves a relative error in velocity field predictions on the order of $10^{-3}$. Furthermore, the proposed method reduces training costs by approximately 20\% compared to a physics-informed neural network. \blueText{To} further verify the effectiveness of GC-FDM in multi-block processing, \blueText{a 30P30N airfoil geometry is considered} and the \blueText{predicted} results are reasonable compared with those given by CFD. \blueText{Finally, the applicability of GC-FDM to three-dimensional (3D) case is tested using a 3D cavity geometry.
Evaluating Open Access Advantages for Citations and Altmetrics (2011-21): A Dynamic and Evolving Relationship
Venue: arXiv cs.DL 2024
Year: 2024

Differences between the impacts of Open Access (OA) and non-OA research have been observed over a wide range of citation and altmetric indicators, usually finding an Open Access Advantage (OAA) within specific fields. However, science-wide analyses covering multiple years, indicators and disciplines are lacking. Using citation counts and six altmetrics for 38.7M articles published 2011-21, we compare OA and non-OA papers. The results show that there is no universal OAA across all disciplines or impact indicators: the OAA for citations tends to be lower for more recent papers, whereas the OAAs for news, blogs and Twitter are consistent across years and unrelated to volume of OA publications, whereas the OAAs for Wikipedia, patents and policy citations are more complex. These results support different hypotheses for different subjects and indicators. The evidence is consistent with OA accelerating research impact in the Medical & Health Sciences, Life Sciences and the Humanities; that increased visibility or discoverability is a factor in promoting the translation of research into socio-economic impact; and that OA is a factor in growing online engagement with research in some disciplines. OAAs are therefore complex, dynamic, multi-factorial and require considerable analysis to understand.
Large Reasoning Models for 3D Floorplanning in EDA: Learning from Imperfections
Venue: arXiv cs.LG 2024
Year: 2024

In this paper, we introduce Dreamweaver, which belongs to a new class of auto-regressive decision-making models known as large reasoning models (LRMs). Dreamweaver is designed to improve 3D floorplanning in electronic design automation (EDA) via an architecture that melds advancements in sequence-to-sequence reinforcement learning algorithms. A significant advantage of our approach is its ability to effectively reason over large discrete action spaces, which is essential for handling the numerous potential positions for various functional blocks in floorplanning. Additionally, Dreamweaver demonstrates strong performance even when trained on entirely random trajectories, showcasing its capacity to leverage sub-optimal or non-expert trajectories to enhance its results. This innovative approach contributes to streamlining the integrated circuit (IC) design flow and reducing the high computational costs typically associated with floorplanning. We evaluate its performance against a current state-of-the-art method, highlighting notable improvements.
How personality and memory of a robot can influence user modeling in Human-Robot Interaction
Venue: arXiv cs.RO 2024
Year: 2024

In recent years, robotics has evolved, placing robots in social contexts, and giving rise to Human-Robot Interaction (HRI). HRI aims to improve user satisfaction by designing autonomous social robots with user modeling functionalities and user-adapted interactions, storing data on people to achieve personalized interactions. Personality, a vital factor in human interactions, influences temperament, social preferences, and cognitive abilities. Despite much research on personality traits influencing human-robot interactions, little attention has been paid to the influence of the robot's personality on the user model. Personality can influence not only temperament and how people interact with each other but also what they remember about an interaction or the person they interact with. A robot's personality traits could therefore influence what it remembers about the user and thus modify the user model and the consequent interactions. However, no studies investigating such conditioning have been found. This paper addresses this gap by proposing distinct user models that reflect unique robotic personalities, exploring the interplay between individual traits, memory, and social interactions to replicate human-like processes, providing users with more engaging and natural experiences
Last-iterate Convergence Separation between Extra-gradient and Optimism in Constrained Periodic Games
Venue: arXiv cs.LG 2024
Year: 2024

Last-iterate behaviors of learning algorithms in repeated two-player zero-sum games have been extensively studied due to their wide applications in machine learning and related tasks. Typical algorithms that exhibit the last-iterate convergence property include optimistic and extra-gradient methods. However, most existing results establish these properties under the assumption that the game is time-independent. Recently, (Feng et al, 2023) studied the last-iterate behaviors of optimistic and extra-gradient methods in games with a time-varying payoff matrix, and proved that in an unconstrained periodic game, extra-gradient method converges to the equilibrium while optimistic method diverges. This finding challenges the conventional wisdom that these two methods are expected to behave similarly as they do in time-independent games. However, compared to unconstrained games, games with constrains are more common both in practical and theoretical studies. In this paper, we investigate the last-iterate behaviors of optimistic and extra-gradient methods in the constrained periodic games, demonstrating that similar separation results for last-iterate convergence also hold in this setting.
Scalable Temporal Motif Densest Subnetwork Discovery
Venue: arXiv cs.DS 2024
Year: 2024

Finding dense subnetworks, with density based on edges or more complex structures, such as subgraphs or $k$-cliques, is a fundamental algorithmic problem with many applications. While the problem has been studied extensively in static networks, much remains to be explored for temporal networks. In this work we introduce the novel problem of identifying the temporal motif densest subnetwork, i.e., the densest subnetwork with respect to temporal motifs, which are high-order patterns characterizing temporal networks. This problem significantly differs from analogous formulations for dense temporal (or static) subnetworks as these do not account for temporal motifs. Identifying temporal motifs is an extremely challenging task, and thus, efficient methods are required. To this end, we design two novel randomized approximation algorithms with rigorous probabilistic guarantees that provide high-quality solutions. We perform extensive experiments showing that our methods outperform baselines. Furthermore, our algorithms scale on networks with up to billions of temporal edges, while baselines cannot handle such large networks. We use our techniques to analyze a financial network and show that our formulation reveals important network structures, such as bursty temporal events and communities of users with similar interests.
Exploring the Impact of AI-generated Image Tools on Professional and Non-professional Users in the Art and Design Fields
Venue: arXiv cs.HC 2024
Year: 2024

The rapid proliferation of AI-generated image tools is transforming the art and design fields, challenging traditional notions of creativity and impacting both professional and non-professional users. For the purposes of this paper, we define 'professional users' as individuals who self-identified in our survey as 'artists,' 'designers,' 'filmmakers,' or 'art and design students,' and 'non-professional users' as individuals who self-identified as 'others.' This study explores how AI-generated image tools influence these different user groups. Through an online survey (N=380) comprising 173 professional users and 207 non-professional users, we examine differences in the utilization of AI tools, user satisfaction and challenges, applications in creative processes, perceptions and impacts, and acceptance levels. Our findings indicate persistent concerns about image quality, cost, and copyright issues. Additionally, the usage patterns of non-professional users suggest that AI tools have the potential to democratize creative processes, making art and design tasks more accessible to individuals without traditional expertise. This study provides insights into the needs of different user groups and offers recommendations for developing more user-centered AI tools, contributing to the broader discussion on the future of AI in the art and design fields.
DIEKAE: Difference Injection for Efficient Knowledge Augmentation and Editing of Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Pretrained Language Models (PLMs) store extensive knowledge within their weights, enabling them to recall vast amount of information. However, relying on this parametric knowledge brings some limitations such as outdated information or gaps in the training data. This work addresses these problems by distinguish between two separate solutions: knowledge editing and knowledge augmentation. We introduce Difference Injection for Efficient Knowledge Augmentation and Editing (DIEKÆ), a new method that decouples knowledge processing from the PLM (LLaMA2-7B, in particular) by adopting a series of encoders. These encoders handle external knowledge and inject it into the PLM layers, significantly reducing computational costs and improving performance of the PLM. We propose a novel training technique for these encoders that does not require back-propagation through the PLM, thus greatly reducing the memory and time required to train them. Our findings demonstrate how our method is faster and more efficient compared to multiple baselines in knowledge augmentation and editing during both training and inference. We have released our code and data at https://github.com/alessioGalatolo/DIEKAE.
MIND: Multimodal Shopping Intention Distillation from Large Vision-language Models for E-commerce Purchase Understanding
Venue: arXiv cs.CL 2024
Year: 2024

Improving user experience and providing personalized search results in E-commerce platforms heavily rely on understanding purchase intention. However, existing methods for acquiring large-scale intentions bank on distilling large language models with human annotation for verification. Such an approach tends to generate product-centric intentions, overlook valuable visual information from product images, and incurs high costs for scalability. To address these issues, we introduce MIND, a multimodal framework that allows Large Vision-Language Models (LVLMs) to infer purchase intentions from multimodal product metadata and prioritize human-centric ones. Using Amazon Review data, we apply MIND and create a multimodal intention knowledge base, which contains 1,264,441 million intentions derived from 126,142 co-buy shopping records across 107,215 products. Extensive human evaluations demonstrate the high plausibility and typicality of our obtained intentions and validate the effectiveness of our distillation framework and filtering mechanism. Additional experiments reveal that our obtained intentions significantly enhance large language models in two intention comprehension tasks.
Speech Emotion Recognition Using CNN and Its Use Case in Digital Healthcare
Venue: arXiv cs.SD 2024
Year: 2024

The process of identifying human emotion and affective states from speech is known as speech emotion recognition (SER). This is based on the observation that tone and pitch in the voice frequently convey underlying emotion. Speech recognition includes the ability to recognize emotions, which is becoming increasingly popular and in high demand. With the help of appropriate factors (such modalities, emotions, intensities, repetitions, etc.) found in the data, my research seeks to use the Convolutional Neural Network (CNN) to distinguish emotions from audio recordings and label them in accordance with the range of different emotions. I have developed a machine learning model to identify emotions from supplied audio files with the aid of machine learning methods. The evaluation is mostly focused on precision, recall, and F1 score, which are common machine learning metrics. To properly set up and train the machine learning framework, the main objective is to investigate the influence and cross-relation of all input and output parameters. To improve the ability to recognize intentions, a key condition for communication, I have evaluated emotions using my specialized machine learning algorithm via voice that would address the emotional state from voice with the help of digital healthcare, bridging the gap between human and artificial intelligence (AI).
Technique Report of CVPR 2024 PBDL Challenges
Venue: arXiv cs.CV 2024
Year: 2024

The intersection of physics-based vision and deep learning presents an exciting frontier for advancing computer vision technologies. By leveraging the principles of physics to inform and enhance deep learning models, we can develop more robust and accurate vision systems. Physics-based vision aims to invert the processes to recover scene properties such as shape, reflectance, light distribution, and medium properties from images. In recent years, deep learning has shown promising improvements for various vision tasks, and when combined with physics-based vision, these approaches can enhance the robustness and accuracy of vision systems. This technical report summarizes the outcomes of the Physics-Based Vision Meets Deep Learning (PBDL) 2024 challenge, held in CVPR 2024 workshop. The challenge consisted of eight tracks, focusing on Low-Light Enhancement and Detection as well as High Dynamic Range (HDR) Imaging. This report details the objectives, methodologies, and results of each track, highlighting the top-performing solutions and their innovative approaches.
SparseCL: Sparse Contrastive Learning for Contradiction Retrieval
Venue: arXiv cs.CL 2024
Year: 2024

Contradiction retrieval refers to identifying and extracting documents that explicitly disagree with or refute the content of a query, which is important to many downstream applications like fact checking and data cleaning. To retrieve contradiction argument to the query from large document corpora, existing methods such as similarity search and crossencoder models exhibit significant limitations. The former struggles to capture the essence of contradiction due to its inherent nature of favoring similarity, while the latter suffers from computational inefficiency, especially when the size of corpora is large. To address these challenges, we introduce a novel approach: SparseCL that leverages specially trained sentence embeddings designed to preserve subtle, contradictory nuances between sentences. Our method utilizes a combined metric of cosine similarity and a sparsity function to efficiently identify and retrieve documents that contradict a given query. This approach dramatically enhances the speed of contradiction detection by reducing the need for exhaustive document comparisons to simple vector calculations. We validate our model using the Arguana dataset, a benchmark dataset specifically geared towards contradiction retrieval, as well as synthetic contradictions generated from the MSMARCO and HotpotQA datasets using GPT-4. Our experiments demonstrate the efficacy of our approach not only in contradiction retrieval with more than 30% accuracy improvements on MSMARCO and HotpotQA across different model architectures but also in applications such as cleaning corrupted corpora to restore high-quality QA retrieval. This paper outlines a promising direction for improving the accuracy and efficiency of contradiction retrieval in large-scale text corpora.