OpenRaiser/Intern-Atlas huggingface dataset:
Predicting Class Distribution Shift for Reliable Domain Adaptive Object Detection
Venue: arXiv cs.CV 2023
Year: 2023

Unsupervised Domain Adaptive Object Detection (UDA-OD) uses unlabelled data to improve the reliability of robotic vision systems in open-world environments. Previous approaches to UDA-OD based on self-training have been effective in overcoming changes in the general appearance of images. However, shifts in a robot's deployment environment can also impact the likelihood that different objects will occur, termed class distribution shift. Motivated by this, we propose a framework for explicitly addressing class distribution shift to improve pseudo-label reliability in self-training. Our approach uses the domain invariance and contextual understanding of a pre-trained joint vision and language model to predict the class distribution of unlabelled data. By aligning the class distribution of pseudo-labels with this prediction, we provide weak supervision of pseudo-label accuracy. To further account for low quality pseudo-labels early in self-training, we propose an approach to dynamically adjust the number of pseudo-labels per image based on model confidence. Our method outperforms state-of-the-art approaches on several benchmarks, including a 4.7 mAP improvement when facing challenging class distribution shift.
Learning-Based Defect Recognitions for Autonomous UAV Inspections
Venue: arXiv cs.RO 2023
Year: 2023

Automatic crack detection and segmentation play a significant role in the whole system of unmanned aerial vehicle inspections. In this paper, we have implemented a deep learning framework for crack detection based on classical network architectures including Alexnet, VGG, and Resnet. Moreover, inspired by the feature pyramid network architecture, a hierarchical convolutional neural network (CNN) deep learning framework which is efficient in crack segmentation is also proposed, and its performance of it is compared with other state-of-the-art network architecture. We have summarized the existing crack detection and segmentation datasets and established the largest existing benchmark dataset on the internet for crack detection and segmentation, which is open-sourced for the research community. Our feature pyramid crack segmentation network is tested on the benchmark dataset and gives satisfactory segmentation results. A framework for automatic unmanned aerial vehicle inspections is also proposed and will be established for the crack inspection tasks of various concrete structures. All our self-established datasets and codes are open-sourced at: https://github.com/KangchengLiu/Crack-Detection-and-Segmentation-Dataset-for-UAV-Inspection
Deep Transfer Tensor Factorization for Multi-View Learning
Venue: arXiv cs.CV 2023
Year: 2023

This paper studies the data sparsity problem in multi-view learning. To solve data sparsity problem in multiview ratings, we propose a generic architecture of deep transfer tensor factorization (DTTF) by integrating deep learning and cross-domain tensor factorization, where the side information is embedded to provide effective compensation for the tensor sparsity. Then we exhibit instantiation of our architecture by combining stacked denoising autoencoder (SDAE) and CANDECOMP/ PARAFAC (CP) tensor factorization in both source and target domains, where the side information of both users and items is tightly coupled with the sparse multi-view ratings and the latent factors are learned based on the joint optimization. We tightly couple the multi-view ratings and the side information to improve cross-domain tensor factorization based recommendations. Experimental results on real-world datasets demonstrate that our DTTF schemes outperform state-of-the-art methods on multi-view rating predictions.
Combined Location Online Weather Data: Easy-to-use Targeted Weather Analysis for Agriculture
Venue: arXiv cs.SI 2023
Year: 2023

The continuing effects of climate change require farmers and growers to have greater understanding of how these changes affect crop production. However, while climatic data is generally available to help provide much of that understanding, it can often be in a form not easy to digest. The proposed Combined Location Online Weather Data (CLOWD) framework is an easy-to-use online platform for analysing recent and historical weather data of any location within Australia at the click of a map. CLOWD requires no programming skills and operates in any HTML5 web browser on PC and mobile devices. It enables comparison between current and previous growing seasons over a range of environmental parameters, and can create a plain-English PDF report for offline use, using natural language generation (NLG). This paper details the platform, the design decisions taken and outlines how farmers and growers can use CLOWD to better understand current growing conditions. Prototypes of CLOWD are now online for PCs and smartphones.
Concerns in Software Development: A Systematic Mapping Study
Venue: arXiv cs.SE 2023
Year: 2023

Context: Successfully addressing stakeholder concerns that are related to software system development and operation is crucial to achieving development goals. The importance of using a systematic approach to addressing these concerns throughout the software development life cycle is growing as more and more systems are employed to handle critical tasks. Objective: The goal of this study is to provide an overview of addressing concerns across the software development life cycle. Method: A systematic mapping study was conducted using a pre-defined protocol. Four digital databases were searched for research literature and primary studies were selected after a three round selection process conducted by multiple researchers. Results: The extracted data are processed and the results are reported from different viewpoints. The results are also analyzed against our research goals. Conclusion: We show that there is a considerable variation in the use of terminologies and addressing concerns in different phases of the software development life cycle.
Surface-biased Multi-Level Context 3D Object Detection
Venue: arXiv cs.CV 2023
Year: 2023

Object detection in 3D point clouds is a crucial task in a range of computer vision applications including robotics, autonomous cars, and augmented reality. This work addresses the object detection task in 3D point clouds using a highly efficient, surface-biased, feature extraction method (wang2022rbgnet), that also captures contextual cues on multiple levels. We propose a 3D object detector that extracts accurate feature representations of object candidates and leverages self-attention on point patches, object candidates, and on the global scene in 3D scene. Self-attention is proven to be effective in encoding correlation information in 3D point clouds by (xie2020mlcvnet). While other 3D detectors focus on enhancing point cloud feature extraction by selectively obtaining more meaningful local features (wang2022rbgnet) where contextual information is overlooked. To this end, the proposed architecture uses ray-based surface-biased feature extraction and multi-level context encoding to outperform the state-of-the-art 3D object detector. In this work, 3D detection experiments are performed on scenes from the ScanNet dataset whereby the self-attention modules are introduced one after the other to isolate the effect of self-attention at each level.
Why BCIs work poorly with the patients who need them the most?
Venue: arXiv cs.HC 2023
Year: 2023

A major objective of Brain-Computer interfaces (BCI) is to restore communication and control in patients with severe motor impairments, like people with Locked-in syndrome. These patients are left only with limited eye and eyelid movements. However, they do not benefit from efficient BCI solutions, yet. Different signals can be used as commands for non-invasive BCI: mu and beta rhythm desynchronization, evoked potentials and slow cortical potentials. Whatever the signal, clinical studies show a dramatic loss of performance in severely impaired patients compared to healthy subjects. Interestingly, the control principle is always the same, namely the replacement of an impossible (overt) movement by a (covert) attentional command. Drawing from the premotor theory of attention, from neuroimaging findings about the functional anatomy of spatial attention, from clinical observations and from recent computational accounts of attention for both action and perception, we explore the hypothesis that these patients undergo negative plasticity that extends their impairment from overt to covert attentional processes.
Text2shape Deep Retrieval Model: Generating Initial Cases for Mechanical Part Redesign under the Context of Case-Based Reasoning
Venue: arXiv cs.AI 2023
Year: 2023

Retrieving the similar solutions from the historical case base for new design requirements is the first step in mechanical part redesign under the context of case-based reasoning. However, the manual retrieving method has the problem of low efficiency when the case base is large. Additionally, it is difficult for simple reasoning algorithms (e.g., rule-based reasoning, decision tree) to cover all the features in complicated design solutions. In this regard, a text2shape deep retrieval model is established in order to support text description-based mechanical part shapes retrieval, where the texts are for describing the structural features of the target mechanical parts. More specifically, feature engineering is applied to identify the key structural features of the target mechanical parts. Based on the identified key structural features, a training set of 1000 samples was constructed, where each sample consisted of a paragraph of text description of a group of structural features and the corresponding 3D shape of the structural features. RNN and 3D CNN algorithms were customized to build the text2shape deep retrieval model. Orthogonal experiments were used for modeling turning. Eventually, the highest accuracy of the model was 0.98; therefore, the model can be effective for retrieving initial cases for mechanical part redesign.
Detection and Segmentation of Pancreas using Morphological Snakes and Deep Convolutional Neural Networks
Venue: arXiv cs.CV 2023
Year: 2023

Pancreatic cancer is one of the deadliest types of cancer, with 25% of the diagnosed patients surviving for only one year and 6% of them for five. Computed tomography (CT) screening trials have played a key role in improving early detection of pancreatic cancer, which has shown significant improvement in patient survival rates. However, advanced analysis of such images often requires manual segmentation of the pancreas, which is a time-consuming task. Moreover, pancreas presents high variability in shape, while occupying only a very small area of the entire abdominal CT scans, which increases the complexity of the problem. The rapid development of deep learning can contribute to offering robust algorithms that provide inexpensive, accurate, and user-independent segmentation results that can guide the domain experts. This dissertation addresses this task by investigating a two-step approach for pancreas segmentation, by assisting the task with a prior rough localization or detection of pancreas. This rough localization of the pancreas is provided by an estimated probability map and the detection task is achieved by using the YOLOv4 deep learning algorithm. The segmentation task is tackled by a modified U-Net model applied on cropped data, as well as by using a morphological active contours algorithm. For comparison, the U-Net model was also applied on the full CT images, which provide a coarse pancreas segmentation to serve as reference. Experimental results of the detection network on the National Institutes of Health (NIH) dataset and the pancreas tumour task dataset within the Medical Segmentation Decathlon show 50.67% mean Average Precision. The best segmentation network achieved good segmentation results on the NIH dataset, reaching 67.67% Dice score.
Fixing Overconfidence in Dynamic Neural Networks
Venue: arXiv cs.LG 2023
Year: 2023

Dynamic neural networks are a recent technique that promises a remedy for the increasing size of modern deep learning models by dynamically adapting their computational cost to the difficulty of the inputs. In this way, the model can adjust to a limited computational budget. However, the poor quality of uncertainty estimates in deep learning models makes it difficult to distinguish between hard and easy samples. To address this challenge, we present a computationally efficient approach for post-hoc uncertainty quantification in dynamic neural networks. We show that adequately quantifying and accounting for both aleatoric and epistemic uncertainty through a probabilistic treatment of the last layers improves the predictive performance and aids decision-making when determining the computational budget. In the experiments, we show improvements on CIFAR-100, ImageNet, and Caltech-256 in terms of accuracy, capturing uncertainty, and calibration error.
Right-Adjoints for Datalog Programs, and Homomorphism Dualities over Restricted Classes
Venue: arXiv cs.LO 2023
Year: 2023

A Datalog program can be viewed as a syntactic specification of a functor from database instances over some schema to database instances over another schema. The same holds more generally for $\exists$Datalog. We establish large classes of Datalog and $\exists$Datalog programs for which the corresponding functor admits a generalized right-adjoint. We employ these results to obtain new insights into the existence of, and methods for constructing, homomorphism dualities within restricted classes of instances. We also derive new results regarding the existence of uniquely characterizing data examples for database queries.
Semantic Image Segmentation: Two Decades of Research
Venue: arXiv cs.CV 2023
Year: 2023

Semantic image segmentation (SiS) plays a fundamental role in a broad variety of computer vision applications, providing key information for the global understanding of an image. This survey is an effort to summarize two decades of research in the field of SiS, where we propose a literature review of solutions starting from early historical methods followed by an overview of more recent deep learning methods including the latest trend of using transformers. We complement the review by discussing particular cases of the weak supervision and side machine learning techniques that can be used to improve the semantic segmentation such as curriculum, incremental or self-supervised learning. State-of-the-art SiS models rely on a large amount of annotated samples, which are more expensive to obtain than labels for tasks such as image classification. Since unlabeled data is instead significantly cheaper to obtain, it is not surprising that Unsupervised Domain Adaptation (UDA) reached a broad success within the semantic segmentation community. Therefore, a second core contribution of this book is to summarize five years of a rapidly growing field, Domain Adaptation for Semantic Image Segmentation (DASiS) which embraces the importance of semantic segmentation itself and a critical need of adapting segmentation models to new environments. In addition to providing a comprehensive survey on DASiS techniques, we unveil also newer trends such as multi-domain learning, domain generalization, domain incremental learning, test-time adaptation and source-free domain adaptation. Finally, we conclude this survey by describing datasets and benchmarks most widely used in SiS and DASiS and briefly discuss related tasks such as instance and panoptic image segmentation, as well as applications such as medical image segmentation.
Transcendence Certificates for D-finite Functions
Venue: arXiv cs.SC 2023
Year: 2023

Although in theory we can decide whether a given D-finite function is transcendental, transcendence proofs remain a challenge in practice. Typically, transcendence is certified by checking certain incomplete sufficient conditions. In this paper we propose an additional such condition which catches some cases on which other tests fail.
Mixed Multi-Model Semantic Interaction for Graph-based Narrative Visualizations
Venue: arXiv cs.HC 2023
Year: 2023

Narrative sensemaking is an essential part of understanding sequential data. Narrative maps are a visual representation model that can assist analysts to understand narratives. In this work, we present a semantic interaction (SI) framework for narrative maps that can support analysts through their sensemaking process. In contrast to traditional SI systems which rely on dimensionality reduction and work on a projection space, our approach has an additional abstraction layer -- the structure space -- that builds upon the projection space and encodes the narrative in a discrete structure. This extra layer introduces additional challenges that must be addressed when integrating SI with the narrative extraction pipeline. We address these challenges by presenting the general concept of Mixed Multi-Model Semantic Interaction (3MSI) -- an SI pipeline, where the highest-level model corresponds to an abstract discrete structure and the lower-level models are continuous. To evaluate the performance of our 3MSI models for narrative maps, we present a quantitative simulation-based evaluation and a qualitative evaluation with case studies and expert feedback. We find that our SI system can model the analysts' intent and support incremental formalism for narrative maps.
An 818-TOPS/W CSNR-31dB SQNR-45dB 10-bit Capacitor-Reconfiguring Computing-in-Memory Macro with Software-Analog Co-Design for Transformers
Venue: arXiv cs.AR 2023
Year: 2023

Transformer inference requires high compute accuracy; achieving this using analog CIMs has been difficult due to inherent computational errors. To overcome this challenge, we propose a Capacitor-Reconfiguring CIM (CR-CIM) to realize high compute accuracy analog CIM with a 10-bit ADC attaining high-area/power efficiency. CR-CIM reconfigures its capacitor array to serve dual purposes: for computation and ADC conversion, achieving significant area savings. Furthermore, CR-CIMs eliminate signal attenuation by keeping the signal charge stationary during operation, leading to a 4x improvement in comparator energy efficiency. We also propose a software-analog co-design technique integrating majority voting into the 10-bit ADC to dynamically optimize the CIM noise performance based on the running layer to further save inference power. Our CR-CIM achieves the highest compute-accuracy for analog CIMs, and the power efficiency of 818 TOPS/W is competitive with the state-of-the-art. Furthermore, the FoM considering SQNR and CSNR is 2.3x and 1.5x better than previous works, respectively. Vision Transformer (ViT) inference is achieved and realizes a highest CIFAR10 accuracy of 95.8% for analog CIMs.
Fast Real-Time Shading for Polygonal Hair
Venue: arXiv cs.GR 2023
Year: 2023

Though a lot of improvement has been made to hair rendering techniques in the recent years, realistic rendering of hair remains a challenge, especially in real time. In this paper, we propose a fast technique to approximate the shading of hair lighted by an environment map, direct lighting or a global illumination system, without having to render deep opacity maps or requiring additional artistic work.
Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learning
Venue: arXiv cs.LG 2023
Year: 2023

Tomorrow's robots will need to distinguish useful information from noise when performing different tasks. A household robot for instance may continuously receive a plethora of information about the home, but needs to focus on just a small subset to successfully execute its current chore. Filtering distracting inputs that contain irrelevant data has received little attention in the reinforcement learning literature. To start resolving this, we formulate a problem setting in reinforcement learning called the $\textit{extremely noisy environment}$ (ENE), where up to $99\%$ of the input features are pure noise. Agents need to detect which features provide task-relevant information about the state of the environment. Consequently, we propose a new method termed $\textit{Automatic Noise Filtering}$ (ANF), which uses the principles of dynamic sparse training in synergy with various deep reinforcement learning algorithms. The sparse input layer learns to focus its connectivity on task-relevant features, such that ANF-SAC and ANF-TD3 outperform standard SAC and TD3 by a large margin, while using up to $95\%$ fewer weights. Furthermore, we devise a transfer learning setting for ENEs, by permuting all features of the environment after 1M timesteps to simulate the fact that other information sources can become relevant as the world evolves. Again, ANF surpasses the baselines in final performance and sample complexity. Our code is available at https://github.com/bramgrooten/automatic-noise-filtering
Gradient-Based Automated Iterative Recovery for Parameter-Efficient Tuning
Venue: arXiv cs.CL 2023
Year: 2023

Pretrained large language models (LLMs) are able to solve a wide variety of tasks through transfer learning. Various explainability methods have been developed to investigate their decision making process. TracIn (Pruthi et al., 2020) is one such gradient-based method which explains model inferences based on the influence of training examples. In this paper, we explore the use of TracIn to improve model performance in the parameter-efficient tuning (PET) setting. We develop conversational safety classifiers via the prompt-tuning PET method and show how the unique characteristics of the PET regime enable TracIn to identify the cause for certain misclassifications by LLMs. We develop a new methodology for using gradient-based explainability techniques to improve model performance, G-BAIR: gradient-based automated iterative recovery. We show that G-BAIR can recover LLM performance on benchmarks after manually corrupting training labels. This suggests that influence methods like TracIn can be used to automatically perform data cleaning, and introduces the potential for interactive debugging and relabeling for PET-based transfer learning methods.
UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling
Venue: arXiv cs.CV 2023
Year: 2023

Large-scale vision-language pre-trained models have shown promising transferability to various downstream tasks. As the size of these foundation models and the number of downstream tasks grow, the standard full fine-tuning paradigm becomes unsustainable due to heavy computational and storage costs. This paper proposes UniAdapter, which unifies unimodal and multimodal adapters for parameter-efficient cross-modal adaptation on pre-trained vision-language models. Specifically, adapters are distributed to different modalities and their interactions, with the total number of tunable parameters reduced by partial weight sharing. The unified and knowledge-sharing design enables powerful cross-modal representations that can benefit various downstream tasks, requiring only 1.0%-2.0% tunable parameters of the pre-trained model. Extensive experiments on 6 cross-modal downstream benchmarks (including video-text retrieval, image-text retrieval, VideoQA, and VQA) show that in most cases, UniAdapter not only outperforms the state-of-the-arts, but even beats the full fine-tuning strategy. Particularly, on the MSRVTT retrieval task, UniAdapter achieves 49.7% recall@1 with 2.2% model parameters, outperforming the latest competitors by 2.0%. The code and models are available at https://github.com/RERV/UniAdapter.