Reviewerly/exHarmony huggingface dataset:
MS-UNet: Swin Transformer U-Net with Multi-scale Nested Decoder for Medical Image Segmentation with Small Training Data

We propose a novel U-Net model named MS-UNet for the medical image segmentation task in this study. Instead of the single-layer U-Net decoder structure used in Swin-UNet and TransUnet, we specifically design a multi-scale nested decoder based on the Swin Transformer for U-Net. The new framework is proposed based on the observation that the single-layer decoder structure of U-Net is too “thin” to exploit enough information, resulting in large semantic differences between the encoder and decoder parts. Things get worse if the number of training sets of data is not sufficiently large, which is common in medical image processing tasks where annotated data are more difficult to obtain than other tasks. Overall, the proposed multi-scale nested decoder structure allows the feature mapping between the decoder and encoder to be semantically closer, thus enabling the network to learn more detailed features. Experiment results show that MS-UNet could effectively improve the network performance with more efficient feature learning capability and exhibit more advanced performance, especially in the extreme case with a small amount of training data. The code is publicly available at: https://github.com/HH446/MS-UNet .
Online training and pruning of photonic neural networks

Photonic neural networks have unique weight-actuating mechanisms and manufacturing variations, resulting in a suboptimal performance by conventional offline training. By incorporating a power-pruning regularization term in the loss function, we demonstrate an online training method that can overcome manufacturing errors and minimize power consumption.
Only Classification Head Is Sufficient for Medical Image Segmentation

Medical image segmentation is a pivotal research domain that has garnered widespread attention in contemporary medical diagnostics. In pursuit of enhancing network efficacy, researchers have taken great efforts to develop various well-designed decoders. Unfortunately, due to the limited medical training data, the issues of underfitting and overfitting frequently arise. To this end, we undertake plentiful experiments to decouple the encoder and decoder components, and obtain a critical finding that excessively complex decoders impede the encoder’s potentiality of feature extraction. Inspired by some remarkable image generation work, we devise a straightforward segmentation network, which incorporates a pre-trained encoder backbone network and a pixel classification head. Our network not only ensures adequate feature decoding ability but also maximizes feature representation capability of the backbone. Experimental results on four datasets of three tasks show the outstanding performance against the state-of-the-art methods. The source code will be publicly available at https://github.com/wei-hongbin/CHNet
Towards Scalable Automatic Modulation Classification via Meta-Learning

Driven by recent technological breakthroughs in deep learning (DL), many recent automatic modulation classification (AMC) methods utilize deep networks to classify the type of modulation in the incoming signal at the receiver. However, existing DL-based approaches suffer from limited scalability, especially for unseen modulations or input signals from new environments not used in training the DL model, thus not ready for real-world systems such as software defined radio devices. In this paper, we introduce a scalable AMC scheme that provides flexibility for new modulations and adaptability to input signals with diverse configurations. We propose a meta-learning framework based on few-shot learning (FSL) to acquire general knowledge and a learning method for AMC tasks. This approach allows the model to recognize new unseen modulations by learning with only a very small number of samples, without requiring the entire model to be retrained. Additionally, we enhance the scalability of the classifier by leveraging a transformer-based encoder, enabling efficient processing of input signals with varying configurations. Extensive evaluations demonstrate that the proposed AMC method outperforms existing techniques across all signal-to-noise ratios (SNRs) on RadioML2018.01A dataset.
InstaTune: Instantaneous Neural Architecture Search During Fine-Tuning

One-Shot Neural Architecture Search (NAS) algorithms often rely on training a hardware agnostic super-network for a domain specific task. Optimal sub-networks are then extracted from the trained super-network for different hardware platforms. However, training super-networks from scratch can be extremely time consuming and compute intensive especially for large models that rely on a two-stage training process of pre-training and fine-tuning. State of the art pre-trained models are available for a wide range of tasks, but their large sizes significantly limits their applicability on various hardware platforms. We propose InstaTune, a method that leverages off-the-shelf pre-trained weights for large models and generates a super-network during the fine-tuning stage. InstaTune has multiple benefits. Firstly, since the process happens during fine-tuning, it minimizes the overall time and compute resources required for NAS. Secondly, the sub-networks extracted are optimized for the target task, unlike prior work that optimizes on the pre-training objective. Finally, InstaTune is easy to "plug and play" in existing frameworks. By using multi-objective evolutionary search algorithms along with lightly trained predictors, we find Pareto-optimal sub-networks that outperform their respective baselines across different performance objectives such as accuracy and MACs. Specifically, we demonstrate that our approach performs well across both unimodal (ViT and BERT) and multi-modal (BEiT-3) transformer based architectures.
ScrollNet: Dynamic Weight Importance for Continual Learning

The principle underlying most existing continual learning (CL) methods is to prioritize stability by penalizing changes in parameters crucial to old tasks, while allowing for plasticity in other parameters. The importance of weights for each task can be determined either explicitly through learning a task-specific mask during training (e.g., parameter isolation-based approaches) or implicitly by introducing a regularization term (e.g., regularization-based approaches). However, all these methods assume that the importance of weights for each task is unknown prior to data exposure. In this paper, we propose ScrollNet as a scrolling neural network for continual learning. ScrollNet can be seen as a dynamic network that assigns the ranking of weight importance for each task before data exposure, thus achieving a more favorable stability-plasticity tradeoff during sequential task learning by reassigning this ranking for different tasks. Additionally, we demonstrate that Scroll-Net can be combined with various CL methods, including regularization-based and replay-based approaches. Experimental results on CIFAR100 and TinyImagenet datasets show the effectiveness of our proposed method.
The Robust Semantic Segmentation UNCV2023 Challenge Results

This paper outlines the winning solutions employed in addressing the MUAD uncertainty quantification challenge held at ICCV 2023. The challenge was centered around semantic segmentation in urban environments, with a particular focus on natural adversarial scenarios. The report presents the results of 19 submitted entries, with numerous techniques drawing inspiration from cutting-edge uncertainty quantification methodologies presented at prominent conferences in the fields of computer vision and machine learning and journals over the past few years. Within this document, the challenge is introduced, shedding light on its purpose and objectives, which primarily revolved around enhancing the robustness of semantic segmentation in urban scenes under varying natural adversarial conditions. The report then delves into the top-performing solutions. Moreover, the document aims to provide a comprehensive overview of the diverse solutions deployed by all participants. By doing so, it seeks to offer readers a deeper insight into the array of strategies that can be leveraged to effectively handle the inherent uncertainties associated with autonomous driving and semantic segmentation, especially within urban environments.
Alignment and Generation Adapter for Efficient Video-text Understanding

Pre-trained models have demonstrated considerable performance, especially in enhancing cross-modal understanding between videos and text. However, fine-tuning them at scale becomes costly and poses challenges for adapting to various downstream tasks. To tackle these challenges, we propose the Alignment-generation Adapter (AGAdapter), establishing semantic coherence between alignment and generation models for efficient video-text adaptation across multiple tasks simultaneously. We propose an alignment adapter with knowledge-sharing to adapt the frozen CLIP model for fine-grained video-language interaction. Additionally, we introduce the generation adapter with prompt tuning to leverage the large language model for captioning. Furthermore, we introduce instruction joint tuning, combining textual and cross-modal instructions, to capture detailed descriptions. Our AGAdapter achieves state-of-the-art performance on video-text retrieval and video captioning tasks, including two benchmarks, MSR-VTT and ActivityNet.
TransInpaint: Transformer-based Image Inpainting with Context Adaptation

Image inpainting aims to generate realistic content for missing regions of an image. Existing methods often struggle to produce visually coherent content for missing regions of an image, which results in blurry or distorted structures around the damaged areas. These methods rely on surrounding texture information and have difficulty in generating content that harmonizes well with the broader context of the image. To address this limitation, we propose a novel model that generates plausible content for missing regions while ensuring that the generated content is consistent with the overall context of the original image. In particular, we introduce a novel context-adaptive transformer for image inpainting (TransInpaint) that relies on the visible content and the position of the missing regions. Additionally, we design a texture enhancement network that combines skip features from the encoder with the coarse features produced by the generator, yielding a more comprehensive and robust representation of image content. Based on extensive evaluations on challenging datasets, our proposed TransInpaint outperforms the cutting-edge generative models for image inpainting in terms of quality, textures, and structures.
Dual Prediction Model and Improvement of Coal Ash Melting Point Based on SMA and Optimal Input Parameters

In order to improve the accuracy of coal ash melting point prediction, the original model was improved in terms of modeling method, input parameters, and generalization ability. Firstly, to address the problem of partial distortion of the ash melting point data, a dual prediction model consisting of Model I, which classifies the ash melting point, and Model II, which accurately predicts the ash melting point, is established by using a support vector machine, and the distorted data are modeled separately from the real data. Secondly, the optimal input parameters of the model are obtained by comparing the prediction results of 256 different input parameters, and the hyperparameters of the model are searched for by the slime mould algorithm (SMA) search. Finally, in order to enhance the generalization ability of the model, an evaluation index considering overfitting is proposed to improve the model. The results show that compared with the existing single prediction model, the mean square error (MSE) of the dual prediction model is reduced by 6.84 %; based on this, the optimal selection of input parameters reduces the MSE by 24.57%; only relying on the SMA to search for hyperparameters leads to overfitting of the model and deterioration of prediction effect; improvement of the model using evaluation index that consider overfitting results in a 4.16% reduction in mean squared error compared to the pre-improvement period. In the end, the improved model reduced the mean square error by 32.31 % compared with the original model, which greatly improved the prediction accuracy.
Learning to Prompt CLIP for Monocular Depth Estimation: Exploring the Limits of Human Language

CLIP is a significant vision-and-language training framework that has shown surprisingly general understanding of the world, with good performance in many openended tasks with little or no additional training. A recent technique has used CLIP to perform 0-shot Monocular Depth Estimation (MDE) by using depth-related prompts, but the use of human language in these prompts presents an unnecessary human bias. In this work, we use continuous learnable tokens in place of discrete human-language words to shed light on the problem. We achieve a significant boost in performance, and find that the learned tokens do not map neatly to depth-related human language, implying that CLIP’s concept of depth is not succinctly expressible in human language. We posit that this may extend to other CLIP concepts, and believe that this finding will spark further research into both the use and interpretation of non-linguistic tokens in all open-ended scene interpretation tasks. Code is available at https://github.com/DylanAuty/PromptLearningCLIP-MDE
Decision Tree Detection Method for Overload Fault of Distribution Network Lines Under Load Disturbance

The conventional decision tree detection matrix of distribution network line overload fault is mostly unidirectional, and the detection range is small, which leads to the time-consuming extension of the final overload fault detection. Therefore, the design and verification analysis of the decision tree detection method of distribution network line overload fault under load disturbance are proposed. According to the current measurement, firstly, the overload fault feature is extracted. Under the load disturbance condition, the range of fault detection is expanded by multi-step method, and the multi-step overload fault detection matrix is designed. Based on this, the decision tree distribution network line overload fault detection model is constructed, and the fault detection processing is realized by adaptive cross overload locking. The test results show that for the six selected overload fault detection points, the final overload fault detection time is controlled 0.3s, which shows that with the help of decision tree principle, the currently designed overload fault detection method is more targeted and stable, and the fault identification and detection efficiency is higher, and the overall detection effect is obviously improved, and the error is controllable, which has practical application value.
Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models

Video Question Answering (VideoQA) has been significantly advanced from the scaling of recent Large Language Models (LLMs). The key idea is to convert the visual information into the language feature space so that the capacity of LLMs can be fully exploited. Existing VideoQA methods typically take two paradigms: (1) learning cross-modal alignment, and (2) using an off-the-shelf captioning model to describe the visual data. However, the first design needs costly training on many extra multi-modal data, whilst the second is further limited by limited domain generalization. To address these limitations, a simple yet effective Retrieving-to-Answer (R2A) framework is proposed. Given an input video, R2A first retrieves a set of semantically similar texts from a generic text corpus using a pre-trained multi-modal model (e.g., CLIP). With both the question and the retrieved texts, a LLM (e.g., DeBERTa) can be directly used to yield a desired answer. Without the need for cross-modal fine-tuning, R2A allows for all the key components (e.g., LLM, retrieval model, and text corpus) to plug-and-play. Extensive experiments on several VideoQA benchmarks show that despite with 1.3B parameters and no fine-tuning, our R2A can outperform the 61× larger Flamingo-80B model [1] even additionally trained on nearly 2.1B multi-modal data.
Memory-Augmented Spatial-Temporal Consistency Network for Video Anomaly Detection

Video anomaly detection (VAD) in intelligent surveillance systems is a crucial yet highly challenging task. Since appearance and motion information is vital for identifying anomalies, existing unsupervised VAD methods usually learn normality from them. However, these approaches tend to consider appearance and motion separately or simply integrate them while ignoring the consistency between them, resulting in sub-optimal performance. To address this problem, we propose a Memory-Augmented Spatial-Temporal Consistency Network, aiming to model the latent consistency between spatial appearance and temporal motion by learning the unified spatiotemporal representation. Additionally, we introduce a spatial-temporal memory fusion module to record spatial and temporal prototypes of regular patterns from the unified spatiotemporal representation, increasing the gap between normal and abnormal events in the feature space. Experimental results on three benchmarks demonstrate the effectiveness of the spatial-temporal consistency for VAD tasks. Our method performs comparably to the state-of-the-art methods with AUCs of 97.6%, 89.3%, and 73.3% on the UCSD Ped2, CUHK Avenue, and ShanghaiTech datasets, respectively.
Self-Supervised Anomaly Detection from Anomalous Training Data via Iterative Latent Token Masking

Anomaly detection and segmentation pose an important task across sectors ranging from medical imaging analysis to industry quality control. However, current unsupervised approaches require training data to not contain any anomalies, a requirement that can be especially challenging in many medical imaging scenarios. In this paper, we propose Iterative Latent Token Masking, a self-supervised framework derived from a robust statistics point of view, translating an iterative model fitting with M-estimators to the task of anomaly detection. In doing so, this allows the training of unsupervised methods on datasets heavily contaminated with anomalous images. Our method stems from prior work on using Transformers, combined with a Vector Quantized-Variational Autoencoder, for anomaly detection, a method with state-of-the-art performance when trained on normal (non-anomalous) data. More importantly, we utilise the token masking capabilities of Transformers to filter out suspected anomalous tokens from each sample’s sequence in the training set in an iterative self-supervised process, thus overcoming the difficulties of highly anomalous training data. Our work also highlights shortfalls in current state-of-the-art self-supervised, self-trained and unsupervised models when faced with small proportions of anomalous training data. We evaluate our method on whole-body PET data in addition to showing its wider application in more common computer vision tasks such as the industrial MVTec Dataset. Using varying levels of anomalous training data, our method showcases a superior performance over several state-of-the-art models, drawing attention to the potential of this approach.
Slovak Question Answering Dataset Based on the Machine Translation of the Squad V2.0

Abstract This paper describes the process of building the first large-scale machinetranslated question answering dataset SQuAD-sk for the Slovak language. The dataset was automatically translated from the original English SQuAD v2.0 using the Marian neural machine translation together with the Helsinki-NLP Opus English-Slovak model. Moreover, we proposed an effective approach for the approximate search of the translated answer in the translated paragraph based on measuring their similarity using their word vectors. In this way, we obtained more than 92% of the translated questions and answers from the original English dataset. We then used this machine-translated dataset to train the Slovak question answering system by fine-tuning monolingual and multilingual BERT-based language models. The scores achieved by EM = 69.48% and F1 = 78.87% for the fine-tuned mBERT model show comparable results of question answering with recently published machinetranslated SQuAD datasets for other European languages.
Slovak Language Models for Basic Preprocessing Tasks in Python

Abstract We propose a Slovak language model for the spaCy library in Python. These models are easy-to-use for basic natural language processing tasks in a single package. The package contains several components for basic preprocessing tasks, such as tokenization, sentence boundary detection, syntactic parsing, lemmatization, named entity recognition, morphology analysis, and word vectors. It is based on the state-of-the-art monolingual SlovakBERT model. Named entity recognition is trained on a separate, publicly available WikiAnn database. The other statistical classifiers use a Slovak Dependency Treebank corpus. Morphological tags are compatible with the conventions of the Slovak National Corpus. The part of speech tags use conventions of the Universal Dependencies framework. We trained a separate word vector model on a web-based corpus. The training uses fastText with Floret modification. We present a series of experiments that confirm that the model performs similarly to other languages for all tasks. Training scripts and data are publicly available.
Multi-agent Perception via Co-attentive Communication Mechanism

Multi-agent collaborative perception has the potential to significantly enhance perception performance by facilitating the exchange of complementary information among agents through communication. Effective communication plays a crucial role in enabling agents to collaborate and exchange valuable information. However, traditional methods face challenges when it comes to communication scheduling. To address these issues, we propose the Co-Attentive Multi-agent Perception (CAMP) model, a learning-based framework for multi-agent collaborative perception. In CAMP, we propose a novel co-attentive scheduler to construct communication among agents from both pixel-level and feature-level perspectives. We conduct extensive experiments on a public dataset and the experimental results clearly illustrate the effectiveness of our proposed CAMP in the multi-agent perception task while using the least amount of bandwidth resources.
Improving the Adversarial Robustness of Object Detection with Contrastive Learning

Object detection plays a crucial role and has wide-ranging applications in computer vision. Nevertheless, object detectors are susceptible to adversarial examples. Some works have been presented to improve the adversarial robustness of object detectors, which, however, often come at the loss of some prediction accuracy. In this paper, we propose a novel adversarial training method that integrates the contrastive learning into the training process to reduce the loss of accuracy. Specifically, we add a contrastive learning module to the primary feature extraction backbone of the target object detector to extract contrastive features. During the training process, the contrastive loss and detection loss are used together to guide the training of detectors. Contrastive learning ensures that clean and adversarial examples are more clustered and are further away from decision boundaries in the high-level feature space, thus increasing the cost of adversarial examples crossing decision boundaries. Numerous experiments on PASCAL-VOC and MS-COCO have shown that our proposed method achieves significantly superior defense performance.