Reviewerly/exHarmony huggingface dataset:
Enhancing Classification Accuracy on Limited Data via Unconditional GAN

Despite significant advances in Deep Neural Networks (DNNs), these models often fall short in real-world scenarios, particularly when faced with a scarcity of training data. In this paper, we introduce a novel method that capitalizes on the power of Generative Adversarial Networks (GANs) to enhance performance in image classification tasks. Our approach specifically involves training the classifier by enforcing a consistency rule across generated unlabeled data synthesized from unconditional GANs. Through the implementation of our proposed methodology, we observed a substantial increase in accuracy - approximately 8.68% on the CIFAR-10 dataset compared to the baseline (which had an accuracy of 54.54%) trained with 500 real images. This notable enhancement in accuracy demonstrates the superiority of our method using class unconditional GANs over the previous techniques aiming to enhance accuracy using class Conditional GANs.
Semantic RGB-D Image Synthesis

Collecting diverse sets of training images for RGB-D semantic image segmentation is not always possible. In particular, when robots need to operate in privacy-sensitive areas like homes, the collection is often limited to a small set of locations. As a consequence, the annotated images lack diversity in appearance and approaches for RGB-D semantic image segmentation tend to overfit the training data. In this paper, we thus introduce semantic RGB-D image synthesis to address this problem. It requires synthesising a realistic-looking RGB-D image for a given semantic label map. Current approaches, however, are unimodal and cannot cope with multi-modal data. Indeed, we show that extending uni-modal approaches to multi-modal data does not perform well. In this paper, we therefore propose a generator for multi-modal data that separates modal-independent information of the semantic layout from the modal-dependent information that is needed to generate an RGB and a depth image, respectively. Furthermore, we propose a discriminator that ensures semantic consistency between the label maps and the generated images and perceptual similarity between the real and generated images. Our comprehensive experiments demonstrate that the proposed method outperforms previous uni-modal methods by a large margin and that the accuracy of an approach for RGB-D semantic segmentation can be significantly improved by mixing real and generated images during training.
RoamML: Distributed Machine Learning at the Tactical Edge

Machine learning is an effective methodology for enabling real-time data-driven decision-making in tactical scenarios, but its application in such scenarios raises many challenges due to data volume, unpredictable connectivity, and infrastructural challenges in edge environments. Furthermore, the need to perform training operations on remote powerful computing nodes might not be suited for tactical edge networks that often lack high bandwidth links, thus causing critical delays in assessing relevant information for decision-making. To overcome these challenges and enable machine learning at the tactical edge, this paper presents RoamML, a novel distributed continual learning approach tailored explicitly for the tactical edge. Built upon the foundational principle that "moving the model is usually much cheaper than transferring extensive datasets", RoamML seamlessly performs training operations by traversing network nodes, according to the data gravity concept. As RoamML encounters new data at each node, it continually trains on the encountered data to ensure that RoamML maintains an up-to-date and accurate model without transferring data to a centralized entity. Experimental results comparing RoamML with a baseline centralized machine learning solution show the potential of the proposed approach, which is capable of closely matching the accuracy of the baseline method.
Deep Learning-Based Demodulation in Impulse Noise Channels

Wireless systems on military platforms often operate in challenging electromagnetic environments. Such environments contain interference which tends to be non-Gaussian. A more suitable model to derive the necessary receiver algorithms is the Symmetric α-Stable distribution. Adapting a receiver to the α-Stable model provides significant performance gain in challenging interference environments compared to Gaussian-based receivers, at the cost of higher computational complexity. With the emergence of deep learning and neural networks, a promising application is replacing the steps of demodulation with a suitable network structure in order to decrease the complexity. In this work, the generalization capability of such networks is explored in order to examine the models’ ability to adapt to new, unseen error correcting codes of different length and type, and if good performance can be achieved in comparison to traditional demodulation as well as if the networks will consider code structure when demodulating.The results show that a networks ability to generalize largely depends on architecture, as well as training data. Furthermore, the models which were unable to generalize made bad decisions based on code structure assumptions, resulting in worse performance compared to the more general models even though it was trained and tested on the same code. It is shown that the proposed model performs well, on par with the α-Stable based method, with significantly lower computational costs.
PRAT: PRofiling Adversarial aTtacks

Intrinsic susceptibility of deep learning to adversarial examples has led to a plethora of attack techniques with a broad common objective of fooling deep models. However, we find slight compositional differences between the algorithms achieving this objective. These differences leave traces that provide important clues for attacker profiling in real-life scenarios. Inspired by this, we introduce a novel problem of ‘PRofiling Adversarial aTtacks’ (PRAT). Given an adversarial example, the objective of PRAT is to identify the attack used to generate it. Under this perspective, we can systematically group existing attacks into different families, leading to the sub-problem of attack family identification, which we also study. To enable PRAT analysis, we introduce a large ‘Adversarial Identification Dataset’ (AID), comprising over 180k adversarial samples generated with 13 popular attacks for image specific/agnostic white/black box setups. We use AID to devise a novel framework for the PRAT objective. Our framework utilizes a Transformer based Global-LOcal Feature (GLOF) module to extract an approximate signature of the adversarial attack, which in turn is used for the identification of the attack. Using AID and our framework, we provide multiple interesting benchmark results for the PRAT problem. The dataset and the code are available at https://github.com/rahulambati/PRAT
CNN based Cuneiform Sign Detection Learned from Annotated 3D Renderings and Mapped Photographs with Illumination Augmentation

Motivated by the challenges of the Digital Ancient Near Eastern Studies (DANES) community, we develop digital tools for processing cuneiform script being a 3D script imprinted into clay tablets used for more than three millennia and at least eight major languages. It consists of thousands of characters that have changed over time and space. Photographs are the most common representations usable for machine learning, while ink drawings are prone to interpretation. Best suited 3D datasets that are becoming available. We created and used the HeiCuBeDa and MaiCuBeDa datasets, which consist of around 500 annotated tablets. For our novel OCR-like approach to mixed image data, we provide an additional mapping tool for transferring annotations between 3D renderings and photographs. Our sign localization uses a RepPoints detector to predict the locations of characters as bounding boxes. We use image data from GigaMesh’s MSII (curvature) based rendering, Phong-shaded 3D models, and photographs as well as illumination augmentation. The results show that using rendered 3D images for sign detection performs better than other work on photographs. In addition, our approach gives reasonably good results for photographs only, while it is best used for mixed datasets. More importantly, the Phong renderings, and especially the MSII renderings, improve the results on photographs, which is the largest dataset on a global scale.
Multi-Exit Resource-Efficient Neural Architecture for Image Classification with Optimized Fusion Block

In this paper, we propose a test-time resource-efficient neural architecture for image classification. Building on MSDNet [12], our multi-exit architecture excels in both anytime classification, which allows progressive updates of predictions for test examples and facilitates early output, and budgeted batch classification, which allows flexible allocation of computational resources across inputs to classify a set of examples within a fixed budget. Our proposed multi-exit architecture achieves state-of-the-art performance on CIFAR10 and CIFAR100 in these two critical scenarios, thanks to a novel feature fusion building block combined with an efficient stem block.
Geometric Contrastive Learning

Contrastive learning has been a long-standing research area due to its versatility and importance in learning representations. Recent works have shown improved results if the learned representations are constrained to be on a hypersphere. However, this prior geometric constraint is not fully utilized during training. In this work, we propose making use of geodesic distances on the hypersphere to learn contrasts between representations. Through empirical results, we show that this contrastive learning approach improves downstream tasks across different contrastive learning frameworks. We show that having geometric inductive priors perform even better in contrastive learning if used along with other correct geometric information.
Dynamic Texts From UAV Perspective Natural Images

Drone-based image processing offers valuable capabilities for surveillance, detection, and tracking in vast areas, aiding in disaster search and rescue, and monitoring artificial events like traffic jams and outdoor activities under adversarial weather conditions. Nonetheless, this technology encounters numerous challenges, including handling variations in scales and perspectives and coping with environmental factors like sky interference and the presence of far and small objects. Besides, ensuring high visibility distance in 3D depth is crucial for safe flights in various settings, including airports, cities, and fields. However, local weather conditions can change rapidly during flights, leading to visibility issues caused by fog and clouds. Due to the cost of visibility measurement sensors, lower-cost methods using portable apparatus are desired for flight routines. Therefore, this paper proposes a camera-based visibility and weather condition estimation approach using complementary multiple Deep Learning (DL) and Vision Language Models (VLM) under adversarial conditions. Experimental results show the superiority of enhanced 2D/3D captions with physical scales over SOTA VLMs.
Order-ViT: Order Learning Vision Transformer for Cancer Classification in Pathology Images

In computational pathology, cancer classification is one of the most widely studied tasks. There exist numerous tools for cancer classification, which are mainly built based upon convolutional neural networks or Transformers. These tools, by and large, formulate cancer classification as a categorical classification problem, which ignores the intrinsic relationship among cancer grades. Herein, we propose an order learning vision transformer for cancer classification that can not only learn the histopathological patterns of individual cancer grades but also utilize the ordering relationship among cancer grades. Built based upon vision transformer, the proposed method simultaneously conducts categorical classification per input sample and order classification for a pair of input and reference samples. Moreover, it introduces a voting scheme to identify less confident samples and to improve the accuracy of the decision on such samples. The proposed method is evaluated on two types of cancer datasets including colorectal and gastric cancers. Experimental results show that the proposed method outperforms other classification models and can facilitate improved cancer diagnosis in clinics.
SATHUR: Self Augmenting Task Hallucinal Unified Representation for Generalized Class Incremental Learning

Class Incremental Learning (CIL) is inspired by the human ability to learn new classes without forgetting previous ones. CIL becomes more challenging in real-world scenarios when the samples in each incremental step are imbalanced. This creates another branch of problem, called Generalized Class Incremental Learning (GCIL) where each incremental step is structured more realistically. Grow When Required (GWR) network, a type of Self-Organizing Map (SOM), dynamically creates and removes nodes and edges for adaptive learning. GWR performs incremental learning from feature vectors extracted by a Convolutional Neural Network (CNN), which acts as a feature extractor. The inherent ability of GWR to form distinct clusters, each corresponding to a class in the feature vector space, regardless of the order of samples or class imbalances, is well suited to achieving GCIL. To enhance GWR's classification performance, a high-quality feature extractor is required. However, when the convolutional layers are adapted at each incremental step, the GWR nodes corresponding to prior knowledge are subjected to near-invalidation. This work introduces the Self Augmenting Task Hallucinal Unified Representation (SATHUR), which re-initializes the GWR network at each incremental step, aligning it with the incrementally updated feature extractor. Comprehensive experimental results demonstrate that our proposed method significantly outperforms other state-of-the-art GCIL methods on CIFAR-100 and CORe50 datasets.
A Comprehensive Study of Transfer Learning under Constraints

Pre-training on an upstream task is widely used in deep learning to boost performance of downstream tasks. Recent studies analyzed pre-training with large datasets and large deep neural network architectures. However, pre-training is very useful in practice when downstream tasks have scarce data and are trained under computational constraints. To assess pre-training performance in this setting, we train different deep architectures with 1M parameters. We create different subsets of ImageNet to study the influence of upstream dataset in detail by varying the total size, but also the ratio between number of classes and samples per class for a constant total size. Then, we use the resulting models in transfer toward six diversified downstream tasks using linear probing and full fine tuning for downstream training. Experimental results confirm previous ones regarding performance saturation in downstream tasks, but we find that saturation occurs faster for compact deep architectures. The use of different ImageNet subsets leads to globally similar performance when enough data is included, regardless of the dataset structure. The comparison of downstream training strategies shows that linear probing can be competitive, particularly for few-shot settings. This is at odds with previous reports, which assert the superiority of full fine tuning. Finally, we observe that the type of deep architecture has a significant effect on results, but that their relative performance varies depending on the downstream training strategy.
FIVA: Facial Image and Video Anonymization and Anonymization Defense

In this paper, we present a new approach for facial anonymization in images and videos, abbreviated as FIVA. Our proposed method is able to maintain the same face anonymization consistently over frames with our suggested identity-tracking and guarantees a strong difference from the original face. FIVA allows for 0 true positives for a false acceptance rate of 0.001. Our work considers the important security issue of reconstruction attacks and investigates adversarial noise, uniform noise, and parameter noise to disrupt reconstruction attacks. In this regard, we apply different defense and protection methods against these privacy threats to demonstrate the scalability of FIVA. On top of this, we also show that reconstruction attack models can be used for detection of deep fakes. Last but not least, we provide experimental results showing how FIVA can even enable face swapping, which is purely trained on a single target image.
SegDA: Maximum Separable Segment Mask with Pseudo Labels for Domain Adaptive Semantic Segmentation

Unsupervised Domain Adaptation (UDA) aims to solve the problem of label scarcity of the target domain by transferring the knowledge from the label rich source domain. Usually, the source domain consists of synthetic images for which the annotation is easily obtained using the well known computer graphics techniques. However, obtaining annotation for real world images (target domain) require lot of manual annotation effort and is very time consuming because it requires per pixel annotation. To address this problem we propose SegDA module to enhance transfer performance of UDA methods by learning the maximum separable segment representation. This resolves the problem of identifying visually similar classes like pedestrian/rider, sidewalk/road etc. We leveraged Equiangular Tight Frame (ETF) classifier inspired from Neural Collapse for maximal separation between segment classes. This causes the source domain pixel representation to collapse to a single vector forming a simplex vertices which are aligned to the maximal separable ETF classifier. We use this phenomenon to propose the novel architecture for domain adaptation of segment representation for target domain. Additionally, we proposed to estimate the noise in labelling the target domain images and update the decoder for noise correction which encourages the discovery of pixels for classes not identified in pseudo labels. We have used four UDA benchmarks simulating synthetic-to-real, daytime-to-nighttime, clear-to-adverse weather scenarios. Our proposed approach outperforms +2.2 mIoU on GTA → Cityscapes, +2.0 mIoU on Synthia → Cityscapes, +5.9 mIoU on Cityscapes → DarkZurich, +2.6 mIoU on Cityscapes → ACDC.
Dual-level Interaction for Domain Adaptive Semantic Segmentation

Self-training approach recently secures its position in domain adaptive semantic segmentation, where a model is trained with target domain pseudo-labels. Current advances have mitigated noisy pseudo-labels resulting from the domain gap. However, they still struggle with erroneous pseudo-labels near the boundaries of the semantic classifier. In this paper, we tackle this issue by proposing a dual-level interaction for domain adaptation (DIDA) in semantic segmentation. Explicitly, we encourage the different augmented views of the same pixel to have not only similar class prediction (semantic-level) but also akin similarity relationship with respect to other pixels (instance-level). As it’s impossible to keep features of all pixel instances for a dataset, we, therefore, maintain a labeled instance bank with dynamic updating strategies to selectively store the informative features of instances. Further, DIDA performs cross-level interaction with scattering and gathering techniques to regenerate more reliable pseudo-labels. Our method outperforms the state-of-the-art by a notable margin, especially on confusing and long-tailed classes. Code is available at https://github.com/RainJamesY/DIDA
Detection of Hate Speech on Twitter for Arabic Iraqi Dialect Using Stochastic Gradient Classifier

People increasingly use social media platforms to communicate and share information worldwide. However, challenges such as verbal misbehaviors and hate messages have been raised and widely disseminated via social media. In the Arabic region, especially in Iraq, Twitter is one of the most prominent social media platforms that has gained popularity. Consequently, an ethical reason behind this research is to build a system that can distinguish Iraqi hate tweets. This work achieved three contributions. First, a dataset of Iraqi tweets has been collected and annotated; this is the first dataset for hate speech in the Iraqi dialect. Second, an algorithm for distinguishing Arabic tweets from other oriental tweets that use the same Arabic glyph was proposed to represent a new addition to the preprocessing steps for the Arabic NLP area. This algorithm was tested and showed efficiency in detecting Urdu about (0.91), Persian (0.73), and Arabic (0.98). Third, the Iraqi hate speech classifier was built using two types of text features depending on the stochastic gradient classifier. The classifier was compared with various machine learning algorithms like SVM logistic regression. The proposed system with stochastic gradient classifier was more efficient than other classifiers achieving precision and recall of 0.80.
Template-guided Illumination Correction for Document Images with Imperfect Geometric Reconstruction

To facilitate the transition into the digital era, it is necessary to digitize printed documents such as forms and invoices. Due to the presence of diverse lighting conditions and geometric distortions in real-world photographs of documents, document image restoration typically consists of two stages: first, geometric unwarping to remove the displacement distortions and, second illumination correction to reinstate the original colors. In this work, we tackle the problem of illumination correction for document images and, thereby, enhance downstream tasks, such as text extraction and document archival. Despite the recent state-of-the-art improvements in geometric unwarping, the reliability of those models is limited. Hence, we aim to reduce lighting impurity under the assumption of imperfectly unwarped documents. To reduce the complexity of the task, we incorporate a-priori known visual cues in the form of template images, which offer additional information about the perfect lighting conditions. In this work, we present a novel approach for integrating prior visual cues in the form of document templates. Our extensive evaluation shows a 15.0 % relative improvement in LPIPS and 6.3 % in CER over the state-of-the-art. We made all code and data publicly available at https://felixhertlein.github.io/illtrtemplate.
Towards Robust Natural-Looking Mammography Lesion Synthesis on Ipsilateral Dual-Views Breast Cancer Analysis

In recent years, many mammographic image analysis methods have been introduced for improving cancer classification tasks. Two major issues of mammogram classification tasks are leveraging multi-view mammographic information and class-imbalance handling. In the first problem, many multi-view methods have been released for concatenating features of two or more views for the training and inference stage. Having said that, most multi-view existing methods are not explainable in the meaning of feature fusion, and treat many views equally for diagnosing. Our work aims to propose a simple but novel method for enhancing examined view (main view) by leveraging low-level feature information from the auxiliary view (ipsilateral view) before learning the high-level feature that contains the cancerous features. For the second issue, we also propose a simple but novel malignant mammogram synthesis framework for upsampling minor class samples. Our easy-to-implement and no-training framework has eliminated the current limitation of the CutMix algorithm which are unreliable synthesized images with random pasted patches, hard-contour problems, and domain shift problems. Our results on VinDr-Mammo and CMMD datasets show the effectiveness of our two new frameworks for both multi-view training and synthesizing mammographic images, outperforming the previous conventional methods in our experimental settings.