Reviewerly/exHarmony huggingface dataset:
Knowledge Distillation with Ensemble Calibration

Knowledge Distillation is a transfer learning and compression technique that aims to transfer hidden knowledge from a teacher model to a student model. However, this transfer often leads to poor calibration in the student model. This can be problematic for high-risk applications that require well-calibrated models to capture prediction uncertainty. To address this issue, we propose a simple and novel technique that enhances the calibration of the student network by using an ensemble of well-calibrated teacher models. We train multiple teacher models using various data-augmentation techniques such as cutout, mixup, CutMix, and AugMix and use their ensemble for knowledge distillation. We evaluate our approach on different teacher-student combinations using CIFAR-10 and CIFAR-100 datasets. Our results demonstrate that our technique improves calibration metrics (such as expected calibration and overconfidence errors) while also increasing the accuracy of the student network.
MoMSGAN: Mode Collapse based Degradation Agnostic Multi-Scale Super-Resolution of Medical Images

Existing super-resolution (SR) models require separate training for different scales of SR because of the fixed upsampling levels in their architecture. We propose a novel one-time training approach for multi-scale SR models with a fixed upsampling unit, that can effectively super-resolve images to various scales. Another persisting problem relevant to medical image SR is that the models are biased to the level of degradation present in the low-resolution (LR) images and generate variations in anatomical structures and pathologies when applied to LR images corresponding to the same high-resolution (HR) image, varying only in degradation. But regardless of the degradation in the LR images, the anatomical structures and clinical findings in the SR images should remain consistent and unique to the patient's specific anatomy and health conditions. We thus propose a novel approach for GANs to blindly constrain the variations in super-resolved medical images. Further, we introduce an ensemble loss function to gain details and accurate textures, while enabling multi-scale SR and making the model unbiased towards degradation in LR images. Experimental results on lung CT scans demonstrate that our model, MoMSGAN, is an efficient one-time training approach for multi-scale SR and is also capable of reducing the degradation bias in reproducing accurate information in medical images, thus establishing state-of-the-art.
MMAG: Mutually Motivated Attention Gates for Simultaneous Extraction of Contextual and Spatial Information from a Monocular Image

In order to effectively interact with its environment, an agent must possess the ability to comprehend the 'what' and the 'where.' Two vision-based approaches, namely semantic segmentation and depth estimation, can be employed to provide this information. This paper introduces a unified model that combines these two tasks and utilizes a shared latent space. The chosen approach for the model involves an encoder-decoder architecture accompanied by a novel attention gate mechanism called MMAG, which has been proven to be highly effective. Furthermore, a common skip connection method, in conjunction with the proposed attention gate, is included to emphasize crucial features for both prediction purposes. By leveraging the complementary information within the shared representations, the model is capable of generating more accurate predictions compared to other joint-prediction models, all while utilizing fewer network parameters. To enhance performance, dilated layers are incorporated, allowing for focused attention through different receptive fields. The model's performance has been evaluated on the NYU Depth v2 and Camvid datasets, demonstrating improved results on both tasks when compared to other state-of-the-art models. With its reduced parameter count, this model is particularly suitable for low-cost robots and has undergone successful testing for inference on the LoCoBot robot.
STTGC-Net: Spatial-Temporal Transformer with Graph Convolution for Skeleton-Based Action Recognition

Skeleton data plays an important role in human action recognition due to the compact and distinct information of human poses provided by the skeleton data. Skeleton-based action recognition is gaining interest due to the availability of Kinect cameras at a reasonable price. With the growing popularity of Geometric deep learning, Graph Convolutional networks (GCN) are extensively used for processing skeleton data, due to their ability to model data topology. It has been found that Spatial-Temporal Graph Convolutional Networks (ST-GCN) are efficient in learning both spatial and temporal dependencies on non-Euclidean data, such as skeleton graphs. However, the state-of-the-art ST-GCN models lack flexibility in feature extraction and do not explicitly consider the high-order Spatio-Temporal significance of the spatial connection topology and intensity of the joints. They lack an attention module that can help us learn together when and where to concentrate on a certain action in the action sequence. The transformer-based methods can effectively capture long-distance dependencies. However, using just a traditional transformer approach overlooks the graph structure in the data, which can provide valuable information about the inter-relationships among the joint points during actions. To address this problem, we propose an architecture named Spatial-Temporal Transformer Network with Graph Convolution (STTGC-Net), that can flexibly capture local and global contexts. Spatial-Temporal Graph Convolutional Networks and Spatial-Temporal Transformer Attention Modules are sequentially fused to create the proposed framework. The ST-GCNs capture temporal dynamics, hierarchy, and local topological information at several levels, and the transformer attention module displays the correlations between joint pairs in global topology via dynamic attention, which resolves the mentioned constraints of the GCN and transformer. We validate our model by conducting tests on the NTU 60 and NTU 120 datasets, which achieve state-of-the-art performance. The code is available through github.
Dense captioning for Text-Image ReID

Text-to-Image (T2I) ReID has attracted a lot of attention in the recent past. CUHK-PEDES, RSTPReid and ICFG-PEDES are the three available benchmarks to evaluate T2I ReID methods. RSTPReid and ICFG-PEDES comprise of identities from MSMT17 but due to limited number of unique persons, the diversity is limited. On the other hand, CUHK-PEDES comprises of 13,003 identities but has relatively shorter text description on average. Further, these datasets are captured in a restricted environment with limited number of cameras. In order to further diversify the identities and provide dense captions, we propose a novel dataset. Our dataset comprises of 20,000 unique identities captured in the wild and provides a rich dataset for text-to-image ReID. With a minimum of 26 words for a description, each image is densely captioned. We further synthetically generate images and fine-grained captions using Stable-diffusion and BLIP models trained on our dataset. We perform elaborate experiments using state-of-art text-to-image ReID models and vision-language pre-trained models and present a comprehensive analysis of the dataset. Our experiments also reveal that synthetically generated data leads to a substantial performance improvement in both same dataset as well as cross dataset settings. We will release the code and dataset.
Dual Stage Semantic Information Based Generative Adversarial Network For Image Super-Resolution✱

Deep learning methods for the super-resolution problem are showing great performance compared to other traditional techniques. However, these methods are unable to learn complex spatial structures and high frequency details; which leads to over-smooth results. In the present paper, a novel Generative Adversarial Network based architecture named as Residue and Semantic feature based Dual Subpixel Generative Adversarial Network has been proposed for generator and discriminator networks to solve super-resolution problem. The generator network is residue and semantic feature based dual subpixel generative architecture. This architecture is divided into two stages: premier residual stage and deuxieme residual stage. These two stages are concatenated together to form a two stage upsamping process, which enhances the feature learning capability of our model. Inter and intra residual connections are made within these two stages; helping us to sustain the high texture details of images. Semantic based information is implanted in generator to enhance the quality of objects in an image. For embedding semantic information in generator, feature maps extracted from pre-trained model are merged with the input image. To stabilize the training process, we introduced spectral normalization in the discriminator. Visual perception and mean opinion score shows that proposed method outperforms the other state-of-the-art methods.
Unique Identity Generation with Global Features from Multimodal Biometric Data

The patterns in biometric data, such as fingerprints, iris, etc., are random and distinct from one individual to another, making them ideal for generating unique identities suitable for many applications. This work proposes creating users' unique identities based on their fingerprint and iris biometric data. The global feature descriptor, which is invariant to scale, rotation, and translation, is considered in this work. The global feature descriptors are obtained from fingerprint and iris images and fused to create a single feature vector. Next, feature selection based on the Laplacian score is applied to remove irrelevant features. Further, a metric-learning-based mechanism is introduced to enhance the feature vectors' intra-class similarity and inter-class separability. Finally, a novel quantization mechanism is used to generate a unique identity. Experiments with multimodal datasets show that the generated identities are distinct, random, revocable, and unlinkable. This identity generation scheme can be utilized in storage security, digital signature schemes, remote authentication protocols, blockchain security, etc.
Single View Homography Estimation for an Inclined Textured Planar Surface: Overcoming the Inverse and Ill-Posed Challenge!

Homography estimation is a crucial step in many computer vision problems involving the planar transformation of an image from one view to another. Generally, from two image views of a scene with a planar patch, one can compute the homography matrix (H) using Direct Linear Transformation (DLT) or its variants. In this study, we work on the inverse scenario to solve an ill-posed problem, where given only a single image (<Formula format="inline"><TexMath><?TeX $I_{\tau }^{P}$?></TexMath><AltText>Math 1</AltText><File name="icvgip23-2-inline1" type="svg"/></Formula>) of an inclined textured planar surface, we attempt to estimate both H and the orthogonal view (<Formula format="inline"><TexMath><?TeX $I_{\tau }^{O}$?></TexMath><AltText>Math 2</AltText><File name="icvgip23-2-inline2" type="svg"/></Formula>) of the planar surface. We propose to solve this problem of homography Estimation using an optimization framework. To the best of our knowledge, there has barely been any work done on this inverse scenario of homography estimation from only a single image view. The research presented in this paper successfully achieves image rectification and homography computation by assuming a perspective transformation on an upright planar patch when viewed by a camera placed exactly in front of it. A cost function with suitable constraints is utilized to refine homography estimation. Experimental results reveal the efficiency of the proposed approach in the case of images of checkerboard and other textured planar surfaces (real-world images with lines and other prominent textures).
Improved Discrepancy based Domain Adaptation Network using combined loss functions and Feature transformations✱

Domain shifts are a common problem in computer vision. As a result, a classifier trained on a source domain cannot perform well on a target domain. Due to this, a source classifier trained to differentiate based on a specific distribution cannot classify well on new data from a different distribution. In this work, we have studied different domain adaptation networks for their complexity and performance and have shown that a simple discrepancy based network can be modified to produce a performance on par with the more complex GAN based networks[8]. This is achieved by picking and selectively combining loss functions and preprocessing transformations. We have shown that the proposed method has produced accuracies 77.62% and 81.67% on popular datasets Office-31 and MNIST/MNIST-M, surpassing the highest reported accuracies by 6.47% and 16.67% compared to seminal papers [16][1] in this field. 1
S-VQA: Sentence-Based Visual Question Answering

Visual Question Answering (VQA) system responds to a natural language question in context of an image. This problem has been primarily formulated as a classification problem with the answers as the finite number of classes. Thus, the generated response consists of a single word or a short phrase. However, this also limits the linguistic capabilities of such a system. In contrast, this work presents a Sentence-based VQA (S-VQA) which responds to questions with complete sentences as answers. The first contribution of this work is the development of a dataset from the Task Directed Image Understanding Challenge (TDIUC) VQA dataset using natural language rules and pretrained para-phrasers. This new dataset is referred to as TDIUC-SVQA. The second contribution involves the performance evaluation of multiple models on the TDIUC-SVQA dataset. This is performed by using two multi-modal models. The Bottom-Up Top-Down Attention based VQA model is combined with LSTM decoder and Attention-on-Attention Network for answer generation. The proposed models are observed to provide improved results compared to the baseline model.
Aggregated Co-attention based Visual Question Answering

Recent developments in the field of Visual Question Answering (VQA) have witnessed promising improvements in performance through contributions in attention based networks. Most such approaches have focused on unidirectional attention that leverage over attention from textual domain (question) on visual space. This work proposes a multistage co-attention framework. Here, co-attention framework performs both image and text attention. The co-attention mechanism is repeated in multiple stages. Attention on different stages may capture some significant and distinct features for learning better contextual information. Thus, aggregation of attention is performed to preserve the information from different stages. The proposed architecture with multiple stage network could suffer from vanishing or exploding gradients. To prevent this, loss at each of the different stages is computed. Extensive experiments and analysis are performed for validating the effects of aggregated attention and stage-wise loss.
Learning from Heterogeneous Data with Deep Gaussian Processes

Deep Gaussian processes (DGPs) are deep models represented by layers of Gaussian processes (GPs). They are flexible Bayesian models capable of capturing highly nonlinear functions while providing well-calibrated uncertainties of the made predictions. Despite these strengths, DGPs encounter difficulties with real-world applications involving heterogeneous datasets and missing values. In this paper, we propose a two-stage framework for DGPs that leverages all available data in such settings. Our inference is performed within a scalable stochastic variational framework, where the variational posterior distributions are reparameterized through sparse GPs. We derive a formulation of the variational lower bound, effectively handling heterogeneous data. Further, through our inference method, we demonstrate the effective use of the GP framework on heterogeneous data. We evaluate our method on a dataset generated by a novel psychosocial screening tool called PROMOTE, which was designed to acquire data for predicting adverse perinatal outcomes and maternal mental health morbidities. Our experimental evaluation of the dataset indicates the effectiveness of our inference framework on various important learning tasks. We demonstrate the model's performance by benchmarking against existing models for high-dimensional heterogeneous data examples.
Bayesian Self-Supervised Learning Using Local and Global Graph Information

Graph-guided learning has well-documented impact in a gamut of network science applications. A prototypical graph-guided learning task deals with semi-supervised learning over graphs, where the goal is to predict the nodal values or labels of unobserved nodes, by leveraging a few nodal observations along with the underlying graph structure. This is particularly challenging under privacy constraints or generally when acquiring nodal observations incurs high cost. In this context, the present work puts forth a Bayesian graph-driven self-supervised learning (Self-SL) approach that: (i) learns powerful nodal embeddings emanating from easier to solve auxiliary tasks that map local to global connectivity information; and, (ii) adopts an ensemble of Gaussian processes (EGPs) with adaptive weights as nodal embeddings are processed online. Unlike most existing deterministic approaches, the novel approach offers accurate estimates of the unobserved nodal values along with uncertainty quantification that is important especially in safety critical applications. Numerical tests on synthetic and real graph datasets showcase merits of the novel EGP-based Self-SL method.
GPT-Based Models Meet Simulation: How to Efficiently use Large-Scale Pre-Trained Language Models Across Simulation Tasks

The disruptive technology provided by large-scale pre-trained language models (LLMs) such as ChatGPT or GPT-4 has received significant attention in several application domains, often with an emphasis on high-level opportunities and concerns. This paper is the first examination regarding the use of LLMs for scientific simulations. We focus on four modeling and simulation tasks, each time assessing the expected benefits and limitations of LLMs while providing practical guidance for modelers regarding the steps involved. The first task is devoted to explaining the structure of a conceptual model to promote the engagement of participants in the modeling process. The second task focuses on summarizing simulation outputs, so that model users can identify a preferred scenario. The third task seeks to broaden accessibility to simulation platforms by conveying the insights of simulation visualizations via text. Finally, the last task evokes the possibility of explaining simulation errors and providing guidance to resolve them.
Using Simulated Narratives to Understand Attribution in the Information Dimension

Conducting a measured response to cyber or information attack is predicated on attribution. When these operations are conducted covertly or through proxies, uncertainty in attribution limits response options. To increase attribution certainty in the information dimension, the authors have developed a suite of supervised machine learning models that attribute an emerging narrative to historical narratives from known actors. These models were first developed on simulated narratives produced with a Large Language Model. Once the supervised classification models were developed and tested on the simulated narratives, they are evaluated on known actor social media narratives from three known actors. The attribution models are language agnostic and offer one-vs-rest and multi-class options. All models performed at relatively high accuracy and can provide decision support for cyber response decisions.
Resilience and Complexity in Socio-Cyber-Physical Systems

Socio-Cyber-Physical Systems are ubiquitous in today’s world. They are inherently complex systems built out of many large-scale systems that encompass different perspectives and numerous stakeholders. This leads to several challenges in managing their complexity and emergent behavior. In addition, these systems tend to include many adaptive and autonomous systems with different goals and different adaptations to environment changes or failures. The design, analysis, and testing of such systems is inherently challenging but is becoming critical due to their wide adoption. In this panel, we aim to discuss some of these challenges and potential solutions.
Trajectory-Oriented Optimization of Stochastic Epidemiological Models

Epidemiological models must be calibrated to ground truth for downstream tasks such as producing forward projections or running what-if scenarios. The meaning of calibration changes in case of a stochastic model since output from such a model is generally described via an ensemble or a distribution. Each member of the ensemble is usually mapped to a random number seed (explicitly or implicitly). With the goal of finding not only the input parameter settings but also the random seeds that are consistent with the ground truth, we propose a class of Gaussian process (GP) surrogates along with an optimization strategy based on Thompson sampling. This Trajectory Oriented Optimization (TOO) approach produces actual trajectories close to the empirical observations instead of a set of parameter settings where only the mean simulation behavior matches with the ground truth.
Approximate Gaussian Process Regression with Pairwise Comparison Data

We use approximate Bayesian inference, together with Gaussian process regression, to create a new estimator for an unknown function in a situation where we can only observe pairwise comparisons of function values at different inputs. Preliminary experimental results suggest that, although information is heavily censored in this setting, it may still be possible to learn the local and global minima of the underlying function. We discuss possible sampling criteria, and explore the performance of the "probability of improvement" strategy numerically.
Multi-Model Fusion Vehicle Supervision System Based on Deep Learning

This paper presents a hybrid vehicle monitoring system for closed campus environments. The system leverages the strengths of YOLO and Faster R-CNN object detection algorithms to enable accurate vehicle detection and tracking across multiple camera views. By using Intersection over Union (IoU) technology to track vehicles in different regions of interest (ROI) and surveillance video frames, it dynamically updates vehicle count information for various areas, generating a vehicle distribution heat map for convenient visual analysis of campus traffic management. A congestion warning is issued when the number of vehicles in an area reaches the preset threshold. Additionally, the system includes a license plate detection module that accurately recognizes license plates by combining STN, CNN, LSTM, and CTC network models, recording license plates when speeding vehicles are detected. Experimental results show that the improved vehicle detection hybrid algorithm achieves an increase of 11.36 percentage points in mAP and has a significant improvement in processing speed. The enhanced network model demonstrates desirable results in license plate detection, with a nearly 2% increase in detection rate and reduced processing time. Overall, the system exhibits superior performance.