Reviewerly/exHarmony huggingface dataset:
MedFuseNet: An attention-based multimodal deep learning model for visual question answering in the medical domain

Medical images are difficult to comprehend for a person without expertise. The scarcity of medical practitioners across the globe often face the issue of physical and mental fatigue due to the high number of cases, inducing human errors during the diagnosis. In such scenarios, having an additional opinion can be helpful in boosting the confidence of the decision maker. Thus, it becomes crucial to have a reliable visual question answering (VQA) system to provide a 'second opinion' on medical cases. However, most of the VQA systems that work today cater to real-world problems and are not specifically tailored for handling medical images. Moreover, the VQA system for medical images needs to consider a limited amount of training data available in this domain. In this paper, we develop MedFuseNet, an attention-based multimodal deep learning model, for VQA on medical images taking the associated challenges into account. Our MedFuseNet aims at maximizing the learning with minimal complexity by breaking the problem statement into simpler tasks and predicting the answer. We tackle two types of answer prediction-categorization and generation. We conducted an extensive set of quantitative and qualitative analyses to evaluate the performance of MedFuseNet. Our experiments demonstrate that MedFuseNet outperforms the state-of-the-art VQA methods, and that visualization of the captured attentions showcases the intepretability of our model's predicted results.
Neural predictor based quantum architecture search

Variational quantum algorithms (VQAs) are widely speculated to deliver quantum advantages for practical problems under the quantum–classical hybrid computational paradigm in the near term. Both theoretical and practical developments of VQAs share many similarities with those of deep learning. For instance, a key component of VQAs is the design of task-dependent parameterized quantum circuits (PQCs) as in the case of designing a good neural architecture in deep learning. Partly inspired by the recent success of AutoML and neural architecture search (NAS), quantum architecture search (QAS) is a collection of methods devised to engineer an optimal task-specific PQC. It has been proven that QAS-designed VQAs can outperform expert-crafted VQAs in various scenarios. In this work, we propose to use a neural network based predictor as the evaluation policy for QAS. We demonstrate a neural predictor guided QAS can discover powerful quantum circuit ansatz, yielding state-of-the-art results for various examples from quantum simulation and quantum machine learning. Notably, neural predictor guided QAS provides a better solution than that by the random-search baseline while using an order of magnitude less of circuit evaluations. Moreover, the predictor for QAS as well as the optimal ansatz found by QAS can both be transferred and generalized to address similar problems.
Multilingual handwritten numeral recognition using a robust deep network joint with transfer learning

Numeral recognition plays a crucial role in creating automated systems such as posting address sorting and license plate recognition. Nowadays, numeral recognition systems which have the capability of recognising multiple languages are highly beneficial due to growing international correspondence and transactions, especially in multilingual countries where several languages are used simultaneously. Therefore, handwritten numeral recognition is more challenging than printed numeral recognition due to having different and complex handwriting styles. Hence, developing a multilingual handwritten system is considered as an important and debatable issue. We address this issue by proposing a language-independent model based on a robust CNN. Our proposed model is composed of language recognition and digit recognition, which aims to handle the recognition of multi-script images. We used transfer-learning in the proposed system to enhance the image quality and consequently the recognition performance. Extensive experiments were conducted to verify the effectiveness of both language and digit recognition procedures. The proposed system was tested with six different languages. The results showed an average accuracy of up to 99.8% for recognising various languages and the associated digits. The robustness and design procedure of the proposed model created a cost-effective extension for recognition of handwritten numerals in other languages.
Improving Contrastive Learning by Visualizing Feature Transformation

Contrastive learning, which aims at minimizing the distance between positive pairs while maximizing that of negative ones, has been widely and successfully applied in unsupervised feature learning, where the design of positive and negative (pos/neg) pairs is one of its keys. In this paper, we attempt to devise a feature-level data manipulation, differing from data augmentation, to enhance the generic contrastive self-supervised learning. To this end, we first design a visualization scheme for pos/neg score <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> distribution, which enables us to analyze, interpret and understand the learning process. To our knowledge, this is the first attempt of its kind. More importantly, leveraging this tool, we gain some significant observations, which inspire our novel Feature Transformation proposals including the extrapolation of positives. This operation creates harder positives to boost the learning because hard positives enable the model to be more view-invariant. Besides, we propose the interpolation among negatives, which provides diversified negatives and makes the model more discriminative. It is the first attempt to deal with both challenges simultaneously. Experiment results show that our proposed Feature Transformation can improve at least 6.0% accuracy on ImageNet-100 over MoCo baseline, and about 2.0% accuracy on ImageNet-1K over the MoCoV2 baseline. Transferring to the downstream tasks successfully demonstrate our model is less task-bias. Visualization tools and codes: https://github.com/DTennant/CL-Visualizing-Feature-Transformation.
ARAE: Adversarially robust training of autoencoders improves novelty detection

Autoencoders have recently been widely employed to approach the novelty detection problem. Trained only on the normal data, the AE is expected to reconstruct the normal data effectively while failing to regenerate the anomalous data. Based on this assumption, one could utilize the AE for novelty detection. However, it is known that this assumption does not always hold. Such an AE can often perfectly reconstruct the anomalous data due to modeling low-level and generic features in the input. We propose a novel training algorithm for the AE that facilitates learning more semantically meaningful features to address this problem. For this purpose, we exploit the fact that adversarial robustness promotes the learning of significant features. Therefore, we force the AE to learn such features by making its bottleneck layer more stable against adversarial perturbations. This idea is general and can be applied to other autoencoder-based approaches as well. We show that despite using a much simpler architecture than the prior methods, the proposed AE outperforms or is competitive to the state-of-the-art on four benchmark datasets and two medical datasets.
Partial Domain Adaptation on Semantic Segmentation

The research of semantic segmentation based on unsupervised domain adaptation greatly alleviates the high-cost bottleneck of manual annotation in deep learning. Inevitably domain gap limits the ability of target domain to learn knowledge from source domain. Previous domain alignment strategies aim to explore maximum domain-invariant space to enlarge knowledge that target domain learns. They based on the assumption that images are symmetrical in categories covering all predefined categories. However, in practical training, there is class asymmetry in image samples inter domains. This will lead to outlier categories and negative transfer, because it lacks the guidance of the corresponding category of the source domain. To tackle this issue, we propose a partial domain adaptive method on semantic segmentation to guide target model to learn category-level knowledge selectively. PAM (Partial Adaptive Map) module is introduced to motivate the target model to obtain more knowledge from non-outliers and less knowledge from outliers to avoid negative transfer. We further analyze the effectiveness of this method from the perspective of JS divergence. Our method has achieved significant improvement without additional discriminators. Experiments on general datasets GTA5 and SYNTHIA can compare with SOTA methods.
Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation

We address the problem of generalized zero-shot semantic segmentation (GZS3) predicting pixel-wise semantic labels for seen and unseen classes. Most GZS3 methods adopt a generative approach that synthesizes visual features of unseen classes from corresponding semantic ones (e.g., word2vec) to train novel classifiers for both seen and unseen classes. Although generative methods show decent performance, they have two limitations: (1) the visual features are biased towards seen classes; (2) the classifier should be retrained whenever novel unseen classes appear. We propose a discriminative approach to address these limitations in a unified framework. To this end, we leverage visual and semantic encoders to learn a joint embedding space, where the semantic encoder transforms semantic features to semantic prototypes that act as centers for visual features of corresponding classes. Specifically, we introduce boundary-aware regression (BAR) and semantic consistency (SC) losses to learn discriminative features. Our approach to exploiting the joint embedding space, together with BAR and SC terms, alleviates the seen bias problem. At test time, we avoid the retraining process by exploiting semantic prototypes as a nearest-neighbor (NN) classifier. To further alleviate the bias problem, we also propose an inference technique, dubbed Apollonius calibration (AC), that modulates the decision boundary of the NN classifier to the Apollonius circle adaptively. Experimental results demonstrate the effectiveness of our framework, achieving a new state of the art on standard benchmarks.
Object Tracking by Jointly Exploiting Frame and Event Domain

Inspired by the complementarity between conventional frame-based and bio-inspired event-based cameras, we propose a multi-modal based approach to fuse visual cues from the frame- and event-domain to enhance the single object tracking performance, especially in degraded conditions (e.g., scenes with high dynamic range, low light, and fast-motion objects). The proposed approach can effectively and adaptively combine meaningful information from both domains. Our approach’s effectiveness is enforced by a novel designed cross-domain attention schemes, which can effectively enhance features based on self- and cross-domain attention schemes; The adaptiveness is guarded by a specially designed weighting scheme, which can adaptively balance the contribution of the two domains. To exploit event-based visual cues in single-object tracking, we construct a large-scale frame-event-based dataset, which we subsequently employ to train a novel frame-event fusion based model. Extensive experiments show that the proposed approach outperforms state-of-the-art frame-based tracking methods by at least 10.4% and 11.9% in terms of representative success rate and precision rate, respectively. Besides, the effectiveness of each key component of our approach is evidenced by our thorough ablation study.
Big Data Contribution in Desktop and Mobile Devices Comparison, Regarding Airlines’ Digital Brand Name Effect

Rising demand for optimized digital marketing strategies has led firms in a hunt to harvest every possible aspect indicating users’ experience and preference. People visit, regularly through the day, numerous websites using both desktop and mobile devices. For businesses to acknowledge device’s usage rates is extremely important. Thus, this research is focused on analyzing each device’s usage and their effect on airline firms’ digital brand name. In the first phase of the research, we gathered web data from 10 airline firms during an observation period of 180 days. We then proceeded in developing an exploratory model using Fuzzy Cognitive Mapping, as well as a predictive and simulation model using Agent-Based Modeling. We inferred that various factors of airlines’ digital brand name are affected by both desktop and mobile usage, with mobile usage having a slightly bigger impact on most of them, with gradually rising values. Desktop device usage also appeared to be quite significant, especially in traffic coming from referral sources. The paper’s contribution has been to provide a handful of time-accurate insights for marketeers, regarding airlines’ digital marketing strategies.
Post-quantum lightweight identity-based two-party authenticated key exchange protocol for Internet of Vehicles with probable security

With Internet of Things (IoT) growing rapidly, the Internet of Vehicles (IoV) has become an essential part of smart cities and has attracted the full attention of both academic and business communities. Because of the public transmission channel, the security and privacy in IoV have paid serious attention. In IoV, it is crucial to generate a secret session key among the various vehicles and road-side units (RSUs) so that they can share the confidential information over the public Internet. Thus, an authenticated key agreement (AKA) protocol should be needed that can achieve the session key requirement in the IoV for secure communication. For this purpose, various AKA techniques has been designed using a number of different tools. Several existing AKA protocols either suffer from different attacks or inefficient for IoV environment due to its excessive communication and computational costs. Many such traditional schemes have used either Diffie–Hellman (DH) or prime factorization type hard problems. These hard problems are vulnerable to the futuristic technologies like quantum computer. Besides, existing quantum resistance AKA protocols use lattice cryptography for its security. However, these protocols either incurs an overhead of certificate management or have excessive communication and computational costs. Hence, there is a need of quantum resistance AKA protocols which removes the certificate overhead and also efficient for the IoV. In this paper, we propose a lattice-based two-party authenticated key agreement (LB-ID-2PAKA) protocol using identity-based cryptography (IBC). The lattice hard problems could resist the quantum computers and IBC could remove the overhead of certificate management. The security strength of proposed LB-ID-2PAKA protocol is analyzed under the random oracle model to show its robustness against the present as well as future quantum attacks. In addition, the resiliency against different security attacks such as man-in-the-middle (MITM) attack, known-key security (K-KS), unknown key-share (UK-S) attack etc. are also included. Further, the performance analysis shows that the proposed LB-ID-2PAKA protocol outperforms the existing protocols and feasible for IoV applications.
Instance-Based Vision Transformer for Subtyping of Papillary Renal Cell Carcinoma in Histopathological Image

Histological subtype of papillary (p) renal cellular and cell-layer level patterns almost cannot be captured by existing CNN-based models in large-size histopathological images, which brings obstacles to directly applying these models to such a fine-grained classification task. This paper proposes a novel instance-based Vision Transformer (i-ViT) to learn robust representations of histopathological images for the pRCC subtyping task by extracting finer features from instance patches (by cropping around segmented nuclei and assigning predicted grades). The proposed i-ViT takes top-K instances as the inputs and aggregates them for capturing both the cellular and cell-layer level patterns by a position-embedding layer, a grade-embedding layer, and a multi-head multi-layer self-attention module. To evaluate the performance of the proposed framework, experienced pathologists select 1162 regions of interest from 171 whole slide images of type 1 and type 2 pRCC. Experimental results show that the proposed method achieves better performance than existing CNN-based models with a significant margin.
E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-Based Stereoscopic Depth Perception

Reconstructing the scene of robotic surgery from the stereo endoscopic video is an important and promising topic in surgical data science, which potentially supports many applications such as surgical visual perception, robotic surgery education and intra-operative context awareness. However, current methods are mostly restricted to reconstructing static anatomy assuming no tissue deformation, tool occlusion and de-occlusion, and camera movement. However, these assumptions are not always satisfied in minimal invasive robotic surgeries. In this work, we present an efficient reconstruction pipeline for highly dynamic surgical scenes that runs at 28 fps. Specifically, we design a transformer-based stereoscopic depth perception for efficient depth estimation and a light-weight tool segmentor to handle tool occlusion. After that, a dynamic reconstruction algorithm which can estimate the tissue deformation and camera movement, and aggregate the information over time is proposed for surgical scene reconstruction. We evaluate the proposed pipeline on two datasets, the public Hamlyn Centre Endoscopic Video Dataset and our in-house DaVinci robotic surgery dataset. The results demonstrate that our method can recover the scene obstructed by the surgical tool and handle the movement of camera in realistic surgical scenarios effectively at real-time speed.
Contrastive Pre-training and Representation Distillation for Medical Visual Question Answering Based on Radiology Images

One of the primary challenges facing medical visual question answering (Med-VQA) is the lack of large-scale well-annotated datasets for training. To overcome this challenge, this paper proposes a two-stage pre-training framework by learning transferable feature representations of radiology images and distilling a lightweight visual feature extractor for Med-VQA. Specifically, we leverage large amounts of unlabeled radiology images to train three teacher models for the body regions of brain, chest, and abdomen respectively via contrastive learning. Then, we distill the teacher models to a lightweight student model that can be used as a universal visual feature extractor for any Med-VQA system. The lightweight feature extractor can be readily fine-tuned on the training radiology images of any Med-VQA dataset, saving the annotation effort while preventing overfitting to small-scale training data. The effectiveness and advantages of the pre-trained model are demonstrated by extensive experiments with state-of-the-art Med-VQA methods on existing benchmarks. The source code and the pre-training dataset can be downloaded from https://github.com/awenbocc/cprd.
Lexical features based malicious URL detection using machine learning techniques

Most sophisticated cyber-attack technique used by the cyber criminals is creating and spreading malicious domain names or malicious URLs through email, messages, popups etc. Malicious URL are the web pages targeted towards the internet user to spread the malware, virus, warms etc once the user visited. Main intension of the attack is to steal the victim information, user credentials or install the malware in the victim’s system. So, it is necessary to adapt the system which should detect the malicious URLs and prevent from the attack. Researchers suggest numerous methods but machine learning based detection method performs better then methods. This paper presents the light weighted method which includes only lexical features of the URL. The result shows the Random Forest classifier performs better than the other classifiers in terms of accuracy.
Continuous-Variable Quantum Teleportation Using a Microwave-Enabled Plasmonic Graphene Waveguide

We present a scheme to generate continuous-variable bipartite entanglement between two optical modes in a hybrid optical-microwave-plasmonic graphene waveguide system. In this scheme, we exploit the interaction of two light fields coupled to the same microwave mode via plasmonic graphene waveguide to generate two-mode squeezing, which can be used for continuous-variable quantum teleportation of the light signals over large distances. Furthermore, we study the teleportation fidelity of an unknown coherent state. The teleportation protocol is robust against the thermal noise associated with the microwave degree of freedom.
Refining Action Segmentation with Hierarchical Video Representations

In this paper, we propose Hierarchical Action Segmentation Refiner (HASR), which can refine temporal action segmentation results from various models by understanding the overall context of a given video in a hierarchical way. When a backbone model for action segmentation estimates how the given video can be segmented, our model extracts segment-level representations based on frame-level features, and extracts a video-level representation based on the segment-level representations. Based on these hierarchical representations, our model can refer to the overall context of the entire video, and predict how the segment labels that are out of context should be corrected. Our HASR can be plugged into various action segmentation models (MS-TCN, SSTDA, ASRF), and improve the performance of state-of-the-art models based on three challenging datasets (GTEA, 50Salads, and Breakfast). For example, in 50Salads dataset, the segmental edit score improves from 67.9% to 77.4% (MS-TCN), from 75.8% to 77.3% (SSTDA), from 79.3% to 81.0% (ASRF). In addition, our model can refine the segmentation result from the unseen backbone model, which was not referred to when training HASR. This generalization performance would make HASR be an effective tool for boosting up the existing approaches for temporal action segmentation. Our code is available at https://github.com/cotton-ahn/HASR_iccv2021.
Synthesized Feature based Few-Shot Class-Incremental Learning on a Mixture of Subspaces

Few-shot class incremental learning (FSCIL) aims to incrementally add sets of novel classes to a well-trained base model in multiple training sessions with the restriction that only a few novel instances are available per class. While learning novel classes, FSCIL methods gradually forget base (old) class training and overfit to a few novel class samples. Existing approaches have addressed this problem by computing the class prototypes from the visual or semantic word vector domain. In this paper, we propose addressing this problem using a mixture of subspaces. Subspaces define the cluster structure of the visual domain and help to describe the visual and semantic domain considering the overall distribution of the data. Additionally, we propose to employ a variational autoencoder (VAE) to generate synthesized visual samples for augmenting pseudo-feature while learning novel classes incrementally. The combined effect of the mixture of subspaces and synthesized features reduces the forgetting and overfitting problem of FSCIL. Extensive experiments on three image classification datasets show that our proposed method achieves competitive results compared to state-of-the-art methods.