Reviewerly/exHarmony huggingface dataset:
A Survey on Multimodal Dialogue Systems: Recent Advances and New Frontiers

Recently, there has been growing interest in the field of multimodal dialogue systems. Different from traditional unimodal dialogue systems, our task needs to understand the context of multiple modalities before responding to users’ utterances. In this paper, we present a detailed survey of the recent advances in multimodal dialogue systems and discuss some possible research directions. In particular, we categorize the dialogue systems into two basic tasks, including outputting textual responses and outputting visual responses. In these tasks, there are two main challenges, that is, the heterogeneity gap and the semantic gap. Then, we analyze the key techniques used to solve these challenges. Moreover, we review benchmark datasets and popular evaluation metrics comprehensively. Finally, we give some promising directions for future works.
Localization of Inpainting Forgery With Feature Enhancement Network

Inpainting the given region of an image is a typical requirement in computer vision. Conventional inpainting, through exemplar-based or diffusion-based strategies, can create realistic inpainted images at a very low cost. Also, such easy-to-use manipulation poses new security threats. Therefore, the detection of inpainting has attracted considerable attention from researchers. However, the existing methods are typically not suitable for the general detection of various inpainting algorithms. Motivated by this, in this work, an efficient feature enhancement network is proposed to locate the inpainted regions in the digital image. First, we design an artifact enhancement block to effectively capture the traces left by diffusion or exemplar-based inpainting. Then, the VGGNet is used as a feature extractor to describe advanced and low-resolution features. Finally, to take full advantage of enhanced features, we concatenate the features obtained by the feature extractor and the up-sampling operations. Extensive experimental evaluations, covering benchmarking, ablation, robustness, generalization, and efficiency studies, confirm the usefulness of the proposed method. This is especially true on the conventional inpainting dataset, our method obtains an average F1 score 7.63% higher than the second-best method. Theoretical and numerical analyses support the effectiveness of our feature enhancement network in representing the artifacts in inpainted images, exhibiting better potential for real-world forensics than various state-of-the-art strategies.
An Improved APAP Algorithm via Line Segment Correction for UAV Multispectral Image Stitching

Unmanned aerial vehicles (UAV) low-altitude remote sensing has been extensively applied in agriculture as an important way of monitoring the growth and physiological conditions of crops. Due to the limited imaging width, multispectral images captured by UAV need to be stitched in order to obtain the panoramic image of the whole area. However, sometimes it is difficult to stitch multispectral images by using the traditional stitching methods when there is blurring or uneven local brightness of the images. Accordingly, in this paper, we propose an effective and robust UAV multispectral image stitching method based on an improved As-Projective-As-Possible (APAP) algorithm via line segment correction. We handle the stitching problem in two steps: firstly, inter-band registration and global homography transformation are used to ensure the precision of stitching results. Then the line segment detection and correction based on line segment detector and least-square estimation is conducted, which can protect the unique line segment structure in farmland area. The experimental results substantiate the superiority of our method both visually and quantitatively when compared with state-of-the-art methods.
Neither “Hear” Nor “Their”: Interrogating Gender Neutrality in Robots

Gender is a social framework through which people organize themselves-and non-human subjects, including robots. Research stretching back decades has found evidence that people tend to gender artificial agents unwittingly, even with the slightest cue of humanlike features in voice, body, role, and other social features. This has led to the notion of gender neutrality in robots: ways in which we can avoid gendering robots in line with human models, as well explorations of extra-human genders. This rapid review critically surveyed the literature to capture the state of art on gender neutrality in robots that interact with people. We present findings on theory, methods, results, and reflexivity. We interrogate the very idea that robot gender/ing can be neutral and explore alternate ways of approaching gendering through the design and study of robots interacting with people.
Modified UNet++ with atrous spatial pyramid pooling for blood cell image segmentation

Blood cell image segmentation is an important part of the field of computer-aided diagnosis. However, due to the low contrast, large differences in cell morphology and the scarcity of labeled images, the segmentation performance of cells cannot meet the requirements of an actual diagnosis. To address the above limitations, we present a deep learning-based approach to study cell segmentation on pathological images. Specifically, the algorithm selects UNet++ as the backbone network to extract multi-scale features. Then, the skip connection is redesigned to improve the degradation problem and reduce the computational complexity. In addition, the atrous spatial pyramid pooling (ASSP) is introduced to obtain cell image information features from each layer through different receptive domains. Finally, the multi-sided output fusion (MSOF) strategy is utilized to fuse the features of different semantic levels, so as to improve the accuracy of target segmentation. Experimental results on blood cell images for segmentation and classification (BCISC) dataset show that the proposed method has significant improvement in Matthew's correlation coefficient (Mcc), Dice and Jaccard values, which are better than the classical semantic segmentation network.
Motion Sensitive Contrastive Learning for Self-supervised Video Representation

Contrastive learning has shown great potential in video representation learning. However, existing approaches fail to sufficiently exploit short-term motion dynamics, which are crucial to various downstream video understanding tasks. In this paper, we propose Motion Sensitive Contrastive Learning (MSCL) that injects the motion information captured by optical flows into RGB frames to strengthen feature learning. To achieve this, in addition to clip-level global contrastive learning, we develop Local Motion Contrastive Learning (LMCL) with frame-level contrastive objectives across the two modalities. Moreover, we introduce Flow Rotation Augmentation (FRA) to generate extra motion-shuffled negative samples and Motion Differential Sampling (MDS) to accurately screen training samples. Extensive experiments on standard benchmarks validate the effectiveness of the proposed method. With the commonly-used 3D ResNet-18 as the backbone, we achieve the top-1 accuracies of 91.5% on UCF101 and 50.3% on Something-Something v2 for video classification, and a 65.6% Top-1 Recall on UCF101 for video retrieval, notably improving the state of the art.
SphereFed: Hyperspherical Federated Learning

Federated Learning aims at training a global model from multiple decentralized devices (i.e. clients) without exchanging their private local data. A key challenge is the handling of non-i.i.d. (independent identically distributed) data across multiple clients that may induce disparities of their local features. We introduce the Hyperspherical Federated Learning (SphereFed) framework to address the non-i.i.d. issue by constraining learned representations of data points to be on a unit hypersphere shared by clients. Specifically, all clients learn their local representations by minimizing the loss with respect to a fixed classifier whose weights span the unit hypersphere. After federated training in improving the global model, this classifier is further calibrated with a closed-form solution by minimizing a mean squared loss. We show that the calibration solution can be computed efficiently and distributedly without direct access of local data. Extensive experiments indicate that our SphereFed approach is able to improve the accuracy of multiple existing federated learning algorithms by a considerable margin (up to 6% on challenging datasets) with enhanced computation and communication efficiency across datasets and model architectures.
Experimental Evaluation Using Head Motion and Augmented Reality to Intuitively Control a Flexible Endoscope

Thoracoscopic procedures require an assistant to hold and control the camera while the surgeon performs the surgical task. This paper presents an approach in which surgeons can control camera orientation using their head movements, allowing them to steer a flexible endoscope without the need for a camera assistant during the operation. Additionally, an augmented reality headset has been integrated into the head movement control system to serve as a virtual display monitor capable of following the user's gaze. Experiments were conducted to assess the feasibility of the head-controlled approach compared to the manual control method by conducting camerapointing experiments performed by clinicians and trained nonclinician participants at two difficulty levels. The results from the camera-pointing experiments have shown that the developed head-controlled endoscope has a statistically faster reaching time performance compared to manual use of the flexible endoscope in high difficulty index tasks with clinician participants (p=0.04), and in both lower and high difficulty index tasks with non-clinician participants (p=0.03). The head-controlled robotic endoscope approach enables surgeons to intuitively control the camera during an operation, while simultaneously performing other tasks using their hands, without sacrificing camera steering accuracy.
Unsupervised Pose-aware Part Decomposition for Man-Made Articulated Objects

Man-made articulated objects exist widely in the real world. However, previous methods for unsupervised part decomposition are unsuitable for such objects because they assume a spatially fixed part location, resulting in inconsistent part parsing. In this paper, we propose PPD (unsupervised Pose-aware Part Decomposition) to address a novel setting that explicitly targets man-made articulated objects with mechanical joints, considering the part poses in part parsing. As an analysis-by-synthesis approach, We show that category-common prior learning for both part shapes and poses facilitates the unsupervised learning of (1) part parsing with abstracted part shapes, and (2) part poses as joint parameters under single-frame shape supervision. We evaluate our method on synthetic and real datasets, and we show that it outperforms previous works in consistent part parsing of the articulated objects based on comparable part pose estimation performance to the supervised baseline.
An optimistic firefly algorithm-based deep learning approach for sentiment analysis of COVID-19 tweets

The unprecedented rise in the number of COVID-19 cases has drawn global attention, as it has caused an adverse impact on the lives of people all over the world. As of December 31, 2021, more than 2, 86, 901, 222 people have been infected with COVID-19. The rise in the number of COVID-19 cases and deaths across the world has caused fear, anxiety and depression among individuals. Social media is the most dominant tool that disturbed human life during this pandemic. Among the social media platforms, Twitter is one of the most prominent and trusted social media platforms. To control and monitor the COVID-19 infection, it is necessary to analyze the sentiments of people expressed on their social media platforms. In this study, we proposed a deep learning approach known as a long short-term memory (LSTM) model for the analysis of tweets related to COVID-19 as positive or negative sentiments. In addition, the proposed approach makes use of the firefly algorithm to enhance the overall performance of the model. Further, the performance of the proposed model, along with other state-of-the-art ensemble and machine learning models, has been evaluated by using performance metrics such as accuracy, precision, recall, the AUC-ROC and the F1-score. The experimental results reveal that the proposed LSTM + Firefly approach obtained a better accuracy of 99.59% when compared with the other state-of-the-art models.
TinyABE: Unrestricted Ciphertext-Policy Attribute-Based Encryption for Embedded Devices and Low-Quality Networks

Ciphertext-policy attribute-based encryption (CP-ABE) has attracted much interest from the practical community to enforce access control in distributed settings such as the Internet of Things (IoT). In such settings, encryption devices are often constrained, having small memories and little computational power, and the associated networks are lossy. To optimize both the ciphertext sizes and the encryption speed is therefore paramount. In addition, the master public key needs to be small enough to fit in the encryption device’s memory. At the same time, the scheme needs to be expressive enough to support common access control models. Currently, however, the state of the art incurs undesirable efficiency trade-offs. Existing schemes often have linear ciphertexts, and consequently, the ciphertexts may be too large and encryption may be too slow. In contrast, schemes with small ciphertexts have extremely large master public keys, and are generally computationally inefficient. In this work, we propose TinyABE: a novel CP-ABE scheme that is expressive and can be configured to be efficient enough for settings with embedded devices and low-quality networks. In particular, we demonstrate that our scheme can be configured such that the ciphertexts are small, encryption is fast and the master public key is small enough to fit in memory. From a theoretical standpoint, the new scheme and its security proof are non-trivial generalizations of the expressive scheme with constant-size ciphertexts by Agrawal and Chase (TCC’16, Eurocrypt’17) and its proof to the unbounded setting. By using techniques of Rouselakis and Waters (CCS’13), we remove the restrictions that the Agrawal-Chase scheme imposes on the keys and ciphertexts, making it thus more flexible. In this way, TinyABE is especially suitable for IoT devices and networks.
Writing in the Air: Unconstrained Text Recognition From Finger Movement Using Spatio-Temporal Convolution

We introduce a new benchmark dataset for the challenging writing in the air (WiTA) task—an elaborate task bridging vision and NLP. WiTA implements an intuitive and natural writing method with finger movement for HCI. Our WiTA dataset will facilitate the development of data-driven WiTA systems which thus far have displayed unsatisfactory performance—due to lack of dataset as well as traditional statistical models they have adopted. Our dataset consists of five sub-datasets in two languages (Korean and English) and amounts to 209,926 video instances from 122 participants. We capture finger movement for WiTA with RGB cameras to ensure wide accessibility and cost-efficiency. Next, we propose spatio-temporal residual network architectures inspired by 3D ResNet. These models perform unconstrained text recognition from finger movement, guarantee a real-time operation ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&gt;$</tex-math></inline-formula> 100 FPS), and will serve as an evaluation standard. Our dataset and the source codes are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Uehwan/WiTA</uri> .
No Token Left Behind: Explainability-Aided Image Classification and Generation

The application of zero-shot learning in computer vision has been revolutionized by the use of image-text matching models. The most notable example, CLIP, has been widely used for both zero-shot classification and guiding generative models with a text prompt. However, the zero-shot use of CLIP is unstable with respect to the phrasing of the input text, making it necessary to carefully engineer the prompts used. We find that this instability stems from a selective similarity score, which is based only on a subset of the semantically meaningful input tokens. To mitigate it, we present a novel explainability-based approach, which adds a loss term to ensure that CLIP focuses on all relevant semantic parts of the input, in addition to employing the CLIP similarity loss used in previous works. When applied to one-shot classification through prompt engineering, our method yields an improvement in the recognition rate, without additional training or fine-tuning. Additionally, we show that CLIP guidance of generative models using our method significantly improves the generated images. Finally, we demonstrate a novel use of CLIP guidance for text-based image generation with spatial conditioning on object location, by requiring the image explainability heatmap for each object to be confined to a pre-determined bounding box. Our code is available at https://github.com/apple/ml-no-token-left-behind .
Surrogate-Assisted Multiobjective Neural Architecture Search for Real-Time Semantic Segmentation

The architectural advancements in deep neural networks have led to remarkable leap-forwards across a broad array of computer vision tasks. Instead of relying on human expertise, neural architecture search (NAS) has emerged as a promising avenue toward automating the design of architectures. While recent achievements on image classification have suggested opportunities, the promises of NAS have yet to be thoroughly assessed on more challenging tasks of semantic segmentation. The main challenges of applying NAS to semantic segmentation arise from two aspects: 1) high-resolution images to be processed; 2) additional requirement of real-time inference speed (i.e., real-time semantic segmentation) for applications such as autonomous driving. To meet such challenges, we propose a surrogate-assisted multiobjective method in this article. Through a series of customized prediction models, our method effectively transforms the original NAS task to an ordinary multiobjective optimization problem. Followed by a hierarchical prescreening criterion for in-fill selection, our method progressively achieves a set of efficient architectures trading-off between segmentation accuracy and inference speed. Empirical evaluations on three benchmark datasets together with an application using Huawei Atlas 200 DK suggest that our method can identify architectures significantly outperforming existing state-of-the-art architectures designed both manually by human experts and automatically by other NAS methods. Code is available from here.
M2SCN: Multi-Model Self-Correcting Network for Satellite Remote Sensing Single-Image Dehazing

Remote sensing (RS) image dehazing is an effective means to enhance the quality of hazy RS images. However, existing dehazing methods are ineffective in dealing with nonhomogeneous RS haze scenes. To tackle this deficiency, we design a multi-model joint estimation (M2JE) module and a self-correcting (SC) module to construct a unified end-to-end network for RS image dehazing, termed the multi-model SC network (M2SCN). Specifically, the M2JE module regards the dehazing process as a multi-model ensemble problem, so as to improve the generalization ability of the model. The SC module can gradually correct the error in the intermedia features extracted by the network, thus enabling the network to deal with nonhomogeneous hazy images. Extensive experiments are conducted to demonstrate that our proposed M2SCN performs favorably against state-of-the-art methods on popular RS image dehazing benchmark datasets.
Performance Analysis of CNN Models for Mobile Device Eye Tracking with Edge Computing

Eye-tracking is a technique used for determining where users are looking and how long they keep their gaze fixed on a particular location. Developments in mobile technology have made mobile applications pervasive; however, eye tracking on mobile devices is still uncommon. This paper proposes a mobile edge computing architecture for eye tracking. We evaluate four lightweight CNN models (LeNet-5, AlexNet, MobileNet, and ShuffleNet) for gaze estimation on mobile devices using a publicly available dataset called GazeCapture. In order to analyse the feasibility of different inference modes such as on-device, edge-based and cloud-based, we conduct an empirical measurement study to quantify inference time, communication time, and resource consumption in these inference modes. Our analysis indicates that while cloud-based inference provides faster predictions, the communication time between the mobile device and the cloud introduces significant latency into the application. This effectively eliminates the ability to perform real-time eye tracking via cloud inference. Furthermore, our findings show that on-device inference performance is limited by energy and memory consumption, making it unsuitable to provide a high-quality user experience. Additionally, we demonstrated that edge-based inference results in a reasonable response time, memory usage, and energy consumption for eye-tracking applications on mobile devices.
Tiny RNN Model with Certified Robustness for Text Classification

Mobile artificial intelligence has recently gained more attention due to the increasing computing power of mobile devices and applications in computer vision, natural language processing, and internet of things. Although large pre-trained language models (e.g., BERT, GPT) have recently achieved the state-of-the-art results on text classification tasks, they are not well suited for latency critical applications on mobile devices. Therefore, it is essential to design tiny models to reduce their memory and computing requirements. Model compression has shown promising results for this goal. However, some significant challenges are yet to be addressed, such as information loss and adversarial robustness. This paper attempts to tackle these challenges through a new training scheme that minimizes the information loss by maximizing the mutual information between the feature representations learned from the large and tiny models. In addition, we propose a certifiably robust defense method named GradMASK that masks a certain proportion of words in an input text. It can defend against both character-level perturbations and word substitution-based attacks. We perform extensive experiments demonstrating the effectiveness of our approach by comparing our tiny RNN models with compact RNNs (e.g., FastGRNN) and compressed RNNs (e.g., PRADO) in clean and adversarial test settings.
On the cross-modal transfer from natural language to code through adapter modules

Pre-trained neural Language Models (PTLM), such as CodeBERT, are recently used in software engineering as models pre-trained on large source code corpora. Their knowledge is transferred to downstream tasks (e.g. code clone detection) via fine-tuning. In natural language processing (NLP), other alternatives for transferring the knowledge of PTLMs are explored through using adapters, compact, parameter efficient modules inserted in the layers of the PTLM. Although adapters are known to facilitate adapting to many downstream tasks compared to fine-tuning the model that require retraining all of the models' parameters- which owes to the adapters' plug and play nature and being parameter efficient-their usage in software engineering is not explored.