Reviewerly/exHarmony huggingface dataset:
Time-Equivariant Contrastive Video Representation Learning

We introduce a novel self-supervised contrastive learning method to learn representations from unlabelled videos. Existing approaches ignore the specifics of input distortions, e.g., by learning invariance to temporal transformations. Instead, we argue that video representation should preserve video dynamics and reflect temporal manipulations of the input. Therefore, we exploit novel constraints to build representations that are equivariant to temporal transformations and better capture video dynamics. In our method, relative temporal transformations between augmented clips of a video are encoded in a vector and contrasted with other transformation vectors. To support temporal equivariance learning, we additionally propose the self-supervised classification of two clips of a video into 1. overlapping 2. ordered, or 3. unordered. Our experiments show that time-equivariant representations achieve state-of-the-art results in video retrieval and action recognition benchmarks on UCF101, HMDB51, and Diving48.
Graph Fusion Network for Text Classification

Text classification is an important and classical problem in natural language processing. Recently, Graph Neural Networks (GNNs) have been widely applied in text classification and achieved outstanding performance. Despite the success of GNNs on text classification, existing methods are still limited in two main aspects. On the one hand, transductive methods cannot easily adapt to new documents. Since transductive methods incorporate all documents into their text graph, they need to reconstruct the whole graph and retrain their system from scratch when new documents come. However, this is not applicable to real-world situations. On the other hand, many state-of-the-art algorithms ignore the quality of text graphs, which may lead to sub-optimal performance. To address these problems, we propose a Graph Fusion Network (GFN), which can overcome these limitations and boost text classification performance. In detail, in the graph construction stage, we build homogeneous text graphs with word nodes, which makes the learning system capable of making inference on new documents without rebuilding the whole text graph. Then, we propose to transform external knowledge into structural information and integrate different views of text graphs to capture more structural information. In the graph reasoning stage, we divide the process into three steps: graph learning, graph convolution, and graph fusion. In the graph learning step, we adopt a graph learning layer to further adapt text graphs. In the graph fusion step, we design a multi-head fusion module to integrate different opinions. Experimental results on five benchmarks demonstrate the superiority of our proposed method.
Conditional Invertible Neural Networks for Medical Imaging

Over recent years, deep learning methods have become an increasingly popular choice for solving tasks from the field of inverse problems. Many of these new data-driven methods have produced impressive results, although most only give point estimates for the reconstruction. However, especially in the analysis of ill-posed inverse problems, the study of uncertainties is essential. In our work, we apply generative flow-based models based on invertible neural networks to two challenging medical imaging tasks, i.e., low-dose computed tomography and accelerated medical resonance imaging. We test different architectures of invertible neural networks and provide extensive ablation studies. In most applications, a standard Gaussian is used as the base distribution for a flow-based model. Our results show that the choice of a radial distribution can improve the quality of reconstructions.
Underwater Image Enhancement With Lightweight Cascaded Network

Due to light scatter and absorption in waterbody, underwater imaging can be easily impaired with low contrast and visual distortion. The resulting images are often unable to meet the quality requirements of human perception and computer processing. Therefore, Underwater Image Enhancement (UIE) has been attracting extensive research efforts. Although deep learning has demonstrated its great success in many vision tasks, its huge amounts of parameters and computations are not conducive to UIE in resource-limited scenarios. In this paper, we address this issue by proposing a Lightweight Cascaded Network (LCNet) based on Laplacian image pyramids. At each pyramid level, we implement cascaded blocks upon a residual network. Specifically, high quality residuals can be progressively predicted with significantly reduced complexity in a coarse-to-fine fashion. Furthermore, these sub-networks are recursively nested to build our LCNet, thereby reducing the overall computational complexity with reused parameters. Extensive experiments demonstrate that the proposed method performs favorably against the state-of-the-arts in terms of visual quality, model parameters and complexity.
Adaptive Resource Optimized Edge Federated Learning in Real-Time Image Sensing Classifications

With the exponential growth of the Internet of things (IoT) in remote sensing image applications, network resource orchestration and data privacy are significant aspects to handle in bigdata cellular networks. The image data sharing procedure toward central cloud servers in order to perform real-time classifications has leaked client personalization and heavily burdened the communication networks. Thus, the deployment of IoT image sensors in privacy-constrained sectors requires an optimized federated learning (FL) scheme to efficiently consider both aspects of securing data privacy and maximizing the model accuracy with sufficient communication and computation resources. In this article, an adaptive model communication scheme with virtual resource optimization for edge FL is proposed by converging a deep q-learning algorithm to enforce a self-learning agent interacting with network functions virtualization orchestrator and software-defined networking based architecture. The agent targets to optimize the resource control policy of virtual multi-access edge computing entities in virtualized infrastructure manager. The proposed scheme trains the learning model and weighs the optimal actions for particular network states by using an epsilon-greedy strategy. In the exploitation phase, the scheme considers multiple spatial-resolution sensing conditions and allocates computation offloading resources for global multiconvolutional neural networks model aggregation based on the congestion states. In the simulation results, the quality of service and global collaborative model performance metrics were evaluated in terms of delay, packet drop ratios, packet delivery ratios, loss values, and overall accuracy.
Software security patch management - A systematic literature review of challenges, approaches, tools and practices

Software security patch management purports to support the process of patching known software security vulnerabilities. Patching security vulnerabilities in large and complex systems is a hugely challenging process that involves multiple stakeholders making several interdependent technological and socio-technical decisions. Given the increasing recognition of the importance of software security patch management, it is important and timely to systematically review and synthesise the relevant literature on this topic. This paper aims at systematically reviewing the state of the art of software security patch management to identify the socio-technical challenges in this regard, reported solutions (i.e., approaches, tools, and practices), the rigour of the evaluation and the industrial relevance of the reported solutions, and to identify the gaps for future research. We conducted a systematic literature review of 72 studies published from 2002 to March 2020, with extended coverage until September 2020 through forward snowballing. We identify 14 socio-technical challenges in software security patch management, 18 solution approaches, tools and practices mapped onto the software security patch management process. We provide a mapping between the solutions and challenges to enable a reader to obtain a holistic overview of the gap areas. The findings also reveal that only 20.8% of the reported solutions have been rigorously evaluated in industrial settings. Our results reveal that 50% of the common challenges have not been directly addressed in the solutions and that most of them (38.9%) address the challenges in one phase of the process, namely vulnerability scanning, assessment and prioritisation. Based on the results that highlight the important concerns in software security patch management and the lack of solutions, we recommend a list of future research directions. This study also provides useful insights about different opportunities for practitioners to adopt new solutions and understand the variations of their practical utility.
Metadiscourse in online advertising: Exploring linguistic and visual metadiscourse in social media advertisements

Advertising is a powerful means of flowing information from sellers to buyers and it highly influences and smartly persuades people to do actions. This paper investigates the frequency and the use of the linguistic and the visual metadiscourse markers in SM advertisements, in addition to the role they play in the construction of persuasion. A corpus of 50 advertisements was extracted from three SM platforms: Instagram, Snapchat and Twitter, and both frequency-based and qualitative methods were integrated in the analysis. Drawing on Hyland’s (2005a, 2005b) models of metadiscourse and Kumpf's visual metadiscourse (2000), the study compared the use of these two distinct types of metadiscourse, and explained how visual metadiscourse complements linguistic metadiscourse in constructing persuasive messages and enticing customers into buying products. Results revealed that all categories of visual metadiscourse, especially chunking, convention, and consistency, were highly evident in the data investigated. Results also showed that engagement markers and directives exhibited higher frequencies than other linguistic metadiscourse and they were employed as effective techniques of persuasive language. The study concluded that metadiscourse, both visually and linguistically, plays a vital role in structuring the discourse, engaging the audience, and catching the consumers’ interest, thus becoming an integral aspect of persuasive writing. • The use of linguistic and visual metadiscourse in online advertising was examined. • Both verbal and visual metadiscourse play a role in the construction of persuasion. • Persuasiveness of the advertisements is achieved by the use of engagement markers. • Visual devices constitute meaning and help the reader get the desired message. • Advertising is a hybrid communicative process based on textual and visual elements.
Low-light Image Enhancement via Breaking Down the Darkness

Images captured in low-light environments often suffer from complex degradation. Simply adjusting light would inevitably result in burst of hidden noise and color distortion. To seek results with satisfied lighting, cleanliness, and realism from degraded inputs, this paper presents a novel framework inspired by the divide-and-rule principle, greatly alleviating the degradation entanglement. Assuming that an image can be decomposed into texture (with possible noise) and color components, one can specifically execute noise removal and color correction along with light adjustment. For this purpose, we propose to convert an image from the RGB colorspace into a luminance-chrominance one. An adjustable noise suppression network is designed to eliminate noise in the brightened luminance, having the illumination map estimated to indicate noise amplification levels. The enhanced luminance further serves as guidance for the chrominance mapper to generate realistic colors. Extensive experiments are conducted to reveal the effectiveness of our design, and demonstrate its superiority over state-of-the-art alternatives both quantitatively and qualitatively on several benchmark datasets. Our code has been made publicly available at https://github.com/mingcv/Bread .
Urdu Sentiment Analysis via Multimodal Data Mining Based on Deep Learning Algorithms

Every day, a massive amount of text, audio, and video data is published on websites all over the world. This valuable data can be used to gauge global trends and public perceptions. Companies are showcasing their preferred advertisements to consumers based on their online behavioral trends. Carefully analyzing this raw data to uncover useful patterns is indeed a challenging task, even more so for a resource-constrained language such as Urdu. A unique Urdu language-based multimodal dataset containing 1372 expressions has been presented in this paper as a first step to address the challenge to reveal useful patterns. Secondly, we have also presented a novel framework for multimodal sentiment analysis (MSA) that incorporates acoustic, visual, and textual responses to detect context-aware sentiments. Furthermore, we have used both decision-level and feature-level fusion methods to improve sentiment polarity prediction. The experimental results demonstrated that integration of multimodal features improves the polarity detection capability of the proposed algorithm from 84.32% (with unimodal features) to 95.35% (with multimodal features).
End-to-End Referring Video Object Segmentation with Multimodal Transformers

The referring video object segmentation task (RVOS) involves segmentation of a text-referred object instance in the frames of a given video. Due to the complex nature of this multimodal task, which combines text reasoning, video understanding, instance segmentation and tracking, existing approaches typically rely on sophisticated pipelines in order to tackle it. In this paper, we propose a simple Transformer-based approach to RVOS. Our framework, termed Multimodal Tracking Transformer (MTTR), models the RVOS task as a sequence prediction problem. Following recent advancements in computer vision and natural language processing, MTTR is based on the realization that video and text can be processed together effectively and elegantly by a single multimodal Transformer model. MTTR is end-to-end trainable, free of text-related inductive bias components and requires no additional mask-refinement post-processing steps. As such, it simplifies the RVOS pipeline considerably compared to existing methods. Evaluation on standard benchmarks reveals that MTTR significantly outperforms previous art across multiple metrics. In particular, MTTR shows impressive +5.7 and +5.0 mAP gains on the A2D-Sentences and JHMDB-Sentences datasets respectively, while processing 76 frames per second. In addition, we report strong results on the public validation set of Refer-YouTube-VOS, a more challenging RVOS dataset that has yet to receive the attention of researchers. The code to reproduce our experiments is avail-able at https://github.com/mttr2021/MTTR.
Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations

A classical problem in computer vision is to infer a 3D scene representation from few images that can be used to render novel views at interactive rates. Previous work focuses on reconstructing pre-defined 3D representations, e.g. textured meshes, or implicit representations, e.g. radiance fields, and often requires input images with precise camera poses and long processing times for each novel scene. In this work, we propose the Scene Representation Transformer (SRT), a method which processes posed or unposed RGB images of a new area, infers a “set-latent scene representation ”, and synthesises novel views, all in a single feed-forward pass. To calculate the scene representation, we propose a generalization of the Vision Transformer to sets of images, enabling global information integration, and hence 3D reasoning. An efficient decoder transformer parameterizes the light field by attending into the scene representation to render novel views. Learning is supervised end-to-end by minimizing a novel-view reconstruction error. We show that this method outperforms recent baselines in terms of PSNR and speed on synthetic datasets, including a new dataset created for the paper. Further, we demonstrate that SRT scales to support interactive visualization and semantic segmentation of real-world outdoor environments using Street View imagery.
MAC-Net: Model-Aided Nonlocal Neural Network for Hyperspectral Image Denoising

Hyperspectral image (HSI) denoising is an ill-posed inverse problem. The underlying physical model is always important to tackle this problem, which is unfortunately ignored by most of the current deep learning (DL)-based methods, producing poor denoising performance. To address this issue, this article introduces an end-to-end model-aided nonlocal neural network (MAC-Net) which simultaneously takes the spectral low-rank model and spatial deep prior into account for HSI noise reduction. Specifically, motivated by the success of the spectral low-rank model in depicting the strong spectral correlations and the nonlocal similarity prior in capturing spatial long-range dependencies, we first build a spectral low-rank model and then integrate a nonlocal U-Net into the model. In this way, we obtain a hybrid model-based and DL-based HSI denoising method where the spatial local and nonlocal multi-scale and spectral low-rank structures are effectively exploited. After that, we cast the optimization and denoising procedure of the hybrid method as a forward process of a neural network and introduce a set of learnable modules to yield our MAC-Net. Compared with traditional model-based methods, our MAC-Net overcomes the difficulties of accurate modeling, thanks to the strong learning and representation ability of DL. Unlike most “black-box” DL-based methods, the spectral low-rank model is beneficial to increase the generalization ability of the network and decrease the requirement of training samples. Experimental results on the natural and remote-sensing HSIs show that MAC-Net achieves state-of-the-art performance over both model-based and DL-based methods. The source code and data of this article will be made publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/bearshng/mac-net</uri> for reproducible research.
Detection and Localization of Multiple Image Splicing Using MobileNet V1

In modern society, digital images have become a prominent source of information and medium of communication. The easy availability of image-altering softwares have greatly reduced the expenses and expertise required to exploit visual tampering. Images can, however, be simply altered using these freely available image editing softwares. Two or more images are combined to generate a new image that can transmit information across social media platforms to influence the people in the society. This information may have both positive and negative consequences. Hence there is a need to develop a technique that will detect and locate a multiple image splicing forgery in an image. This research work proposes multiple image splicing forgery detection using Mask R-CNN, with a backbone as a MobileNet V1. It also calculates the percentage score of a forged region of multiple spliced images. The comparative analysis of the proposed work with the variants of ResNet is performed. The proposed model is trained and tested using the MISD dataset, and it is observed that the proposed model outperforms the variants of ResNet models (ResNet 51,101 and 151). The proposed model achieves an average precision of 82% on Multiple Image Splicing Dataset, 74% on CASIA 1.0, 81% on WildWeb, and 86% on Columbia Gray. The F1 Score of the proposed method on MISD was 67%, 64% on CASIA 1.0 68% on WildWeb, and 61% on Columbia Gray, outperforming ResNet variants.
Syntactic model-based human body 3D reconstruction and event classification via association based features mining and deep learning

The study of human posture analysis and gait event detection from various types of inputs is a key contribution to the human life log. With the help of this research and technologies humans can save costs in terms of time and utility resources. In this paper we present a robust approach to human posture analysis and gait event detection from complex video-based data. For this, initially posture information, landmark information are extracted, and human 2D skeleton mesh are extracted, using this information set we reconstruct the human 2D to 3D model. Contextual features, namely, degrees of freedom over detected body parts, joint angle information, periodic and non-periodic motion, and human motion direction flow, are extracted. For features mining, we applied the rule-based features mining technique and, for gait event detection and classification, the deep learning-based CNN technique is applied over the mpii-video pose, the COCO, and the pose track datasets. For the mpii-video pose dataset, we achieved a human landmark detection mean accuracy of 87.09% and a gait event recognition mean accuracy of 90.90%. For the COCO dataset, we achieved a human landmark detection mean accuracy of 87.36% and a gait event recognition mean accuracy of 89.09%. For the pose track dataset, we achieved a human landmark detection mean accuracy of 87.72% and a gait event recognition mean accuracy of 88.18%. The proposed system performance shows a significant improvement compared to existing state-of-the-art frameworks.
A modified Marine predators algorithm for solving single- and multi-objective combined economic emission dispatch problems

Decreasing the emission of greenhouse gases and the fuel cost are the main principles of the combined economic emission dispatch (CEED) optimization problem. In this work, a modified version of the Marine Predators Algorithm (MMPA), is proposed and applied for solving single- and bi-objective CEED problems. MMPA is suggested to improve the performance of conventional MPA. It includes a comprehensive learning approach to share the best experiences among all the individuals to avoid premature convergence. The MMPA increases the populations' effectiveness to reach the optimum fitness solution and this is helpful in cases of single- and bi-objective functions. The proposed MMPA has been mathematically assessed with twenty-eight 50-dimensional benchmarks of CEC2017. Also, it is successfully tested on four electrical power generation systems (each of them with 3, 5, 6, and 26 generating units respectively). The results achieved by MMPA are compared with those obtained by the original MPA and other recent optimization methods. Concerning the bi-objective problem, the Pareto approach is combined with the proposed MMPA and conventional MPA techniques to reach a group of non-dominated (ND) solutions, and then the fuzzy method is utilized to choose the best compromise solution. The reached results display that the proposed MMPA technique gives better performance than the conventional MPA and the others.