Reviewerly/exHarmony huggingface dataset:
Modeling Multi-Label Action Dependencies for Temporal Action Localization

Real-world videos contain many complex actions with inherent relationships between action classes. In this work, we propose an attention-based architecture that models these action relationships for the task of temporal action localization in untrimmed videos. As opposed to previous works that leverage video-level co-occurrence of actions, we distinguish the relationships between actions that occur at the same time-step and actions that occur at different time-steps (i.e. those which precede or follow each other). We define these distinct relationships as action dependencies. We propose to improve action localization performance by modeling these action dependencies in a novel attention-based Multi-Label Action Dependency (MLAD) layer. The MLAD layer consists of two branches: a Cooccurrence Dependency Branch and a Temporal Dependency Branch to model co-occurrence action dependencies and temporal action dependencies, respectively. We observe that existing metrics used for multi-label classification do not explicitly measure how well action dependencies are modeled, therefore, we propose novel metrics that consider both co-occurrence and temporal dependencies between action classes. Through empirical evaluation and extensive analysis, we show improved performance over state-of-the-art methods on multi-label action localization benchmarks (MultiTHUMOS and Charades) in terms of f-mAP and our proposed metric. Code is publicly available at https://github.com/ptirupat/MLAD.
The influence of search engine optimization on Google's results

Search engine optimization (SEO) can significantly influence what is shown on the result pages of commercial search engines. However, it is unclear what proportion of (top) results have actually been optimized. We developed a tool that uses a semi-automatic approach to detect, based on a given URL, whether SEO measures were taken. In this multi-dimensional approach, we analyze the HTML code from which we extract information on SEO and analytics tools. Further, we extract SEO indicators on the page level and the website level (e.g., page descriptions and loading time of a website). We amend this approach by using lists of manually classified websites and use machine learning methods to improve the classifier. An analysis based on three datasets with a total of 1,914 queries and 256,853 results shows that a large fraction of pages found in Google is at least probably optimized, which is in line with statements from SEO experts saying that it is tough to gain visibility in search engines without applying SEO techniques.
Image Inpainting with External-internal Learning and Monochromic Bottleneck

Although recent inpainting approaches have demonstrated significant improvement with deep neural networks, they still suffer from artifacts such as blunt structures and abrupt colors when filling in the missing regions. To address these issues, we propose an external-internal inpainting scheme with a monochromic bottleneck that helps image inpainting models remove these artifacts. In the external learning stage, we reconstruct missing structures and details in the monochromic space to reduce the learning dimension. In the internal learning stage, we propose a novel internal color propagation method with progressive learning strategies for consistent color restoration. Extensive experiments demonstrate that our proposed scheme helps image inpainting models produce more structure-preserved and visually compelling results.
Reinforced Attention for Few-Shot Learning and Beyond

Few-shot learning aims to correctly recognize query samples from unseen classes given a limited number of support samples, often by relying on global embeddings of images. In this paper, we propose to equip the backbone network with an attention agent, which is trained by reinforcement learning. The policy gradient algorithm is employed to train the agent towards adaptively localizing the representative regions on feature maps over time. We further design a reward function based on the prediction of the held-out data, thus helping the attention mechanism to generalize better across the unseen classes. The extensive experiments show, with the help of the reinforced attention, that our embedding network has the capability to progressively generate a more discriminative representation in few-shot learning. Moreover, experiments on the task of image classification also show the effectiveness of the proposed design.
A Multiplexed Network for End-to-End, Multilingual OCR

Recent advances in OCR have shown that an end-to-end (E2E) training pipeline that includes both detection and recognition leads to the best results. However, many existing methods focus primarily on Latin-alphabet languages, often even only case-insensitive English characters. In this paper, we propose an E2E approach, Multiplexed Multilingual Mask TextSpotter, that performs script identification at the word level and handles different scripts with different recognition heads, all while maintaining a unified loss that simultaneously optimizes script identification and multiple recognition heads. Experiments show that our method outperforms the single-head model with similar number of parameters in end-to-end recognition tasks, and achieves state-of-the-art results on MLT17 and MLT19 joint text detection and script identification benchmarks. We believe that our work is a step towards the end-to-end trainable and scalable multilingual multi-purpose OCR system. Our code and model will be released.
VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs

We present VX2TEXT, a framework for text generation from multimodal inputs consisting of video plus text, speech, or audio. In order to leverage transformer networks, which have been shown to be effective at modeling language, each modality is first converted into a set of language embeddings by a learnable tokenizer. This allows our approach to perform multimodal fusion in the language space, thus eliminating the need for ad-hoc cross-modal fusion modules. To address the non-differentiability of tokenization on continuous inputs (e.g., video or audio), we utilize a relaxation scheme that enables end-to-end training. Furthermore, unlike prior encoder-only models, our network includes an autoregressive decoder to generate open-ended text from the multimodal embeddings fused by the language encoder. This renders our approach fully generative and makes it directly applicable to different "video+x to text" problems without the need to design specialized network heads for each task. The proposed framework is not only conceptually simple but also remarkably effective: experiments demonstrate that our approach based on a single architecture outperforms the state-of-the-art on three videobased text-generation tasks—captioning, question answering and audio-visual scene-aware dialog.
Lazy parameter tuning and control

Most evolutionary algorithms have multiple parameters and their values drastically affect the performance. Due to the often complicated interplay of the parameters, setting these values right for a particular problem is a challenging task. This task becomes even more complicated when the optimal parameter values change significantly during the run of the algorithm since then a dynamic parameter choice is necessary.
Variational Self-attention Network for Sequential Recommendation

Sequential recommendation has become an attractive topic in recommender systems. Existing sequential recommendation methods, including the methods based on the state-of-the-art self-attention mechanism, usually employ deterministic neural networks to represent user preferences as fixed-points in the latent feature spaces. However, the fixed-point vector lacks the ability to capture the uncertainty and dynamics of user preferences that are prevalent in recommender systems. In this paper, we propose a new Variational Self-Attention Network (VSAN), which introduces a variational autoencoder (VAE) into the self-attention network to capture latent user preferences. Specifically, we represent the obtained self-attention vector as density via variational inference, whose variance well characterizes the uncertainty of user preferences. Furthermore, we employ self-attention networks to learn the inference process and generative process of VAE, which well captures long-range and local dependencies. Finally, we evaluate our proposed method VSAN with two public real-world datasets. Our experimental results show the effectiveness of our model compared to the state-of-the-art approaches.
Automated evolutionary approach for the design of composite machine learning pipelines

The effectiveness of the machine learning methods for real-world tasks depends on the proper structure of the modeling pipeline. The proposed approach is aimed to automate the design of composite machine learning pipelines, which is equivalent to computation workflows that consist of models and data operations. The approach combines key ideas of both automated machine learning and workflow management systems. It designs the pipelines with a customizable graph-based structure, analyzes the obtained results, and reproduces them. The evolutionary approach is used for the flexible identification of pipeline structure. The additional algorithms for sensitivity analysis, atomization, and hyperparameter tuning are implemented to improve the effectiveness of the approach. Also, the software implementation on this approach is presented as an open-source framework. The set of experiments is conducted for the different datasets and tasks (classification, regression, time series forecasting). The obtained results confirm the correctness and effectiveness of the proposed approach in the comparison with the state-of-the-art competitors and baseline solutions.
Test-Time Fast Adaptation for Dynamic Scene Deblurring via Meta-Auxiliary Learning

In this paper, we tackle the problem of dynamic scene deblurring. Most existing deep end-to-end learning approaches adopt the same generic model for all unseen test images. These solutions are sub-optimal, as they fail to utilize the internal information within a specific image. On the other hand, a self-supervised approach, SelfDeblur, enables internal training within a test image from scratch, but it does not fully take advantage of large external datasets. In this work, we propose a novel self-supervised meta-auxiliary learning to improve the performance of deblurring by integrating both external and internal learning. Concretely, we build a self-supervised auxiliary reconstruction task that shares a portion of the network with the primary deblurring task. The two tasks are jointly trained on an external dataset. Furthermore, we propose a meta-auxiliary training scheme to further optimize the pretrained model as a base learner, which is applicable for fast adaptation at test time. During training, the performance of both tasks is coupled. Therefore, we are able to exploit the internal information at test time via the auxiliary task to enhance the performance of deblurring. Extensive experimental results across evaluation datasets demonstrate the effectiveness of test-time adaptation of the proposed method.
M<sup>3</sup>P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training

We present M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> P, a Multitask Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training into a unified framework via multitask pre-training. Our goal is to learn universal representations that can map objects occurred in different modalities or texts expressed in different languages into a common semantic space. In addition, to explicitly encourage fine-grained alignment between images and non-English languages, we also propose Multimodal Code-switched Training (MCT) to combine monolingual pre-training and multimodal pre-training via a code-switch strategy. Experiments are performed on the multilingual image retrieval task across two benchmark datasets, including MSCOCO and Multi30K. M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> P can achieve comparable results for English and new state-of-the-art results for non-English languages.
An automatic Computer-Aided Diagnosis system based on the Multimodal fusion of Breast Cancer (MF-CAD)

The risk of death incurred by breast cancer is rising exponentially, especially among women. The early breast cancer diagnosis before it metastasizes helps medical staff controlling this disease, which decreases the risk of death. This made early breast cancer detection a crucial problem. Different imaging modalities offer complementary information concerning the same lesion helps to increase the performance of thcy fusing several modalities. This paper proposes a computerized features classification of breast cancer lesions through both the Dynamic Contrast-Enhanced Magnetic Resonance Imaging (DCE-MRI) and Digital Mammographic images (MGs). This study aims to investigate a Multimodal Fusion-based Computer-Aided Diagnosis (CAD) system, called MF-CAD, based on multivariate analysis of different modalities, for breast cancer mass detection. In this paper, firstly a new local feature descriptor is proposed in feature extraction, namely, the Gradient Local Information Pattern (GLIP), where we consider the gradient magnitude and orientation as well as the local differences as local binary features for DCE-MRI (or MGs) modality. Secondly, the fusion scheme is conducted using the Canonical Correlation Analysis (CCA) to highlight the intrinsic relation between these modalities. Finally, for comparative purposes, several selected machine learning classifiers (K-Nearest Neighbors, Support Vector Machine, Random forests, Artificial Neural Networks and Radial Basis Function Neural Network (RBFNN)) are used to distinguish between mass and No-mass breast images.Evaluation experiments of the diagnostic performances of our MF-CAD system are conducted over private datasets that contain both MG and DCE-MRI images acquired from 286 patients, which are “Breast DCE-MRI”, “Breast-MG” and “Breast Multimodal” datasets. Experimental results of the proposed MF-CAD system achieved an Area Under the ROC Curve (AUC) value of 99.10% using RBFNN classifier, while for each single modality alone, the best AUC values of 97.20% and 93.50% are obtained respectively for MG and DCE-MRI modalities using random forest classifier. A comparative study with recent existing approaches shows the competitive performances of the proposed approach.
White blood cell type identification using multi-layer convolutional features with an extreme-learning machine

White blood cells (WBCs) are the main component of the immune system that have a major role in protecting the body against different types of infections arising due to viruses, bacteria, fungi, and so on. The WBCs are generally referred to as 5 main subtypes: lymphocytes, monocytes, neutrophils, eosinophils, and basophils. Recognizing and counting each type of WBC is important for diagnosing and treating various disorders, such as infectious diseases, autoimmune disorders, immune deficiencies, leukemia, etc. To this end, a fast and accurate WBC classification model is crucial. This study offers a new model that works with a deep neural network—namely, multi-layer (ML) convolutional features of the AlexNet architecture followed by a feature selection (FS) strategy (MLANet-FS) for WBC-type identification. The proposed model exploits multi-layer convolutional features from different layers of the AlexNet model to provide rich discriminative detail, because different convolutional layers contain different visual characteristics of WBCs, and thereafter, linear fusion of these features occurs automatically. FS strategy is used to select the most distinguishing features from the feature fusion pool. Next, an extreme-learning machine (ELM) is employed to learn a discriminative model of WBC type identification. The proposed MLANet-FS-ELM model was evaluated in extensive experiments on the WBC benchmark dataset. It achieved 99.99% training accuracy and 99.12% testing accuracy, demonstrating that the proposed model outperforms alternative methods in the literature developed for WBC identification.
Forecasting the importance of product attributes using online customer reviews and Google Trends

During the early stage of product design, product manufacturers seek to identify the most relevant product features that will meet the demands and needs of consumers. Conventionally, several surveys have to be undertaken during the time interval between product design and the launch of anew product, to understand any changes on the importance of the product attributes. However, the process is time-consuming and costly. Recently, online customer reviews have been generated on many websites and can be used to analyse the change of the importance of the product attributes. Also, Google Trends has been adopted in previous studies to understand consumers interests in certain products over a period of time and can be considered in analysing the change in product attributes importance. However, no such kinds of studies have been reported. This study aims to present an empirical approach that uses online big data, to identify and predict product design attributes of products that will be relevant to consumers in the future. To achieve this aim, we propose a methodology for forecasting the future importance of product attributes based on online customer reviews and Google Trends. A case study on an electric hairdryer is presented to illustrate the proposed methodology. Validation tests on the proposed fuzzy rough set time series method were conducted. The test results indicate that the proposed method outperforms the fuzzy time series, the fuzzy k medioid clustering time series and the ANFIS method in terms of forecasting accuracy. Our results contribute to the processes of new product development and can potentially assist R&D managers to establish methodologies and processes for product designs capable of generating higher returns.
Detecting Ethereum Ponzi Schemes Based on Improved LightGBM Algorithm

As more investors adopt to enter the field of blockchain investment, the Ponzi scheme, a traditional investment scam, has emerged as a hidden fraud in smart contracts. Although some proposed solutions have paid attention to detecting Ponzi schemes in the blockchain, two problems remain: features for detecting Ponzi schemes are incomplete, and algorithms for detecting Ponzi schemes are not sufficiently efficient. Therefore, we innovatively extract the bytecode feature and combine it with user transaction and opcode frequencies to get more comprehensive features. With these features, we propose a smart contract Ponzi scheme identification method based on the improved LightGBM algorithm. Experiments conducted on the real data set of Ethereum prove that our proposed method has improved accuracy dramatically in terms of the <inline-formula> <tex-math notation="LaTeX">$F$ </tex-math></inline-formula>-score index and the AUC index compared with the state-of-the-art methods. In addition, model training speed is improved significantly. Therefore, our method more accurately identifies Ponzi schemes in smart contracts, thus reducing investment risk.
Rotation Equivariant Siamese Networks for Tracking

Rotation is among the long prevailing, yet still unresolved, hard challenges encountered in visual object tracking. The existing deep learning-based tracking algorithms use regular CNNs that are inherently translation equivariant, but not designed to tackle rotations. In this paper, we first demonstrate that in the presence of rotation instances in videos, the performance of existing trackers is severely affected. To circumvent the adverse effect of rotations, we present rotation-equivariant Siamese networks (RE-SiamNets), built through the use of group-equivariant convolutional layers comprising steerable filters. SiamNets allow estimating the change in orientation of the object in an unsupervised manner, thereby facilitating its use in relative 2D pose estimation as well. We further show that this change in orientation can be used to impose an additional motion constraint in Siamese tracking through imposing restriction on the change in orientation between two consecutive frames. For benchmarking, we present Rotation Tracking Benchmark (RTB), a dataset comprising a set of videos with rotation instances. Through experiments on two popular Siamese architectures, we show that RESiamNets handle the problem of rotation very well and outperform their regular counterparts. Further, RE-SiamNets can accurately estimate the relative change in pose of the target in an unsupervised fashion, namely the in-plane rotation the target has sustained with respect to the reference frame. Code and data can be accessed at https://github.com/dkgupta90/re-siamnet.
Lifelong Person Re-Identification via Adaptive Knowledge Accumulation

Person re-identification (ReID) methods always learn through a stationary domain that is fixed by the choice of a given dataset. In many contexts (e.g., lifelong learning), those methods are ineffective because the domain is continually changing in which case incremental learning over multiple domains is required potentially. In this work we explore a new and challenging ReID task, namely lifelong person re-identification (LReID), which enables to learn continuously across multiple domains and even generalise on new and unseen domains. Following the cognitive processes in the human brain, we design an Adaptive Knowledge Accumulation (AKA) framework that is endowed with two crucial abilities: knowledge representation and knowledge operation. Our method alleviates catastrophic forgetting on seen domains and demonstrates the ability to generalize to unseen domains. Correspondingly, we also provide a new and large-scale benchmark for LReID. Extensive experiments demonstrate our method outperforms other competitors by a margin of 5.8% mAP in generalising evaluation. The codes will be available at https://github.com/TPCD/LifelongReID.
Image Restoration for Under-Display Camera

The new trend of full-screen devices encourages us to position a camera behind a screen. Removing the bezel and centralizing the camera under the screen brings larger display-to-body ratio and enhances eye contact in video chat, but also causes image degradation. In this paper, we focus on a newly-defined Under-Display Camera (UDC), as a novel real-world single image restoration problem. First, we take a 4k Transparent OLED (T-OLED) and a phone Pentile OLED (P-OLED) and analyze their optical systems to understand the degradation. Second, we design a Monitor-Camera Imaging System (MCIS) for easier real pair data acquisition, and a model-based data synthesizing pipeline to generate Point Spread Function (PSF) and UDC data only from display pattern and camera measurements. Finally, we resolve the complicated degradation using deconvolution-based pipeline and learning-based methods. Our model demonstrates a real-time high-quality restoration. The presented methods and results reveal the promising research values and directions of UDC.