Reviewerly/exHarmony huggingface dataset:
Self-Supervised Video-Based Action Recognition With Disturbances

Self-supervised video-based action recognition is a challenging task, which needs to extract the principal information characterizing the action from content-diversified videos over large unlabeled datasets. However, most existing methods choose to exploit the natural spatio-temporal properties of video to obtain effective action representations from a visual perspective, while ignoring the exploration of the semantic that is closer to human cognition. For that, a self-supervised Video-based Action Recognition method with Disturbances called VARD, which extracts the principal information of the action in terms of the visual and semantic, is proposed. Specifically, according to cognitive neuroscience research, the recognition ability of humans is activated by visual and semantic attributes. An intuitive impression is that minor changes of the actor or scene in video do not affect one person's recognition of the action. On the other hand, different humans always make consistent opinions when they recognize the same action video. In other words, for an action video, the necessary information that remains constant despite the disturbances in the visual video or the semantic encoding process is sufficient to represent the action. Therefore, to learn such information, we construct a positive clip/embedding for each action video. Compared to the original video clip/embedding, the positive clip/embedding is disturbed visually/semantically by Video Disturbance and Embedding Disturbance. Our objective is to pull the positive closer to the original clip/embedding in the latent space. In this way, the network is driven to focus on the principal information of the action while the impact of sophisticated details and inconsequential variations is weakened. It is worthwhile to mention that the proposed VARD does not require optical flow, negative samples, and pretext tasks. Extensive experiments conducted on the UCF101 and HMDB51 datasets demonstrate that the proposed VARD effectively improves the strong baseline and outperforms multiple classical and advanced self-supervised action recognition methods.
Tourism Demand Prediction after COVID-19 with Deep Learning Hybrid CNN–LSTM—Case Study of Vietnam and Provinces

The tourism industry experienced a positive increase after COVID-19 and is the largest segment in the foreign exchange contribution in developing countries, especially in Vietnam, where China has begun reopening its borders and lifted the pandemic limitation on foreign travel. This research proposes a hybrid algorithm, combined convolution neural network (CNN) and long short-term memory (LSTM), to accurately predict the tourism demand in Vietnam and some provinces. The number of new COVID-19 cases worldwide and in Vietnam is considered a promising feature in predicting algorithms, which is novel in this research. The Pearson matrix, which evaluates the correlation between selected features and target variables, is computed to select the most appropriate input parameters. The architecture of the hybrid CNN–LSTM is optimized by utilizing hyperparameter fine-tuning, which improves the prediction accuracy and efficiency of the proposed algorithm. Moreover, the proposed CNN–LSTM outperformed other traditional approaches, including the backpropagation neural network (BPNN), CNN, recurrent neural network (RNN), gated recurrent unit (GRU), and LSTM algorithms, by deploying the K-fold cross-validation methodology. The developed algorithm could be utilized as the baseline strategy for resource planning, which could efficiently maximize and deeply utilize the available resource in Vietnam.
Appearance-based gaze estimation with feature fusion of multi-level information elements

Abstract Gaze estimation is a fundamental task in many applications of cognitive sciences, human–computer interaction, and robotics. The purely data-driven appearance-based gaze estimation methods may suffer from a lack of interpretability, which prevents their applicability to pervasive scenarios. In this study, a feature fusion method with multi-level information elements is proposed to improve the comprehensive performance of the appearance-based gaze estimation model. The multi-level feature extraction and expression are carried out from the originally captured images, and a multi-level information element matrix is established. A gaze conduction principle is formulated for reasonably fusing information elements from the established matrix. According to the gaze conduction principle along with the matrix, a multi-level information element fusion (MIEF) model for gaze estimation is proposed. Then, several input modes and network structures of the MIEF model are designed, and a series of grouping experiments are carried out on a small-scale sub-dataset. Furthermore, the optimized input modes and network structures of the MIEF model are selected for training and testing on the whole dataset to verify and compare model performance. Experimental results show that optimizing the feature combination in the input control module and fine-tuning the computational architecture in the feature extraction module can improve the performance of the gaze estimation model, which would enable the reduction of the model by incorporating the critical features and thus improve the performance and accessibility of the method. Compared with the reference baseline, the optimized model based on the proposed feature fusion method of multi-level information elements can achieve efficient training and improve the test accuracy in the verification experiment. The average error is 1.63 cm on phones on the GazeCapture dataset, which achieves comparable accuracy with state-of-the-art methods.
Performance measurement of crypto funds

Crypto funds (CFs) are a growing intermediary in cryptocurrency markets. We evaluate CF performance using metrics based on alphas, value at risk, lower partial moments, and maximum drawdown. The performance of actively managed CFs is heterogeneous: While the average fund in our sample does not outperform the overall cryptocurrency market, there seem to be some few funds with superior skills. Given the non-normal nature of fund returns, the choice of the performance measure affects the rank orders of funds. Compared to the Sharpe ratio, the most commonly applied metric in the asset management practice, performance measures based on alphas and maximum drawdown lead to diverging fund rankings. Depending on their ranking order of preferences, CF investors should consider a bundle of metrics for fund selection and performance measurement.
Differential diagnosis of secondary hypertension based on deep learning

Secondary hypertension is associated with higher risks of target organ damage and cardiovascular and cerebrovascular disease events. Early aetiology identification can eliminate aetiologies and control blood pressure. However, inexperienced doctors often fail to diagnose secondary hypertension, and comprehensively screening for all causes of high blood pressure increases health care costs. To date, deep learning has rarely been involved in the differential diagnosis of secondary hypertension. Relevant machine learning methods cannot combine textual information such as chief complaints with numerical information such as the laboratory examination results in electronic health records (EHRs), and the use of all features increases health care costs. To reduce redundant examinations and accurately identify secondary hypertension, we propose a two-stage framework that follows clinical procedures. The framework carries out an initial diagnosis process in the first stage, on which basis patients are recommended for disease-related examinations, followed by differential diagnoses of different diseases based on the different characteristics observed in the second stage. We convert the numerical examination results into descriptive sentences, thus blending textual and numerical characteristics. Medical guidelines are introduced through label embedding and attention mechanisms to obtain interactive features. Our model was trained and evaluated using a cross-sectional dataset containing 11,961 patients with hypertension from January 2013 to December 2019. The F1 scores of our model were 0.912, 0.921, 0.869 and 0.894 for primary aldosteronism, thyroid disease, nephritis and nephrotic syndrome and chronic kidney disease, respectively, which are four kinds of secondary hypertension with high incidence rates. The experimental results show that our model can powerfully use the textual and numerical data contained in EHRs to provide effective decision support for the differential diagnosis of secondary hypertension.
Bilingual word embedding fusion for robust unsupervised bilingual lexicon induction

Great progress has been made in unsupervised bilingual lexicon induction (UBLI) by aligning the source and target word embeddings independently trained on monolingual corpora. The common assumption of most UBLI models is that the embedding spaces of two languages are approximately isomorphic (i.e., similar in geometric structure). Therefore, the performance is bound by the degree of isomorphism, especially on etymologically and typologically distant languages. Near-zero UBLI results have been reported for them. To address this problem, we propose a method to increase the isomorphism based on bilingual word embedding fusion. In particular, the features from the source embeddings are integrated into the target embeddings, and vice versa. Therefore, the resulting structures of source and target embeddings are similar to each other. The method does not require any form of supervision and can be applied to any language pair. On a benchmark dataset of bilingual lexicon induction, our approach can achieve competitive or superior performance compared to the state-of-the-art methods, with particularly strong results being found on distant languages.
Expert Perspectives on Explainability

Jürgen Cito is interviewing Vijayaraghavan Murali (VM), a software engineer at Meta, and Eddie Aftandilian (EA), a principal researcher at GitHub Next.
A Bibliometric Analysis of the Use of the Metaverse in Education Over Three Decades

Since Facebook announced itself as Meta, the metaverse has been popular with educationalists. However, scanty studies have bibliographically analyzed the use of the metaverse in education. This study complements the missing link in the literature by bibliometrically analyzing the research into the use of the metaverse in education using both VOSviewer and CitNetExplorer. The study identified the top authors, organizations, countries, keywords, documents, sources, the research trends, and challenges of the metaverse used in education, together with the effects of the metaverse on learning environments, interactions, educational outcomes, and learning attitudes. This study paves a solid ground and provides a meaningful reference for future research in the use of the metaverse in education. Implications for future research were also provided.
Mix-Teaching: A Simple, Unified and Effective Semi-Supervised Learning Framework for Monocular 3D Object Detection

Semi-supervised learning (SSL) has promising potential for improving model performance using both labelled and unlabelled data. Since recovering 3D information from 2D images is an ill-posed problem, the current state-of-the-art methods of monocular 3D object detection (Mono3D) have relatively low precision and recall, making semi-supervised learning for Mono3D tasks challenging and understudied. In this work, we propose a unified and effective semi-supervised learning framework called Mix-Teaching that can be applied to most monocular 3D object detectors. Based on the idea of decomposition and recombination, unlabelled samples are firstly decomposed into collections of image patches with high-quality predictions and collections of background images containing no objects. The student model is then trained on the mixed images containing dense instances with high-quality pseudo-labels generated by the recombination operation. In addition, we propose an uncertainty-based filter to distinguish high-quality pseudo-labels from noisy predictions during the decomposition process. As results in KITTI and nuScenes benchmarks, Mix-Teaching consistently improves MonoFlex and GUPNet by significant margins under various labeling ratios. Our method achieves around +6.34% <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$AP_{3D}$ </tex-math></inline-formula> improvement against the GUPNet on the validation set when using only 10% labelled data. Using the full training set and the additional 38K raw images from KITTI, it can further improve the MonoFlex by +4.65% absolute improvement on <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$AP_{3D}$ </tex-math></inline-formula> for car detection, reaching 18.54% <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$AP_{3D}$ </tex-math></inline-formula> , which ranks the 1st place among all monocular based methods on the KITTI test leaderboard.
Low-latency Mini-batch GNN Inference on CPU-FPGA Heterogeneous Platform

Mini-batch inference of Graph Neural Networks (GNNs) is a key problem in many real-world applications. In this paper, we develop a computationally efficient mapping of GNNs onto CPU-FPGA heterogeneous platforms to achieve low-latency mini-batch inference. While the lightweight preprocessing algorithm of GNNs can be efficiently mapped onto the CPU platform, on the FPGA platform, we design a novel GNN hardware accelerator with an adaptive datapath denoted as Adaptive Computation Kernel (ACK) that can execute various computation kernels of GNNs with low-latency: (1) for dense computation kernels expressed as matrix multiplication, ACK works as a systolic array with fully localized connections, (2) for sparse computation kernels, ACK follows the scatter-gather paradigm and works as multiple parallel pipelines to support the irregular connectivity of graphs. The proposed task scheduling hides the CPU-FPGA data communication overhead to reduce the inference latency. We develop a fast design space exploration algorithm to generate a single accelerator for multiple target GNN models. We implement our accelerator on a state-of-the-art CPU-FPGA platform and evaluate the performance using three representative models (GCN, GraphSAGE, GAT). Results show that our CPU-FPGA implementation achieves 21.4−50.8×, 2.9 − 21.6×, 4.7× latency reduction compared with state-of-the-art implementations on CPU-only, CPU-GPU and CPU-FPGA platforms.
Impact of Image Blur on Classification and Augmentation of Deep Convolutional Networks

Blur is a common phenomenon in image acquisition that negatively influences the recognition rate of most classifiers. This paper studies the influence of image blurring of various types and sizes on the recognition rate achieved by a deep convolutional network. We confirm that the blur significantly decreases the performance if the network has been trained on clear images only. When the training set is augmented with blurred samples, the recognition rate becomes sufficiently high even if the blur in query images is of different size than the blur used for training. However, this is mostly not true if query images contain blur of a different type from the one used for training.
Face ShapeNets for 3D Face Recognition

In this paper, we present a deep learning-based method for 3D face recognition.Unlike some previous works, our process does not rely on face representation methods as a proxy step to be accepted by Convolutional Neural Networks (CNNs).Applying 2D CNNs to irregular domains such as 3D meshes is challenging.Therefore, we employed 3D ShapeNets to recognize faces covering the full 3D shape since 3D face datasets are available and 3D data augmentation techniques to enlarge 3D datasets are widespread.The reduced size of 3D datasets is overcome by an appropriate 3D data augmentation to train our model.3D ShapeNets are commonly used to recognize and analyse objects.To the best of the authors' knowledge, this is the first time they are used for face recognition.This research work focuses on the preprocessing step.Whatever the nature of the face image is (either 2D or 3D representation), and whatever the acquisition conditions are, a 3D regular mesh of each input face image is first generated.Furthermore, meshes are converted to voxels to get the occupancy grid across all possible views.Finally, 3D ShapeNets are trained and recognition tests are performed.Indeed, 3D ShapeNets prove the efficiency and efficacy of 3D shape analysis task in 3D face recognition.The experimental results show that 3D face recognition using deep 3D CNNs such as 3D ShapeNets leads to significant improvement over the state-of-the-art performance on LFPW, BU3DFE, and FRAV3D datasets, with competitive recognition rates of 94.25%, 97.9% and 98.31% respectively.
On the Robustness of Dialogue History Representation in Conversational Question Answering: A Comprehensive Study and a New Prompt-based Method

Abstract Most work on modeling the conversation history in Conversational Question Answering (CQA) reports a single main result on a common CQA benchmark. While existing models show impressive results on CQA leaderboards, it remains unclear whether they are robust to shifts in setting (sometimes to more realistic ones), training data size (e.g., from large to small sets) and domain. In this work, we design and conduct the first large-scale robustness study of history modeling approaches for CQA. We find that high benchmark scores do not necessarily translate to strong robustness, and that various methods can perform extremely differently under different settings. Equipped with the insights from our study, we design a novel prompt-based history modeling approach and demonstrate its strong robustness across various settings. Our approach is inspired by existing methods that highlight historic answers in the passage. However, instead of highlighting by modifying the passage token embeddings, we add textual prompts directly in the passage text. Our approach is simple, easy to plug into practically any model, and highly effective, thus we recommend it as a starting point for future model developers. We also hope that our study and insights will raise awareness to the importance of robustness-focused evaluation, in addition to obtaining high leaderboard scores, leading to better CQA systems.1
TAT: Targeted backdoor attacks against visual object tracking

Visual object tracking (VOT) is a fundamental computer vision task that aims to track a target in a sequence of video frames. It has been broadly adopted in safety- and security-critical applications, such as self-driving systems and traffic control systems. However, the VOT models (i.e., the trackers) that rely on third-party training resources face a severe threat of backdoor attacks, which refer to the type of the attacks that poison a portion of training data and mislead the tracker to track a wrong target. A surge of research interest has arisen in backdoor attacks in the domain of image classification, as a measure to expose the potential security risks of the classifiers and inspire new defense techniques. Despite the prosperity of the research in backdoor attacks in image classification, there still lacks investigation in backdoor attacks against VOT, due to their unique challenges: first, the architecture of a VOT model is much more complicated than that of an image classifier; second, VOT targets a sequence of video frames rather than individual images. To bridge the gap, we propose a novel and effective targeted backdoor attack approach TAT specifically against VOT tasks. In particular, TAT includes a basic version TAT-BA that can achieve effective and stealthy backdoor attacks against VOT trackers, and an advanced version TAT-DA that can evade two representative defense techniques. Our large-scale experimental evaluation demonstrates the effectiveness and the stealthiness of TAT. Moreover, we also demonstrate the performances of TAT-BA under real-world settings and the abilities of TAT-DA to counter defense techniques. The code will be available at https://github.com/MisakaZipi/TAT.
An optimization based framework for region wise optimal clusters in MR images using hybrid objective

Swarm Intelligence based methods are amongst the highly efficient approaches for optimization in image clustering. Optimal clustering has been studied in many real-world applications, such as medical and aerial image segmentation. Region wise clustering is a class of challenges in image region segmentation. Uncertain convergence and high computational load are critical issues in the region-wise image clustering due to local optimum and the NP-hard cluster computation. Meta-heuristics approaches are efficient to achieve global optimum by including better search space exploration techniques. This paper develops a framework for cluster optimization by selecting the seeds in pathological medical resonance (MR) images using a variant of firefly optimization. The heuristics based method uses Gaussian random walk for convergence that occasionally results in local optima; therefore, we have investigated the firefly method with more search space exploration techniques and improved region-wise objective. Our framework applies the levy flights for exploration and compared with other search spaces like Cauchy, and Gaussian random walk. The intra-cluster and inter-cluster-based hybrid objective is converged swiftly. The framework has been compared with two of its variants and three other meta-heuristic-based methods, namely simulated annealing, PSO, and Cuckoo search. The MSE, PSNR, structural similarity(SSIM), and feature similarity(FSIM) based evaluation indices are measured for normal and abnormal MR images and listed in table-5, 6. Reported indices values for our frame work are better than existing methods. Figure-9, 10, 11 compares the stochastic search spaces among Levy flights, Cauchy random walk, Gaussian random walk and observed Levy flights as better search space. In section-3.5, The convergence for proposed framework is shown for multi-objective function against the single objective in two normal and abnormal images. Single objective converged from 188 to 119 and multi-objective converged from 196 to 117 for first image. For second image, the single objective converged from 168 to 94 and multi-objective converged from 183 to 93. Finally, we have illustrated the convergence criteria and computation complexity on publicly available MR data sets.