Reviewerly/exHarmony huggingface dataset:
DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Prediction

Convolution neural networks (CNNs) and Transformers have their own advantages and both have been widely used for dense prediction in multi-task learning (MTL). Most of the current studies on MTL solely rely on CNN or Transformer. In this work, we present a novel MTL model by combining both merits of deformable CNN and query-based Transformer for multi-task learning of dense prediction. Our method, named DeMT, is based on a simple and effective encoder-decoder architecture (i.e., deformable mixer encoder and task-aware transformer decoder). First, the deformable mixer encoder contains two types of operators: the channel-aware mixing operator leveraged to allow communication among different channels (i.e., efficient channel location mixing), and the spatial-aware deformable operator with deformable convolution applied to efficiently sample more informative spatial locations (i.e., deformed features). Second, the task-aware transformer decoder consists of the task interaction block and task query block. The former is applied to capture task interaction features via self-attention. The latter leverages the deformed features and task-interacted features to generate the corresponding task-specific feature through a query-based Transformer for corresponding task predictions. Extensive experiments on two dense image prediction datasets, NYUD-v2 and PASCAL-Context, demonstrate that our model uses fewer GFLOPs and significantly outperforms current Transformer- and CNN-based competitive models on a variety of metrics. The code is available at https://github.com/yangyangxu0/DeMT.
Contrastive Masked Autoencoders for Self-Supervised Video Hashing

Self-Supervised Video Hashing (SSVH) models learn to generate short binary representations for videos without ground-truth supervision, facilitating large-scale video retrieval efficiency and attracting increasing research attention. The success of SSVH lies in the understanding of video content and the ability to capture the semantic relation among unlabeled videos. Typically, state-of-the-art SSVH methods consider these two points in a two-stage training pipeline, where they firstly train an auxiliary network by instance-wise mask-and-predict tasks and secondly train a hashing model to preserve the pseudo-neighborhood structure transferred from the auxiliary network. This consecutive training strategy is inflexible and also unnecessary. In this paper, we propose a simple yet effective one-stage SSVH method called ConMH, which incorporates video semantic information and video similarity relationship understanding in a single stage. To capture video semantic information for better hashing learning, we adopt an encoder-decoder structure to reconstruct the video from its temporal-masked frames. Particularly, we find that a higher masking ratio helps video understanding. Besides, we fully exploit the similarity relationship between videos by maximizing agreement between two augmented views of a video, which contributes to more discriminative and robust hash codes. Extensive experiments on three large-scale video datasets (i.e., FCVID, ActivityNet and YFCC) indicate that ConMH achieves state-of-the-art results. Code is available at https://github.com/huangmozhi9527/ConMH.
CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language Models

Pre-trained programming language (PL) models (such as CodeT5, CodeBERT, GraphCodeBERT, etc.,) have the potential to automate software engineering tasks involving code understanding and code generation. However, these models operate in the natural channel of code, i.e., primarily concerned with the human understanding of code. They are not robust to changes in the input and thus, are potentially susceptible to adversarial attacks in the natural channel. We propose, Code Attack, a simple yet effective black-box attack model that uses code structure to generate effective, efficient, and imperceptible adversarial code samples and demonstrates the vulnerabilities of the state-of-the-art PL models to code-specific adversarial attacks. We evaluate the transferability of CodeAttack on several code-code (translation and repair) and code-NL (summarization) tasks across different programming languages. Code Attack outperforms state-of-the-art adversarial NLP attack models to achieve the best overall drop in performance while being more efficient, imperceptible, consistent, and fluent. The code can be found at https://github.com/reddy-lab-code-research/CodeAttack.
Deep Learning-Based Micro Facial Expression Recognition Using an Adaptive Tiefes FCNN Model

The scientific community and media have increasingly recognized the significance of microexpressions as indicators for detecting deception, as they reveal genuine emotions that individuals attempt to conceal.To capitalize on these subtle cues of deceit, researchers have developed applications capable of automatically detecting and recognizing microexpressions, which are typically imperceptible to the human eye.Facial expressions serve as fundamental ground truth determinants in multimedia applications.Earlier models, such as GA, RFO, X-Boosting, and Gradient Boosting, demonstrate greater efficiency in terms of time and accuracy.However, not all applications are capable of detecting micro facial expressions.In this study, a deep learning-based Tiefes FCNN model is designed specifically for micro facial expression recognition.Implemented using Python software, the proposed model consists of two stages: first, pre-processing is performed using image segmentation, followed by the application of a deep learning model employing Tiefes FCNN technology in the second stage.The experimental results exhibit significant performance improvements, including an accuracy of 99.02%, precision of 98.82%, F1-score of 97.8%, PSNR of 56.31, and CC of 96.31.
Siamese-Discriminant Deep Reinforcement Learning for Solving Jigsaw Puzzles with Large Eroded Gaps

Jigsaw puzzle solving has recently become an emerging research area. The developed techniques have been widely used in applications beyond puzzle solving. This paper focuses on solving Jigsaw Puzzles with Large Eroded Gaps (JPwLEG). We formulate the puzzle reassembly as a combinatorial optimization problem and propose a Siamese-Discriminant Deep Reinforcement Learning (SD2RL) to solve it. A Deep Q-network (DQN) is designed to visually understand the puzzles, which consists of two sets of Siamese Discriminant Networks, one set to perceive the pairwise relations between vertical neighbors and another set for horizontal neighbors. The proposed DQN considers not only the evidence from the incumbent fragment but also the support from its four neighbors. The DQN is trained using replay experience with carefully designed rewards to guide the search for a sequence of fragment swaps to reach the correct puzzle solution. Two JPwLEG datasets are constructed to evaluate the proposed method, and the experimental results show that the proposed SD2RL significantly outperforms state-of-the-art methods.
Spatial-Spectral Transformer for Hyperspectral Image Denoising

Hyperspectral image (HSI) denoising is a crucial preprocessing procedure for the subsequent HSI applications. Unfortunately, though witnessing the development of deep learning in HSI denoising area, existing convolution-based methods face the trade-off between computational efficiency and capability to model non-local characteristics of HSI. In this paper, we propose a Spatial-Spectral Transformer (SST) to alleviate this problem. To fully explore intrinsic similarity characteristics in both spatial dimension and spectral dimension, we conduct non-local spatial self-attention and global spectral self-attention with Transformer architecture. The window-based spatial self-attention focuses on the spatial similarity beyond the neighboring region. While, the spectral self-attention exploits the long-range dependencies between highly correlative bands. Experimental results show that our proposed method outperforms the state-of-the-art HSI denoising methods in quantitative quality and visual results. The code is released at https://github.com/MyuLi/SST.
TopicFM: Robust and Interpretable Topic-Assisted Feature Matching

This study addresses an image-matching problem in challenging cases, such as large scene variations or textureless scenes. To gain robustness to such situations, most previous studies have attempted to encode the global contexts of a scene via graph neural networks or transformers. However, these contexts do not explicitly represent high-level contextual information, such as structural shapes or semantic instances; therefore, the encoded features are still not sufficiently discriminative in challenging scenes. We propose a novel image-matching method that applies a topic-modeling strategy to encode high-level contexts in images. The proposed method trains latent semantic instances called topics. It explicitly models an image as a multinomial distribution of topics, and then performs probabilistic feature matching. This approach improves the robustness of matching by focusing on the same semantic areas between the images. In addition, the inferred topics provide interpretability for matching the results, making our method explainable. Extensive experiments on outdoor and indoor datasets show that our method outperforms other state-of-the-art methods, particularly in challenging cases. The code is available at https://github.com/TruongKhang/TopicFM.
Revisiting Classifier: Transferring Vision-Language Models for Video Recognition

Transferring knowledge from task-agnostic pre-trained deep models for downstream tasks is an important topic in computer vision research. Along with the growth of computational capacity, we now have open-source vision-language pre-trained models in large scales of the model architecture and amount of data. In this study, we focus on transferring knowledge for video classification tasks. Conventional methods randomly initialize the linear classifier head for vision classification, but they leave the usage of the text encoder for downstream visual recognition tasks undiscovered. In this paper, we revise the role of the linear classifier and replace the classifier with the different knowledge from pre-trained model. We utilize the well-pretrained language model to generate good semantic target for efficient transferring learning. The empirical study shows that our method improves both the performance and the training speed of video classification, with a negligible change in the model. Our simple yet effective tuning paradigm achieves state-of-the-art performance and efficient training on various video recognition scenarios, i.e., zero-shot, few-shot, general recognition. In particular, our paradigm achieves the state-of-the-art accuracy of 87.8% on Kinetics-400, and also surpasses previous methods by 20~50% absolute top-1 accuracy under zero-shot, few-shot settings on five video datasets. Code and models are available at https://github.com/whwu95/Text4Vis.
StyleTalk: One-Shot Talking Head Generation with Controllable Speaking Styles

Different people speak with diverse personalized speaking styles. Although existing one-shot talking head methods have made significant progress in lip sync, natural facial expressions, and stable head motions, they still cannot generate diverse speaking styles in the final talking head videos. To tackle this problem, we propose a one-shot style-controllable talking face generation framework. In a nutshell, we aim to attain a speaking style from an arbitrary reference speaking video and then drive the one-shot portrait to speak with the reference speaking style and another piece of audio. Specifically, we first develop a style encoder to extract dynamic facial motion patterns of a style reference video and then encode them into a style code. Afterward, we introduce a style-controllable decoder to synthesize stylized facial animations from the speech content and style code. In order to integrate the reference speaking style into generated videos, we design a style-aware adaptive transformer, which enables the encoded style code to adjust the weights of the feed-forward layers accordingly. Thanks to the style-aware adaptation mechanism, the reference speaking style can be better embedded into synthesized videos during decoding. Extensive experiments demonstrate that our method is capable of generating talking head videos with diverse speaking styles from only one portrait image and an audio clip while achieving authentic visual effects. Project Page: https://github.com/FuxiVirtualHuman/styletalk.
Classification of asymmetry in mammography via the DenseNet convolutional neural network

<h2>Abstract</h2><h3>Purpose</h3> To investigate the effectiveness of a deep learning system based on the DenseNet convolutional neural network in diagnosing benign and malignant asymmetric lesions in mammography. <h3>Methods</h3> Clinical and image data from 460 women aged 23–82 years (47.57 ± 8.73 years) with asymmetric lesions who underwent mammography at Shenzhen People's Hospital, Shenzhen Luohu District People's Hospital, and Shenzhen Hospital of Peking University from December 2019 to December 2020 were retrospectively analyzed. Two senior radiologists, two junior radiologists, and the DL system read the mammographic images of 460 patients, respectively, and finally recorded the BI-RADS classification of asymmetric lesions. We then used the area under the curve (AUC) of the receiver operating characteristic (ROC) to evaluate the diagnostic efficacy and the difference between AUCs by the Delong method. <h3>Results</h3> Specificity (0.909 vs. 0.835, 0.790, <mml:math><mml:msup><mml:mrow><mml:mi>χ</mml:mi></mml:mrow><mml:mn>2</mml:mn></mml:msup></mml:math>=8.21 and 17.22, <i>p</i>＜0.05) and precision (0.872 vs. 0.763, 0.726, <mml:math><mml:msup><mml:mrow><mml:mi>χ</mml:mi></mml:mrow><mml:mn>2</mml:mn></mml:msup></mml:math>=9.23 and 5.22, <i>p</i>＜0.05) of the DL system in the diagnosis of benign and malignant asymmetric lesions were higher than those of junior radiologist A and B, and there was a statistically significant difference between AUCs (0.778 vs. 0.579, 0.564, Z = 4.033 and 4.460, <i>p</i>＜0.05). Furthermore, the AUC (0.778 vs. 0.904, 0.862, Z = 3.191, and 2.167, <i>p</i>＜0.05) of benign and malignant asymmetric lesions diagnosed by the DL system was lower than that of senior radiologist A and senior radiologist B. <h3>Conclusions</h3> The DL system based on the DenseNet convolution neural network has high diagnostic efficiency, which can help junior radiologists evaluate benign and malignant asymmetric lesions more accurately. It can also improve diagnostic accuracy and reduce missed diagnoses caused by inexperienced junior radiologists.
FMDNet: An Efficient System for Face Mask Detection Based on Lightweight Model during COVID-19 Pandemic in Public Areas

A new artificial intelligence-based approach is proposed by developing a deep learning (DL) model for identifying the people who violate the face mask protocol in public places. To achieve this goal, a private dataset was created, including different face images with and without masks. The proposed model was trained to detect face masks from real-time surveillance videos. The proposed face mask detection (FMDNet) model achieved a promising detection of 99.0% in terms of accuracy for identifying violations (no face mask) in public places. The model presented a better detection capability compared to other recent DL models such as FSA-Net, MobileNet V2, and ResNet by 24.03%, 5.0%, and 24.10%, respectively. Meanwhile, the model is lightweight and had a confidence score of 99.0% in a resource-constrained environment. The model can perform the detection task in real-time environments at 41.72 frames per second (FPS). Thus, the developed model can be applicable and useful for governments to maintain the rules of the SOP protocol.
Ethics of Artificial Intelligence in Education: Student Privacy and Data Protection

Rapid advances in artificial intelligence (AI) technology are profoundly altering human societies and lifestyles. Individuals face a variety of information security threats while enjoying the conveniences and customized services made possible by AI. The widespread use of AI in education has prompted widespread public concern regarding AI ethics in this field. The protection of pupil data privacy is an urgent matter that must be addressed. On the basis of a review of extant interpretations of AI ethics and student data privacy, this article examines the ethical risks posed by AI technology to student personal information and provides recommendations for addressing concerns regarding student data security.
Machine Learning-Based Crowd behavior Analysis and Forecasting

In many places today, the world's overcrowding causes crowded conditions. Analysis of crowd activity is a developing field of study. It is common knowledge that mob activity can forecast what might happen during an event. Crowd management could be very effective if situations like riots, mass lynchings, traffic jams, accidents, stampedes, etc. could be predicted beforehand. In this paper, we propose a new multicolumn convolutional neural network (MCNN) based technique for predicting mob behavior. The features of the incoming image are first analyzed and extracted. The approximated number of the gathering is then established, and image cropping is completed. For each area of the image, low level characteristics are retrieved. The objects in the picture are then created as density images. Using our method, the gathered characteristics and their object density maps are then linearly mapped. At last, we forecast and quantify the population using the MCNN algorithm. For the ShanghaiTech dataset, we have evaluated our method using actual data.
AraBERTopic: A Neural Topic Modeling Approach for News Extraction from Arabic Facebook Pages using Pre-trained BERT Transformer Model

Topic modeling algorithms can better understand data by extracting meaningful words from text collection, but the results are often inconsistent, and consequently difficult to interpret.Enrich the model with more contextual knowledge can improve coherence.Recently, neural topic models have emerged, and the development of neural models, in general, was pushed by BERT-based representations.We propose in this paper, a model named AraBERTopic to extract news from Facebook pages.Our model combines the Pre-training BERT transformer model for the Arabic language (AraBERT) and neural topic model ProdLDA.Thus, compared with the standard LDA, pre-trained BERT sentence embeddings produce more meaningful and coherent topics using different embedding models.Results show that our AraBERTopic model gives 0.579 in topic coherence.
The developmental trajectory of object recognition robustness: Children are like small adults but unlike big deep neural networks

In laboratory object recognition tasks based on undistorted photographs, both adult humans and deep neural networks (DNNs) perform close to ceiling. Unlike adults’, whose object recognition performance is robust against a wide range of image distortions, DNNs trained on standard ImageNet (1.3M images) perform poorly on distorted images. However, the last 2 years have seen impressive gains in DNN distortion robustness, predominantly achieved through ever-increasing large-scale datasets—orders of magnitude larger than ImageNet. Although this simple brute-force approach is very effective in achieving human-level robustness in DNNs, it raises the question of whether human robustness, too, is simply due to extensive experience with (distorted) visual input during childhood and beyond. Here we investigate this question by comparing the core object recognition performance of 146 children (aged 4–15 years) against adults and against DNNs. We find, first, that already 4- to 6-year-olds show remarkable robustness to image distortions and outperform DNNs trained on ImageNet. Second, we estimated the number of images children had been exposed to during their lifetime. Compared with various DNNs, children’s high robustness requires relatively little data. Third, when recognizing objects, children—like adults but unlike DNNs—rely heavily on shape but not on texture cues. Together our results suggest that the remarkable robustness to distortions emerges early in the developmental trajectory of human object recognition and is unlikely the result of a mere accumulation of experience with distorted visual input. Even though current DNNs match human performance regarding robustness, they seem to rely on different and more data-hungry strategies to do so.
Remote Sensing for Soil Organic Carbon Mapping and Monitoring

Remote sensing soil properties in a coherent manner is now feasible from regional to global scales [...]
AI Security for Geoscience and Remote Sensing: Challenges and future trends

Recent advances in artificial intelligence (AI) have significantly intensified research in the geoscience and remote sensing (RS) field. AI algorithms, especially deep learning-based ones, have been developed and applied widely to RS data analysis. The successful application of AI covers almost all aspects of Earth-observation (EO) missions, from low-level vision tasks like superresolution, denoising, and inpainting, to high-level vision tasks like scene classification, object detection, and semantic segmentation. Although AI techniques enable researchers to observe and understand the earth more accurately, the vulnerability and uncertainty of AI models deserve further attention, considering that many geoscience and RS tasks are highly safety critical. This article reviews the current development of AI security in the geoscience and RS field, covering the following five important aspects: adversarial attack, backdoor attack, federated learning (FL), uncertainty, and explainability. Moreover, the potential opportunities and trends are discussed to provide insights for future research. To the best of the authors’ knowledge, this article is the first attempt to provide a systematic review of AI security-related research in the geoscience and RS community. Available code and datasets are also listed in the article to move this vibrant field of research forward.
Pedagogical strategies based on inverted classroom - Integration of ICT in naval technologies at the Escuela Naval de Suboficiales A.R.C. Barranquilla

This study explored the interrelation between technology and education, focusing on the implementation of the flipped classroom model in the framework of technology-supported subjects. The convergence between these two fields was analyzed, not only for its contemporary relevance but also for its constant potential in the future. The methodology used was applied research, which facilitated an effective integration of Information and Communication Technologies (ICT) in the conception and development of academic activities. The results obtained showed acceptance and commitment on the part of the students when content was provided to them through multiple formats and platforms. It was concluded that the versatility and accessibility of ICT, applied appropriately, could enhance the students' learning experience. This study, therefore, provided a valuable contribution to understanding how emerging technologies could be effectively utilized in the educational context.