Reviewerly/exHarmony huggingface dataset:
An Effective Video Transformer With Synchronized Spatiotemporal and Spatial Self-Attention for Action Recognition

Authors:
raw_author_name: Saghir Alfasly
raw_affiliation_string: Shenzhen Key Laboratory of Advanced Machine Learning and Applications, College of Mathematics and Statistics, Shenzhen University, Shenzhen, China
raw_author_name: Charles K Chui
raw_affiliation_string: Department of Statistics, Stanford University, Stanford, CA, USA
raw_author_name: Qingtang Jiang
raw_affiliation_string: Department of Mathematics and Statistics, University of Missouri&#x2013;St. Louis, St. Louis, MO, USA
raw_author_name: Jian Lu
raw_affiliation_string: Shenzhen Key Laboratory of Advanced Machine Learning and Applications, College of Mathematics and Statistics, Shenzhen University, Shenzhen, China
raw_author_name: Chen Xu
raw_affiliation_string: Shenzhen Key Laboratory of Advanced Machine Learning and Applications, College of Mathematics and Statistics, Shenzhen University, Shenzhen, China

Abstract: Convolutional neural networks (CNNs) have come to dominate vision-based deep neural network structures in both image and video models over the past decade. However, convolution-free vision Transformers (ViTs) have recently outperformed CNN-based models in image recognition. Despite this progress, building and designing video Transformers have not yet obtained the same attention in research as image-based Transformers. While there have been attempts to build video Transformers by adapting image-based Transformers for video understanding, these Transformers still lack efficiency due to the large gap between CNN-based models and Transformers regarding the number of parameters and the training settings. In this work, we propose three techniques to improve video understanding with video Transformers. First, to derive better spatiotemporal feature representation, we propose a new spatiotemporal attention scheme, termed synchronized spatiotemporal and spatial attention (SSTSA), which derives the spatiotemporal features with temporal and spatial multiheaded self-attention (MSA) modules. It also preserves the best spatial attention by another spatial self-attention module in parallel, thereby resulting in an effective Transformer encoder. Second, a motion spotlighting module is proposed to embed the short-term motion of the consecutive input frames to the regular RGB input, which is then processed with a single-stream video Transformer. Third, a simple intraclass frame interlacing method of the input clips is proposed that serves as an effective video augmentation method. Finally, our proposed techniques have been evaluated and validated with a set of extensive experiments in this study. Our video Transformer outperforms its previous counterparts on two well-known datasets, Kinetics400 and Something-Something-v2.
Toward Zero-Trust Security for the Metaverse

Authors:
raw_author_name: Ruizhi Cheng
raw_affiliation_string: George Mason University,USA
raw_author_name: Songqing Chen
raw_affiliation_string: George Mason University,USA
raw_author_name: Bo Han
raw_affiliation_string: George Mason University,USA

Abstract: By focusing on immersive interaction among users, the burgeoning Metaverse can be viewed as a natural extension of existing social media. Similar to traditional online social networks, there are numerous security and privacy issues in the Metaverse (e.g., attacks on user authentication and impersonation). In this paper, we develop a holistic research agenda for zero-trust user authentication in social virtual reality (VR), an early prototype of the Metaverse. Our proposed research includes four concrete steps: investigating biometrics-based authentication that is suitable for continuously authenticating VR users, leveraging federated learning (FL) for protecting user privacy in biometric data, improving the accuracy of continuous VR authentication with multimodal data, and boosting the usability of zero-trust security with adaptive VR authentication. Our preliminary study demonstrates that conventional FL algorithms are not well suited for biometrics-based authentication of VR users, leading to an accuracy of less than 10%. We discuss the root cause of this problem, the associated open challenges, and several future directions for realizing our research vision.
DRBM-ClustNet: A Deep Restricted Boltzmann–Kohonen Architecture for Data Clustering

Authors:
raw_author_name: Senthilnath, J.
raw_affiliation_string: Agency for Science, Technology and Research (A&#x002A;STAR), Institute for Infocomm Research, Fusionopolis, Singapore
raw_author_name: Kulkarni, Sushant
raw_affiliation_string: IIT Madras, Chennai, India
raw_author_name: Benediktsson, Jón Atli
raw_affiliation_string: Faculty of Electrical and Computer Engineering, University of Iceland, Reykjavik, Iceland

Abstract: A Bayesian deep restricted Boltzmann-Kohonen architecture for data clustering termed deep restricted Boltzmann machine (DRBM)-ClustNet is proposed. This core-clustering engine consists of a DRBM for processing unlabeled data by creating new features that are uncorrelated and have large variance with each other. Next, the number of clusters is predicted using the Bayesian information criterion (BIC), followed by a Kohonen network (KN)-based clustering layer. The processing of unlabeled data is done in three stages for efficient clustering of the nonlinearly separable datasets. In the first stage, DRBM performs nonlinear feature extraction by capturing the highly complex data representation by projecting the feature vectors of d dimensions into n dimensions. Most clustering algorithms require the number of clusters to be decided a priori; hence, here, to automate the number of clusters in the second stage, we use BIC. In the third stage, the number of clusters derived from BIC forms the input for the KN, which performs clustering of the feature-extracted data obtained from the DRBM. This method overcomes the general disadvantages of clustering algorithms, such as the prior specification of the number of clusters, convergence to local optima, and poor clustering accuracy on nonlinear datasets. In this research, we use two synthetic datasets, 15 benchmark datasets from the UCI Machine Learning repository, and four image datasets to analyze the DRBM-ClustNet. The proposed framework is evaluated based on clustering accuracy and ranked against other state-of-the-art clustering methods. The obtained results demonstrate that the DRBM-ClustNet outperforms state-of-the-art clustering algorithms.
Guidance Through Surrogate: Toward a Generic Diagnostic Attack

Authors:
raw_author_name: Muzammal Naseer
raw_affiliation_string: Computer Vision Department, Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
raw_author_name: Salman Khan
raw_affiliation_string: Computer Vision Department, Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
raw_author_name: Fatih Porikli
raw_affiliation_string: Qualcomm, San Diego, CA, USA
raw_author_name: Fahad Shahbaz Khan
raw_affiliation_string: Computer Vision Department, Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates

Abstract: Adversarial training (AT) is an effective approach to making deep neural networks robust against adversarial attacks. Recently, different AT defenses are proposed that not only maintain a high clean accuracy but also show significant robustness against popular and well-studied adversarial attacks, such as projected gradient descent (PGD). High adversarial robustness can also arise if an attack fails to find adversarial gradient directions, a phenomenon known as "gradient masking." In this work, we analyze the effect of label smoothing on AT as one of the potential causes of gradient masking. We then develop a guided mechanism to avoid local minima during attack optimization, leading to a novel attack dubbed guided projected gradient attack (G-PGA). Our attack approach is based on a "match and deceive" loss that finds optimal adversarial directions through guidance from a surrogate model. Our modified attack does not require random restarts a large number of attack iterations or a search for optimal step size. Furthermore, our proposed G-PGA is generic, thus it can be combined with an ensemble attack strategy as we demonstrate in the case of auto-attack, leading to efficiency and convergence speed improvements. More than an effective attack, G-PGA can be used as a diagnostic tool to reveal elusive robustness due to gradient masking in adversarial defenses.
Efficient <i>N:M</i> Sparse DNN Training Using Algorithm, Architecture, and Dataflow Co-Design

Authors:
raw_author_name: Chao Fang
raw_affiliation_string: School of Electronic Science and Engineering, Nanjing University, Nanjing, China
raw_author_name: Aojun Zhou
raw_affiliation_string: CUHK-Sensetime Joint Lab, The Chinese University of Hong Kong, Hong Kong, China
raw_author_name: Zhongfeng Wang
raw_affiliation_string: School of Electronic Science and Engineering, Nanjing University, Nanjing, China

Abstract: Sparse training is one of the promising techniques to reduce the computational cost of DNNs while retaining high accuracy. In particular, N:M fine-grained structured sparsity, where only N out of consecutive M elements can be nonzero, has attracted attention due to its hardware-friendly pattern and capability of achieving a high sparse ratio. However, the potential to accelerate N:M sparse DNN training has not been fully exploited, and there is a lack of efficient hardware supporting N:M sparse training. To tackle these challenges, this paper presents a computation-efficient training scheme for N:M sparse DNNs using algorithm, architecture, and dataflow co-design. At the algorithm level, a bidirectional weight pruning method, dubbed BDWP, is proposed to leverage the N:M sparsity of weights during both forward and backward passes of DNN training, which can significantly reduce the computational cost while maintaining model accuracy. At the architecture level, a sparse accelerator for DNN training, namely SAT, is developed to neatly support both the regular dense operations and the computation-efficient N:M sparse operations. At the dataflow level, multiple optimization methods ranging from interleave mapping, pre-generation of N:M sparse weights, and offline scheduling, are proposed to boost the computational efficiency of SAT. Finally, the effectiveness of our training scheme is evaluated on a Xilinx VCU1525 FPGA card using various DNN models and datasets. Experimental results show the SAT accelerator with the BDWP sparse training method under 2:8 sparse ratio achieves an average speedup of 1.75x over that with the dense training, accompanied by a negligible accuracy loss of 0.56% on average. Furthermore, our proposed training scheme significantly improves the training throughput by 2.97~25.22x and the energy efficiency by 1.36~3.58x over prior FPGA-based accelerators.
Latent variable modeling with applications to education assessment and NSF-REU projects for engineering students

Authors:
raw_author_name: Vazgen Shekoyan
raw_author_name: sunil Dehipawala
raw_affiliation_string: Queensborough Community College
raw_author_name: Rex Taibu

Abstract: Abstract The latent variable modeling technique in education assessment is illustrated using problem solving examples in first semester physics for engineering and technology students. The regression modeling of data in the Newton second law laboratory class has equivalence to an analysis with the internal force serving as the latent variable. A path diagram would contain the internal force as a latent variable which manifest its influence on the observed datasets, namely, acceleration and force. Similarly, the pre-score and post-score data collected for education assessment can be studied with latent variable modeling to broaden the scope beyond regression which requires an independent variable. The numerical and flexibility advantages of latent variable modeling versus regression modeling in education assessment are presented. Free LISREL student version working details and free confirmatory factor analysis Microsoft Excel software from university faculty are presented. The adaptation of Queensborough Community College Physics faculty education research into Queensborough NSF-REU projects for engineering students is discussed.
Interpolation-Based Contrastive Learning for Few-Label Semi-Supervised Learning

Authors:
raw_author_name: Yang, Xihong
raw_affiliation_string: School of Computer, National University of Defense Technology, Changsha, China
raw_author_name: Hu, Xiaochang
raw_affiliation_string: College of Intelligence Science and Technology, National University of Defense Technology, Changsha, China
raw_author_name: Zhou, Sihang
raw_affiliation_string: College of Intelligence Science and Technology, National University of Defense Technology, Changsha, China
raw_author_name: Liu, Xinwang
raw_affiliation_string: School of Computer, National University of Defense Technology, Changsha, China
raw_author_name: Zhu, En
raw_affiliation_string: School of Computer, National University of Defense Technology, Changsha, China

Abstract: Semi-supervised learning (SSL) has long been proved to be an effective technique to construct powerful models with limited labels. In the existing literature, consistency regularization-based methods, which force the perturbed samples to have similar predictions with the original ones have attracted much attention for their promising accuracy. However, we observe that the performance of such methods decreases drastically when the labels get extremely limited, e.g., 2 or 3 labels for each category. Our empirical study finds that the main problem lies with the drift of semantic information in the procedure of data augmentation. The problem can be alleviated when enough supervision is provided. However, when little guidance is available, the incorrect regularization would mislead the network and undermine the performance of the algorithm. To tackle the problem, we: 1) propose an interpolation-based method to construct more reliable positive sample pairs and 2) design a novel contrastive loss to guide the embedding of the learned network to change linearly between samples so as to improve the discriminative capability of the network by enlarging the margin decision boundaries. Since no destructive regularization is introduced, the performance of our proposed algorithm is largely improved. Specifically, the proposed algorithm outperforms the second best algorithm (Comatch) with 5.3% by achieving 88.73% classification accuracy when only two labels are available for each class on the CIFAR-10 dataset. Moreover, we further prove the generality of the proposed method by improving the performance of the existing state-of-the-art algorithms considerably with our proposed strategy. The corresponding code is available at https://github.com/xihongyang1999/ICL_SSL.
Maneuvering Target State Estimation in Range-Squared Coordinate

Authors:
raw_author_name: Keyi Li
raw_affiliation_string: School of Electronics and Information Engineering, Harbin Institute of Technology, Harbin, China
raw_author_name: Gongjian Zhou
raw_affiliation_string: School of Electronics and Information Engineering, Harbin Institute of Technology, Harbin, China

Abstract: In tracking applications without angle measurements, multiple sensors are usually applied. Accurate range-Doppler (R-D) estimates with a single sensor play an important role in clutter eliminating and date fusion in multi-sensor target tracking. In this paper, maneuvering models are first established in the range-squared (RS) coordinate, then the R-D estimation of a maneuvering target is investigated. According to the Cartesian nearly constant acceleration and constant turn models, pseudostate vectors are defined in the RS coordinate and linear time-evolution equations of pseudostate are derived to describe the corresponding maneuvers. The motion models are incorporated into the converted measurement Kalman filter to extract R-D estimates from converted measurements. Corresponding to the models, the filter initialization approach, which utilizes the prior information on motion model and kinematic parameters of target, is developed. Based on the model set in the RS coordinate and the interacting multiple model (IMM) method, a recursive estimator is developed to produce pseudostate estimates of a maneuvering target. Final R-D estimates are provided through the conversion adopting the scaled unscented transformation outside the estimation recursions to avoid the propagation of nonlinearity approximation errors. Numerical experiments in different scenarios demonstrate that the maneuvering models accurately describe the time-evolution of pseudostate and the IMM estimator achieves superior estimation accuracy and consistency thanks to the exploitation of linear motion models.
Bayesian multi-level modelling for predicting single and double feature visual search

Authors:
raw_author_name: Anna Nowakowska
raw_affiliation_string: School of Psychology and Vision Sciences, University of Leicester, Leicester, LE1 7RH, UK; School of Psychology, University of Aberdeen, Aberdeen, AB24 3FX, UK
raw_author_name: Alasdair D.F. Clarke
raw_affiliation_string: Department of Psychology, University of Essex, Colchester, CO4 3SQ, UK

Abstract: Performance in visual search tasks is frequently summarised by "search slopes" - the additional cost in reaction time for each additional distractor. While search tasks with a shallow search slopes are termed efficient (pop-out, parallel, feature), there is no clear dichotomy between efficient and inefficient (serial, conjunction) search. Indeed, a range of search slopes are observed in empirical data. The Target Contrast Signal (TCS) Theory is a rare example of quantitative model that attempts to predict search slopes for efficient visual search. One study using the TCS framework has shown that the search slope in a double-feature search (where the target differs in both colour and shape from the distractors) can be estimated from the slopes of the associated single-feature searches. This estimation is done using a contrast combination model, and a collinear contrast integration model was shown to outperform other options. In our work, we extend TCS to a Bayesian multi-level framework. We investigate modelling using normal and shifted-lognormal distributions, and show that the latter allows for a better fit to previously published data. We run a new fully within-subjects experiment to attempt to replicate the key original findings, and show that overall, TCS does a good job of predicting the data. However, we do not replicate the finding that the collinear combination model outperforms the other contrast combination models, instead finding that it may be difficult to conclusively distinguish between them.
SLAPP: Subgraph-level attention-based performance prediction for deep learning models

Authors:
raw_author_name: Zhenyi Wang
raw_affiliation_string: School of Computer Science and Technology, Xidian University, Xi’an, 710071, China; The Key Laboratory of Smart Human-Computer Interaction and Wearable Technology of Shaanxi Province, Xi’an, 710071, China
raw_author_name: Pengfei Yang
raw_affiliation_string: School of Computer Science and Technology, Xidian University, Xi’an, 710071, China; The Key Laboratory of Smart Human-Computer Interaction and Wearable Technology of Shaanxi Province, Xi’an, 710071, China
raw_author_name: Chengmin Lin
raw_affiliation_string: School of Computer Science and Technology, Xidian University, Xi’an, 710071, China; The Key Laboratory of Smart Human-Computer Interaction and Wearable Technology of Shaanxi Province, Xi’an, 710071, China
raw_author_name: Wenkai Lv
raw_affiliation_string: School of Computer Science and Technology, Xidian University, Xi’an, 710071, China; The Key Laboratory of Smart Human-Computer Interaction and Wearable Technology of Shaanxi Province, Xi’an, 710071, China
raw_author_name: Quan Wang
raw_affiliation_string: School of Computer Science and Technology, Xidian University, Xi’an, 710071, China; The Key Laboratory of Smart Human-Computer Interaction and Wearable Technology of Shaanxi Province, Xi’an, 710071, China

Abstract: The intricacy of the Deep Learning (DL) landscape, brimming with a variety of models, applications, and platforms, poses considerable challenges for the optimal design, optimization, or selection of suitable DL models. One promising avenue to address this challenge is the development of accurate performance prediction methods. However, existing methods reveal critical limitations. Operator-level methods, proficient at predicting the performance of individual operators, often neglect broader graph features, which results in inaccuracies in full network performance predictions. On the contrary, graph-level methods excel in overall network prediction by leveraging these graph features but lack the ability to predict the performance of individual operators. To bridge these gaps, we propose SLAPP, a novel subgraph-level performance prediction method. Central to SLAPP is an innovative variant of Graph Neural Networks (GNNs) that we developed, named the Edge Aware Graph Attention Network (EAGAT). This specially designed GNN enables superior encoding of both node and edge features. Through this approach, SLAPP effectively captures both graph and operator features, thereby providing precise performance predictions for individual operators and entire networks. Moreover, we introduce a mixed loss design with dynamic weight adjustment to reconcile the predictive accuracy between individual operators and entire networks. In our experimental evaluation, SLAPP consistently outperforms traditional approaches in prediction accuracy, including the ability to handle unseen models effectively. Moreover, when compared to existing research, our method demonstrates a superior predictive performance across multiple DL models.
Bayesian uncertainty quantification for anaerobic digestion models

Authors:
raw_author_name: Benjamin Guedj
raw_affiliation_string: Department of Computer science [University College of London]; Inria Lille - Nord Europe; MOdel for Data Analysis and Learning; The Inria London Programme; University College of London [London]

Abstract: Uncertainty quantification is critical for ensuring adequate predictive power of computational models used in biology. Focusing on two anaerobic digestion models, this article introduces a novel generalized Bayesian procedure, called VarBUQ, ensuring a correct tradeoff between flexibility and computational cost. A benchmark against three existing methods (Fisher's information, bootstrapping and Beale's criteria) was conducted using synthetic data. This Bayesian procedure offered a good compromise between fitting ability and confidence estimation, while the other methods proved to be repeatedly overconfident. The method's performances notably benefitted from inductive bias brought by the prior distribution, although it requires careful construction. This article advocates for more systematic consideration of uncertainty for anaerobic digestion models and showcases a new, computationally efficient Bayesian method. To facilitate future implementations, a Python package called 'aduq' is made available.