Reviewerly/exHarmony huggingface dataset:
Application of Adaptive Weighted Strong Tracking Unscented Kalman Filter in Non-Cooperative Maneuvering Target Tracking

An adaptive weighted strong tracking unscented Kalman filter is proposed in this paper for long-range relative navigation alongside non-cooperative maneuvering targets. First, an equation for obtaining the relative motion of two bodies is derived, it can be well adapted for a problem that has medium or long-distance. Secondly, a variance statistics function is introduced in the method to calculate residual weight in real time. The residual weight can be used to adjust the contribution of different measurement information to the fading factor. In this way, the sensitivity of the system to small pulse maneuvers is improved. Finally, the mean and covariance of the posterior state are calculated by the unscented transformation. A replacement equation for the fading factor is derived to improve the first-order approximation accuracy for a strong tracking system. Impulsive maneuvers with three different magnitudes are employed in a series of tests. Results from different methods showed that the proposed method could effectively detect pulse maneuvers with low latency. The proposed method is also numerically stable.
Panoramic video in education: A systematic literature review from 2011 to 2021

Abstract Background Panoramic video as a novel instructional media with increasing educational application. However, empirical research on panoramic video instruction is mostly episodic with conditional and inconclusive results. Objectives This study reviewed 10 years (2011–2021) of research on panoramic video in education to synthesize, meta‐analyse, and critically evaluate the state‐of‐art research findings, which can inform future educational practice and research directions. Methods Following the protocol of systematic review, this study selected 83 articles from four major research databases and analysed them from five interrelated aspects: instructional context, technological features, instructional design, research design, and instructional effect. A meta‐analysis was also conducted to determine the overall effectiveness of panoramic video on education and the moderating factors. Results and Conclusions The results revealed a rapid upward trend in publication since 2018 with unequal distribution among regions and learning domains. Despite the technical advancement in immersion and interaction, panoramic videos were found to mainly support the acquisition of content and factual knowledge through direct instruction, and lack in‐video scaffolding and assessment. Moreover, many interventions were one‐time, short‐term studies that are susceptible of novelty effect. Lastly, an overall medium effect ( g = 0.711) of panoramic video was reported and six significant moderating factors were identified based on the meta‐analysis results. Major takeaways and implications Panoramic videos are ubiquitously effective across grade levels, learning domains, and pedagogies, but are more suitable for teaching conceptual knowledge. Design features such as interaction, scaffolding, assessment, iteration, and video length deserve our special attention in panoramic video instruction.
Project-based learning approach to the formation of digital competencies of students of universities in Russia and India

Modern challenges in society require universities to move from a traditional to a competency-based technology integrated learning environment. Students must master a variety of newer complex competencies such as: learn to independently search for information, analyze it, and apply it to solve various real time problems through their content knowledge. One of the possible tools for developing these newer competencies is ICT integrated project-based learning (PBL). The present study conducts an ICT integrated practicum to the internationally paired undergraduate students of Indian and Russian universities. The practicum included a collaborative work towards a common theme. Findings of the study reveals that the outputs generated by ICT integrated PBL provided confidence to the participants to use technology tools in a global learning environment to solve real time problems. Study suggested requirements for project organization, technology tools, and a design to implement such projects. Further, it reveals how virtual teams work on a project in an online environment. Study recommends future researchers to focus on more such empirical studies and its efficacy.
A hybrid approach to three-way conversational recommendation

Conversational recommendation is ubiquitous in e-commerce, while three-way recommendation provides friendly choices for service providers and users. However, their combination has not been studied yet. In this paper, we introduce the three-way conversational recommendation problem and design the hybrid conversational recommendation (HTCR) algorithm to address it. First, a new recommendation problem is defined by considering the man–machine interaction as well as the misclassification and promotion costs. The optimization objective of the problem is to minimize the total cost. Second, a popularity-based technique is designed for user cold-start recommendation, where the user maturity is responsible for deciding when HTCR turns to the second technique. Third, an incremental matrix factorization technique is designed for regular recommendation. It is efficient since only a few rounds of training are needed for newly acquired user feedback. Experiments were carried out on four well-known datasets, including Jester, MovieLens 100K, MovieLens 1M and Yelp. Results demonstrated that our algorithm outperformed state-of-the-art ones in terms of average cost.
PPCNN: An efficient privacy‐preserving CNN training and inference framework

Convolutional neural network (CNN) is one of the representative models of deep learning, commonly used to analyze visual images. CNN model is more accurate when trained on large amounts of data from multiple sources, and the huge training cost makes the model much more valuable. However, data from various sources is often privacy-sensitive. Therefore, the privacy of these data should be protected during CNN model training and inference. In this paper, we propose an efficient and secure two-party computation (2PC) framework PPCNN for privacy-preserving CNN training and inference. Specifically, we use a new secret sharing technique introduced in ABY2.0 to securely compute various computational tasks involved in the CNN training and inference processes. This secret sharing technique can significantly reduce the communication overhead. Meanwhile, we assign these computationally intensive tasks to cloud servers to reduce the computational burden on local devices. We demonstrate the security of these protocols in the semihonest model. In addition, we use the MP-SPDZ library to simulate our PPCNN framework, and the experiments prove its high efficiency and accuracy.
Dcgnn: A Single-Stage 3d Object Detection Network Based on Density Clustering and Graph Neural Network

Currently, single-stage point-based 3D object detection network remains underexplored. Many approaches worked on point cloud space without optimization and failed to capture the relationships among neighboring point sets. In this paper, we propose DCGNN, a novel single-stage 3D object detection network based on density clustering and graph neural networks. DCGNN utilizes density clustering ball query to partition the point cloud space and exploits local and global relationships by graph neural networks. Density clustering ball query optimizes the point cloud space partitioned by the original ball query approach to ensure the key point sets containing more detailed features of objects. Graph neural networks are very suitable for exploiting relationships among points and point sets. Additionally, as a single-stage 3D object detection network, DCGNN achieved fast inference speed. We evaluate our DCGNN on the KITTI dataset. Compared with the state-of-the-art approaches, the proposed DCGNN achieved better balance between detection performance and inference time.
Key Point-Based Driver Activity Recognition

We present a key point-based activity recognition framework, built upon pre-trained human pose estimation and facial feature detection models. Our method extracts complex static and movement-based features from key frames in videos, which are used to predict a sequence of key-frame activities. Finally, a merge procedure is employed to identify robust activity segments while ignoring outlier frame activity predictions. We analyze the different components of our framework via a wide array of experiments and draw conclusions with regards to the utility of the model and ways it can be improved. Results show our model is competitive, taking the 11th place out of 27 teams submitting to Track 3 of the 2022 AI City Challenge.
Does Federated Dropout actually work?

Model sizes are limited in Federated Learning due to network bandwidth and on-device memory constraints. The success of increasing model sizes in other machine learning domains motivates the development of methods for training large-scale models in Federated Learning. To this end, [3] draws inspiration from dropout and proposes Federated Dropout: an algorithm where clients train randomly selected subsets of a larger server model. Despite the promising empirical results and the many other works that build on it [1],[8],[13], we argue in this paper that the metrics used to measure the performance of Federated Dropout and its variants are misleading. We propose and perform new experiments which suggest that Federated Dropout is actually detrimental to scaling efforts. We show how a simple ensembling technique outperforms Federated Dropout and other baselines. We perform ablations that suggest that the best performing variations of Federated Dropout approximate ensembling. The simplicity of ensembling allows for easy, practical implementations. Furthermore, ensembling naturally leverages the parallelizable nature of Federated Learning— recall that it is easy to train several models independently because there are a lot of clients and server-compute is not the bottleneck. Ensembling’s strong performance against our baselines suggests that Federated Learning models may be more easily scaled than previously thought with more sophisticated ensembling strategies e.g., via boosting.
Unpaired Real-World Super-Resolution with Pseudo Controllable Restoration

Current super-resolution methods rely on the bicubic down-sampling assumption in order to develop the ill-posed reconstruction of the low-resolution image. Not surprisingly, these approaches fail when using real-world low-resolution images due to the presence of artifacts and intrinsic noise absent in the bicubic setup. Consequently, attention is increasingly paid to techniques that alleviate this problem and super-resolve real-world images. As acquiring paired real-world datasets is a challenging problem, real-world super-resolution solutions are traditionally tackled as a blind problem or as an unpaired data-driven problem. The former makes assumptions about the downsampling operations, the latter uses unpaired training to learn the real distributions. Recently, blind approaches have dominated this problem by assuming a diverse bank of degradations, whereas the unpaired solutions have shown under-performance due to the two-staged training. In this paper, we propose an unpaired real-world super-resolution method that performs on par, or even better than blind paired approaches by introducing a pseudo-controllable restoration module in a fully end-to-end system.
Multiple Object Detection and Tracking in the Thermal Spectrum

Multiple Object Tracking (MOT) is an integral part of machine vision research. Most tracking-by-detection based MOT solutions utilize video streams from RGB cameras for their operation. However, for real-world applications, it is necessary to utilize sensors that operate in different spectrums to accommodate for varying lighting conditions. Since object detection is the first step of the tracking pipeline in tracking-by-detection approaches, we compare the performance of state-of-the-art object detectors when trained on color images to their performance when trained on thermal images. We introduce a new dataset for multiple object tracking with thermal images and corresponding RGB images and show that state-of-the-art trackers perform better on thermal images, especially in poor lighting conditions. Finally, we propose the use of a dynamic cut-off thresh-old for tracking-by-detection approaches that factors the size of a predicted box to enhance the tracker association. Our dataset and source code are publicly available at https://github.com/wassimea/thermalMOT
Guiding Attention using Partial-Order Relationships for Image Captioning

The use of attention models for automated image captioning has enabled many systems to produce accurate and meaningful descriptions for images. Over the years, many novel approaches have been proposed to enhance the attention process using different feature representations. In this paper, we extend this approach by creating a guided attention network mechanism, that exploits the relationship between the visual scene and text-descriptions using spatial features from the image, high-level information from the topics, and temporal context from caption generation, which are embedded together in an ordered embedding space. A pairwise ranking objective is used for training this embedding space which allows similar images, topics and captions in the shared semantic space to maintain a partial order in the visual-semantic hierarchy and hence, helps the model to produce more visually accurate captions. The experimental results based on MSCOCO dataset shows the competitiveness of our approach, with many state-of-the-art models on various evaluation metrics.
A Multiview Depth-based Motion Capture Benchmark Dataset for Human Motion Denoising and Enhancement Research

The field of human motion enhancement is a rapidly expanding field of study in which depth-based motion capture (D-Mocap) is improved to generate a more accurate counterpart for demanding high precision real-world applications. The D-Mocap that is initially generated relies on commercially available SDKs or open source tools to produce the initial skeletal sequence which works best in an ideal front-facing camera setup. This in turn creates a challenging initialization for human motion enhancement when the camera is not positioned in the ideal forward facing position. Currently there are no multiview D-Mocap datasets which have corresponding time-synced and skeleton-matched optical motion capture (Mocap) reference data for view-invariant motion enhancement. We develop a multiview D-Mocap dataset extended from the popular and comprehensive Berkeley MHAD dataset [29]. In addition, we analyze the performance of the D-Mocap data generated through a series of open source tools, highlighting the difficulty and the need to produce robust results in a rear-facing camera setup due to a 21.4% increase in average joint position error over front-facing data. Finally, we analyze the results of some recent human motion enhancement algorithms with regard to a front-facing camera setup versus a rear-facing one.
MV-TAL: Mulit-view Temporal Action Localization in Naturalistic Driving

Human risky behavior in driving is an important visual recognition problem. In this paper, we propose a multi-view temporal action localization system based on the grayscale video to achieve action recognition in naturalistic driving. Specifically, we adopted SwinTransformer as feature extractor, and a single framework to detect boundary and class at the same time. Also, we improve multiple loss function for explicit constraints of embedded feature distributions. Our proposed framework achieves the overall F1-score of 0.3154 on A2 dataset.
Cross-modal Image Synthesis within Dual-Energy X-ray Security Imagery

Dual-energy X-ray scanners are used for aviation security screening given their capability to discriminate materials inside passenger baggage. To facilitate manual operator inspection, a pseudo-colouring is assigned to the effective composition of the material. Recently, paired image to image translation models based on conditional Generative Adversarial Networks (cGAN) have shown to be effective for image colourisation. In this work, we investigate the use of such a model to translate from the raw X-ray energy responses (high, low, effective-Z) to the pseudo-coloured images and vice versa. Specifically, given N X-ray modalities, we train a cGAN conditioned in N − m domains to generate the remaining m representation. Our method achieves a mean squared error (MSE) of 16.5 and a structural similarity index (SSIM) of 0.9815 when using the raw modalities to generate the pseudo-colour representation. Additionally, raw X-ray high energy, low energy and effective-Z projections were generated given the pseudo-colour image with minimum MSE of 2.57, 5.63 and 1.43, and maximum SSIM of 0.9953, 0.9901 and 0.9921. Furthermore, we assess the quality of our synthesised pseudo-colour reconstructions by measuring the performance of two object detection models originally trained on real X-ray pseudo-colour images over our generated pseudo-colour images. Interestingly, our generated pseudo-colour images obtain marginally improved detection performance than the corresponding real X-ray pseudo-colour images, showing that meaningful representations are synthesized and that these reconstructions are applicable for differing aviation security tasks.
Multi-Dimensional Vision Transformer Compression via Dependency Guided Gaussian Process Search

Vision transformers (ViT) have recently attracted considerable attentions, but the huge computational cost remains an issue for practical deployment. Previous ViT pruning methods tend to prune the model along one dimension solely, which may suffer from excessive reduction and lead to sub-optimal model quality. In contrast, we advocate a multi-dimensional ViT compression paradigm, and propose to harness the redundancy reduction from attention head, neuron and sequence dimensions jointly. Firstly, we propose a statistical dependence based pruning criterion that is generalizable to different dimensions for identifying the deleterious components. Moreover, we cast the multidimensional ViT compression as an optimization problem, objective of which is to learn an optimal pruning policy across the three dimensions while maximizing the compressed model’s accuracy under a computational budget. The problem is solved by an adapted Gaussian process search with expected improvement. Experimental results show that our method effectively reduces the computational cost of various ViT models. For example, our method reduces 40% FLOPs without top-1 accuracy loss for DeiT and T2T-ViT models on the ImageNet dataset, outperforming previous state-of-the-art ViT pruning methods.
Modeling Missing Annotations for Incremental Learning in Object Detection

Despite the recent advances in the field of object detection, common architectures are still ill-suited to incrementally detect new categories over time. They are vulnerable to catastrophic forgetting: they forget what has been already learned while updating their parameters in absence of the original training data. Previous works extended standard classification methods in the object detection task, mainly adopting the knowledge distillation framework. However, we argue that object detection introduces an additional problem, which has been overlooked. While objects belonging to new classes are learned thanks to their annotations, if no supervision is provided for other objects that may still be present in the input, the model learns to associate them to background regions. We propose to handle these missing annotations by revisiting the standard knowledge distillation framework. Our approach outperforms current state-of-the-art methods in every setting of the Pascal-VOC dataset. We further propose an extension to instance segmentation, outperforming the other baselines.
Image Quality Assessment with Transformers and Multi-Metric Fusion Modules

Image quality assessment is crucial for low-level vision tasks such as compression, super-resolution, denoising and etc. It guides researchers how to design networks, design loss functions, and decide the optimization direction of networks. A good quality assessment metric should comform to people’s subjective feelings as much as possible. Traditional PSNR and MS-SSIM have more and more obvious shortcomings in quality evaluation with the popularity of GANs. Inspired by metrics such as LPIPS, IQT, etc., we decided to design a metric that is learned by the network itself. In this paper, we use a ConvNeXt-Tiny network to extract features and calculate nonlinear residuals between reference images and distorted images. We feed residuals into transformers to compare the degree of distortion. In addition, we use multi-metric fusion to improve the performance of our network. Our model achieves 0.780 accuracy on CLIC validation set. Our code is available at https://github.com/JiangWeibeta/IQA-TMFM.
DeepACO: A Robust Deep Learning-based Automatic Checkout System

The retail industry has seen an increasing growth of artificial intelligence and computer vision applications. Of the various topics, automatic checkout (ACO) in retail stores or supermarkets has emerged as one of the critical tasks in this area. Several problems stem from real-world scenarios such as object occlusion, blurring from scanning motion, and similarity in scanned items. Moreover, the challenge also comes from the difficulty of collecting training images that reflect the realistic checkout scenarios due to continuous updates of the products. This paper proposes a deep learning-based automatic checkout system (DeepACO) to recognize, localize, track, and count products as they move along a retail check-out conveyor belt. The DeepACO follows the detect-and-track approach, i.e., applying trackers on detected bounding boxes. It also provides a completed pipeline for generating large training datasets under various environments from synthetic data. The proposed system has been evaluated on the 2022 AI City Challenge Track 4 benchmark. Compared to other state-of-the-art solutions, it has shown outstanding results, achieving top-2 on the test-set A with the F1 score of 0.4783.
SISL:Self-Supervised Image Signature Learning for Splicing Detection &amp; Localization

Recent algorithms for image manipulation detection almost exclusively use deep network models. These approaches require either dense pixelwise groundtruth masks, camera ids, or image metadata to train the networks. On one hand, constructing a training set to represent the countless tampering possibilities is impractical. On the other hand, social media platforms or commercial applications are often constrained to remove camera ids as well as metadata from images. A self-supervised algorithm for training manipulation detection models without dense groundtruth or camera/image metadata would be extremely useful for many forensics applications. In this paper, we propose a self-supervised approach for training splicing detection/localization models from frequency transform of images. To identify the spliced regions, our deep network learns a representation to capture an image-specific signature by enforcing (image) self consistency. We experimentally demonstrate that our proposed model can yield similar or better performances as compared to multiple existing methods on standard datasets without relying on labels or metadata.