Reviewerly/exHarmony huggingface dataset:
A Visual Saliency-Based Neural Network Architecture for No-Reference Image Quality Assessment

Deep learning has recently been used to study blind image quality assessment (BIQA) in great detail. Yet, the scarcity of high-quality algorithms prevents from developing them further and being used in a real-time scenario. Patch-based techniques have been used to forecast the quality of an image, but they typically award the picture quality score to an individual patch of the image. As a result, there would be a lot of misleading scores coming from patches. Some regions of the image are important and can contribute highly toward the right prediction of its quality. To prevent outlier regions, we suggest a technique with a visual saliency module which allows the only important region to bypass to the neural network and allows the network to only learn the important information required to predict the quality. The neural network architecture used in this study is Inception-ResNet-v2. We assess the proposed strategy using a benchmark database (KADID-10k) to show its efficacy. The outcome demonstrates better performance compared with certain popular no-reference IQA (NR-IQA) and full-reference IQA (FR-IQA) approaches. This technique is intended to be utilized to estimate the quality of an image being acquired in real time from drone imagery.
Self-paced Adaptive Bipartite Graph Learning for Consensus Clustering

Consensus clustering provides an elegant framework to aggregate multiple weak clustering results to learn a consensus one that is more robust and stable than a single result. However, most of the existing methods usually use all data for consensus learning, whereas ignoring the side effects caused by some unreliable or difficult data. To address this issue, in this article, we propose a novel self-paced consensus clustering method with adaptive bipartite graph learning to gradually involve data from more reliable to less reliable ones in consensus learning. At first, we construct an initial bipartite graph from the base results, where the nodes represent the clusters and instances, and the edges indicate that an instance belongs to a cluster. Then, we adaptively learn a structured bipartite graph from this initial one by self-paced learning, i.e., we automatically determine the reliability of each edge with adaptive cluster similarity measuring and involve the edges in bipartite graph learning in order of their reliability. At last, we obtain the final consensus result from the learned structured bipartite graph. We conduct extensive experiments on both toy and benchmark datasets, and the results show the effectiveness and superiority of our method. The codes of this article are released in http://Doctor-Nobody.github.io/codes/code_SCCABG.zip.
DenseBert4Ret: Deep bi-modal for image retrieval

In this study, we focus on the task of desired image retrieval from an extensive database. This comes under the purview of image retrieval systems. Users may input an image and a text as a query and expect the system to retrieve an image. The retrieved image should be close to the users' desires expressed in the query. Nowadays, digital media generates more than petabytes of imagery data in one day. Moreover, due to the internet, this massive amount of data is readily available to users. However, extracting the desired images from such a colossal databank is a challenging task. Users always prefer to extract an image that reflects their wishful thinking, and the user may desire to alter their visionary thoughts according to their abstract thoughts. For example, Elizabeth wants to have a laptop that is similar to her friend's laptop. However, she wants to have the same kind of laptop with a built-in GPU and in silver color. So she expects the e-business platform to show a laptop according to her wish. This paper attempts to devise a multi-modal algorithm for such tasks. It takes care of the user's visual and textual query. It has a query image and text as input and retrieves an image similar to the input image but modified according to the text query. This study focuses on a multi-modal image retrieval system that processes both image and text as input queries. Users can input an image and a text query to modify the image or add more information on it. The text reflects the desired modifications in the image. We proposed a bi-modal image retrieval system named "DenseBert4Ret" that learns image and text features concurrently. As the name indicates, DenseNet and BERT models are used for image and text features extractions, respectively. It is based on deep learning techniques used for the joint representation of image and text features. We trained the model, which forces the input image to be modified according to the user's textual query. We used deep information learning to train and test our model on three challenging real-world datasets, i.e., MIT States, Fshion200K and FashionIQ. We also show that the proposed model outperforms its predecessor with tuned parameters.
Deep learning of monocular depth, optical flow and ego-motion with geometric guidance for UAV navigation in dynamic environments

Computer vision-based depth estimation and visual odometry provide perceptual information useful for robot navigation tasks like obstacle avoidance. However, despite the proliferation of state-of-the-art convolutional neural network (CNN) models for monocular depth, ego-motion and optical flow estimation, a relatively low volume of work has been reported on their practical applications in unmanned aerial vehicle (UAV) navigation. This is due to well-known challenges — embedded hardware constraints, viewpoint variations, scarcity of aerial image datasets, and intricacies of dynamic environments. We address these limitations to facilitate real-world deployment of CNN in UAV navigation. First, we devise efficient confidence weighted adaptive network (Cowan) training framework that iteratively leverages intermediate prediction confidences to enforce cross-task consistency over corresponding image regions. This achieves competitive accuracy with a lightweight CNN capable of real-time execution on resource-constrained embedded systems. Second, we devise a test-time refinement method that adapts the network to dynamic environments while simultaneously improving accuracy. To accomplish this, we first update ego-motion using pose information from on-board inertial measurement unit (IMU). Then, we decompose the UAV’s motion into constituent vectors, and for each axis, we formulate geometric relationships between depth and translation. Based on this information, we triangulate corresponding points acquired through optical flow. Finally, we enforce geometric consistency between the initially updated pose and triangulated depth. Cowan with geometric guided refinement (Cowan-GGR) achieves significant accuracy and robustness. Field tests show the proposed model is capable of accurate depth and object-level motion perception in real-world dynamic environments, thus proving its efficacy in facilitating UAV navigation.
The Building That Teaches: Exploring Augmented Reality Affordances in Academic Incubators

Efforts to enhance the experience of interior environments have led to experimentation with augmented reality (AR) technology to encourage users to participate in the built space using their mobile devices. To investigate the role of AR technologies in interior design experiences, we highlight the interdisciplinary design of a mobile AR-application to assess and enhance the effectiveness of building design strategies for a Leadership in Energy and Environmental Design (LEED)-Gold academic incubator. Drawing on user feedback data, observations, and in-depth interviews of 15 building occupants, we identified the affordances of mobile-AR for interior design. Our findings suggest approaches for contextualizing post-occupancy user feedback and interior design across the digital-physical spatial continuum and illustrate how AR has the potential to expand the goals of post-occupancy evaluations, beyond improved evaluation to enhancement, by allowing the building to teach occupants about resources while nudging them to utilize spatial features designed to enhance wellness. As hybrid interior environments are becoming more dependent on a convergence of the digital and physical, we found a series of strategies and practices for enhancing user experiences. A key contribution of our research is a framework, grounded in affordance theory, useful for designing and examining hybrid spaces at the intersection of AR and interior design.
MobyDeep: A lightweight CNN architecture to configure models for text classification

Nowadays, trends in deep learning for text classification are addressed to create complex models to deal with huge datasets. Deeper models are usually based on cutting edge neural network architectures, achieving good results in general but demanding better hardware than shallow ones. In this work, a new Convolutional Neural Network (CNN) architecture (MobyDeep) for text classification tasks is proposed. Designed as a configurable tool, resultant models (MobyNets) are able to manage big corpora sizes under low computational costs. To achieve those milestones, the architecture was conceived to produce lightweight models, having their internal layers based on a new proposed convolutional block. That block was designed and customized by adapting ideas from image to text processing, helping to squeezing model sizes and to reduce computational costs. The architecture was also designed as a residual network, covering complex functions by extending models up to 28 layers. Moreover, middle layers were optimized by residual connections, helping to remove fully connected layers on top and resulting in Fully CNN. Corpus were chosen from the recent literature, aiming to define real scenarios when comparing configured MobyDeep models with other state-of the-art works. Thus, three models were configured in 8, 16 and 28 layers respectively, offering competitive accuracy results.
Shift Pose: A Lightweight Transformer-like Neural Network for Human Pose Estimation

High-performing, real-time pose detection and tracking in real-time will enable computers to develop a finer-grained and more natural understanding of human behavior. However, the implementation of real-time human pose estimation remains a challenge. On the one hand, the performance of semantic keypoint tracking in live video footage requires high computational resources and large parameters, which limiting the accuracy of pose estimation. On the other hand, some transformer-based models were proposed recently with outstanding performance and much fewer parameters and FLOPs. However, the self-attention module in the transformer is not computationally friendly, which makes it difficult to apply these excellent models to real-time jobs. To overcome the above problems, we propose a transformer-like model, named ShiftPose, which is regression-based approach. The ShiftPose does not contain any self-attention module. Instead, we replace the self-attention module with a non-parameter operation called the shift operator. Meanwhile, we adapt the bridge-branch connection, instead of a fully-branched connection, such as HRNet, as our multi-resolution integration scheme. Specifically, the bottom half of our model adds the previous output, as well as the output from the top half of our model, corresponding to its resolution. Finally, the simple, yet promising, disentangled representation (SimDR) was used in our study to make the training process more stable. The experimental results on the MPII datasets were 86.4 PCKH, 29.1PCKH@0.1. On the COCO dataset, the results were 72.2 mAP and 91.5 AP50, 255 fps on GPU, with 10.2M parameters, and 1.6 GFLOPs. In addition, we tested our model for single-stage 3D human pose estimation and draw several useful and exploratory conclusions. The above results show good performance, and this paper provides a new method for high-performance, real-time attitude detection and tracking.
Vehicle trajectory data extraction from the horizontal curves of mountainous roads

Trajectory data is essential for understanding the driver-vehicle-road interaction which is crucial for safety and operational assessment. The video image-processing technique is useful to collect such data on the entire traffic stream. Existing techniques assume that the road is planar which limits the application of the video-based traffic data collection to the plain terrains. With the objective to collect trajectory data from mountainous terrain, the present study proposes an approach to convert the pixel coordinates into real-world coordinates. The hypothesis was that a non-planar scene could be converted into a sequence of piece-wise planar scenes with a separate projection matrix for each planar region. The analysis shows that the proposed method could effectively transform the pixel coordinates into the real-world coordinates. Comparison of the estimated and observed speed/path profiles indicate the adequacy of the proposed method in collecting video-based trajectory data from mountainous terrain.
MetaViz – A graphical meta-model instantiator for generating information dashboards and visualizations

This work presents the application of meta-modeling to the data visualization and dashboards' domain to obtain two main products: a system that allows the instantiation of the meta-model through a graphical interface and a code generator that takes the instantiation of the meta-model as an input to generate visualizations and dashboards. A domain engineering approach complemented with an example-driven methodology was iteratively employed to develop the dashboard meta-model. This meta-model was subsequently used as an input to implement a code generator of information dashboards. These two artifacts were finally combined to design and develop the architecture of MetaViz. Through this process, it was possible to generate visualizations and dashboards using visual elements and basic interactions. MetaViz allows the generation of basic charts (line charts, scatter plots, pie charts, etc.) as well as more complex displays with interactive behavior along different views, layouts, and operations. The development of MetaViz has served as proof of the viability and benefits of applying these methodologies to a complex domain, but also to set the foundations of a system that allows users to trace every single element from a data visualization to its most primitive values.
Use of modeling to map potentially toxic elements and assess the risk to human health in soils affected by mining activity

Determining the levels and spatial distribution of potentially toxic elements (PTEs) is essential in designing mitigating strategies to remediate contaminated areas. Efficient prediction of PTEs is difficult due to high spatial variability and multiple sources of contamination. The use of machine learning algorithms is an alternative for the improvement of the predictions. Objective of the study was to model the spatial distribution of PTEs in soils surrounding a tailings pile and to assess potential risks to human health. A total set of 116 soil samples was collected to a depth of 0–10 cm. The covariate database was composed of morphometric variables derived from a DEM and the creation of a tailings pile location covariate. The models tested were Cubist (CUB), Random Forest (RF), Support Vector Machine (SVM), and Linear Model (Lm). Based on the best predictive model, maps showing the spatial distribution, pollution and risk to human health were created. The SVM model had the best performance for predicting spatially the soil contents of Pb and Zn, with lower values of mean absolute error (MAE − 744.71; 306.36) and higher values of R2 (0.56; 0.52). It was possible to verify that the mining tailings pile is the main source of Pb and Zn contamination in the study area. The enrichment factor (EF) and the geoaccumulation index (Igeo) of Pb and Zn show that the soils close to the tailings piles are extremely contaminated (Igeo > 5 and EF > 40), and pose risks to human health (Total carcinogenic risk > 1 × 10–6), especially for children. Therefore, new studies must be carried out with the aim of monitoring and rehabilitating the contaminated areas.
Research on a Real-Time Monitoring Method for the Three-Dimensional Straightness of a Scraper Conveyor Based on Binocular Vision

Measuring the straightness of the scraper conveyor, which is an indispensable piece of equipment in a fully mechanized coal face, can prevent accidents such as derailment of the shearer and is also important for the precise positioning of the shearer and the accurate control of the hydraulic support. The existing scraper conveyor straightness measurement methods have the disadvantages of inconsistent measurement cost, accuracy, and reliability as well as low dimension of straightness description. To this end, this paper proposes a method for monitoring the three-dimensional straightness of a scraper conveyor based on binocular vision. Trapezoidal window matching technology was used to realize the image acquisition of multiple stations and multiple sensors. The bit pose acquisition model of binocular vision 3D reconstruction was used to obtain the 3D coordinates of the feature points on the sign board in each local coordinate system and the bit pose information of each sign board. The pose relay videometric method was used to convert the straightness in each local coordinate system to the global coordinate system to realize the 3D reconstruction of the scraper conveyor. Finally, the straightness measurement test of the preset scraper conveyor was carried out, and the error analysis was carried out by comparing with the manual measurement results, which showed that the measurement errors were all within ±30 mm. At the same time, the standard deviation of errors in both directions was small, which indicates that the straightness measurement method for the scraper conveyor in this paper has high reliability. In addition, the visual measurement process is independent of each other, so there is no error accumulation in the visual straightness measurement method. The analysis shows that the method of measuring the straightness of a scraper conveyor based on binocular vision can realize continuous and real-time monitoring of the scraper conveyor.
Blind-noise image denoising with block-matching domain transformation filtering and improved guided filtering

Abstract The adaptive block size processing method in different image areas makes block-matching and 3D-filtering (BM3D) have a very good image denoising effect. Based on these observation, in this paper, we improve BM3D in three aspects: adaptive noise variance estimation, domain transformation filtering and nonlinear filtering. First, we improve the noise-variance estimation method of principle component analysis using multilayer wavelet decomposition. Second, we propose compressive sensing based Gaussian sequence Hartley domain transform filtering to reduce noise. Finally, we perform edge-preserving smoothing on the preprocessed image using the guided filtering based on total variation. Experimental results show that the proposed denoising method can be competitive with many representative denoising methods on the evaluation criteria of PSNR. However, it is worth further research on the visual quality of denoised images.
A fusion framework for vision-based indoor occupancy estimation

Building occupancy information is essential for energy saving, comfort improvement, and security management. Existing vision-based indoor occupancy measurement methods have achieved remarkable progress; however, they mainly focus on single-vision situations, i.e., cameras at room entrances or interiors. These methods struggle to achieve high accuracy because of the complex indoor environments. For example, they often fail to detect occupants and generate many false positives. In this paper, to address these issues, we propose a novel fusion framework for occupancy detection and estimation based on two different perspectives. First, we design a head detection method combined with indoor scene knowledge to filter false positives and recover missed detection. Second, we propose a two-vision entrance counting method to refine the predicted results. Finally, we propose a cumulative error clearing strategy named dynamic Bayesian fusion (DBF), which integrates entrance counting and static estimation. Our framework achieves superior performance through ablation studies compared to existing methods on practical building surveillance videos, with occupancy estimation SCOREs of 99.2%, 98.5%, and 94.9%. Our framework can clear cumulative errors and stabilize estimation results. Practical experiments validate its potential for building energy saving and comfort improvement. The code is available at https://github.com/kailaisun/FFO.
Security and Privacy Threats to Federated Learning: Issues, Methods, and Challenges

Federated learning (FL) has nourished a promising method for data silos, which enables multiple participants to construct a joint model collaboratively without centralizing data. The security and privacy considerations of FL are focused on ensuring the robustness of the global model and the privacy of participants’ information. However, the FL paradigm is under various security threats from the adversary aggregator and participants. Therefore, it is necessary to comprehensively identify and classify potential threats to provide a theoretical basis for FL with security guarantees. In this paper, a unique classification of attacks, which reviews state-of-the-art research on security and privacy issues for FL, is constructed from the perspective of malicious threats based on different computing parties. Specifically, we categorize attacks with respect to performed by aggregator and participant, highlighting the Deep Gradients Leakage attacks and Generative Adversarial Networks attacks. Following an overview of attack methods, we discuss the primary mitigation techniques against security risks and privacy breaches, especially the application of blockchain and Trusted Execution Environments. Finally, several promising directions for future research are discussed.
User Identification Utilizing Minimal Eye-Gaze Features in Virtual Reality Applications

Emerging Virtual Reality (VR) displays with embedded eye trackers are currently becoming a commodity hardware (e.g., HTC Vive Pro Eye). Eye-tracking data can be utilized for several purposes, including gaze monitoring, privacy protection, and user authentication/identification. Identifying users is an integral part of many applications due to security and privacy concerns. In this paper, we explore methods and eye-tracking features that can be used to identify users. Prior VR researchers explored machine learning on motion-based data (such as body motion, head tracking, eye tracking, and hand tracking data) to identify users. Such systems usually require an explicit VR task and many features to train the machine learning model for user identification. We propose a system to identify users utilizing minimal eye-gaze-based features without designing any identification-specific tasks. We collected gaze data from an educational VR application and tested our system with two machine learning (ML) models, random forest (RF) and k-nearest-neighbors (kNN), and two deep learning (DL) models: convolutional neural networks (CNN) and long short-term memory (LSTM). Our results show that ML and DL models could identify users with over 98% accuracy with only six simple eye-gaze features. We discuss our results, their implications on security and privacy, and the limitations of our work.