Reviewerly/exHarmony huggingface dataset:
A No-Reference Deep Learning Quality Assessment Method for Super-Resolution Images Based on Frequency Maps

To support the application scenarios where high-resolution (HR) images are urgently needed, various single image super-resolution (SISR) algorithms are developed. However, SISR is an ill-posed inverse problem, which may bring artifacts like texture shift, blur, etc. to the reconstructed images, thus it is necessary to evaluate the quality of super-resolution images (SRIs). Note that most existing image quality assessment (IQA) methods were developed for synthetically distorted images, which may not work for SRIs since their distortions are more diverse and complicated. Therefore, in this paper, we propose a no-reference deep-learning image quality assessment method based on frequency maps because the artifacts caused by SISR algorithms are quite sensitive to frequency information. Specifically, we first obtain the high-frequency map (HM) and low-frequency map (LM) of SRI by using Sobel operator and piecewise smooth image approximation. Then, a two-stream network is employed to extract the quality-aware features of both frequency maps. Finally, the features are regressed into a single quality value using fully connected layers. The experimental results show that our method outperforms all compared IQA models on the selected three super-resolution quality assessment (SRQA) databases.
Advancing the Design of Visual Debugging Tools for Roboticists

Programming robots is a challenging task exacer-bated by software bugs, faulty hardware, and environmental fac-tors. When coding issues arise, traditional debugging techniques such as output logs or print statements that may help in typical computer applications are not always useful for roboticists. As a result, roboticists often leverage visualizations that depict various aspects of robot, sensor, and environment states. In this paper, we explore various design approaches towards such visualizations for robotics debugging support, including 3D visualizations presented on 2D displays, as in the popular RViz tool within the ROS ecosystem, visualizations in a two-dimensional graphical user interfaces (2D GUI), and emerging immersive three-dimensional (3D) augmented reality (AR). We present a qualitative evaluation of feedback gathered from 24 roboticists across two universities who used one of these debugging tools and synthesize design guidelines for advancing robotics debugging interfaces.
Boosting Robustness of Image Matting with Context Assembling and Strong Data Augmentation

Deep image matting methods have achieved increasingly better results on benchmarks (e.g., Composition-1k/alphamatting.com). However, the robustness, including robustness to trimaps and generalization to images from different domains, is still underexplored. Although some works propose to either refine the trimaps or adapt the algorithms to real-world images via extra data augmentation, none of them has taken both into consideration, not to mention the significant performance deterioration on benchmarks while using those data augmentation. To fill this gap, we propose an image matting method which achieves higher robustness (RMat) via multilevel context assembling and strong data augmentation targeting matting. Specifically, we first build a strong matting framework by modeling ample global information with transformer blocks in the encoder, and focusing on details in combination with convolution layers as well as a low-level feature assembling attention block in the decoder. Then, based on this strong baseline, we analyze current data augmentation and explore simple but effective strong data augmentation to boost the baseline model and contribute a more generalizable matting method. Compared with previous methods, the proposed method not only achieves state-of-the-art results on the Composition-1k benchmark (11 % improvement on SAD and 27% improvement on Grad) with smaller model size, but also shows more robust generalization results on other benchmarks, on real-world images, and also on varying coarse-to-fine trimaps with our extensive experiments. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> This work was in part done when YD was an intern at Adobe and CS was with The University of Adelaide. CS is the corresponding author. Project page: https://dongdong93.github.io/RMat/.
A High-Speed FPGA-Based True Random Number Generator Using Metastability With Clock Managers

True random number generators (TRNGs) are fundamentals in many important security applications. Though they exploit randomness sources that are typical of the analog domain, digital-based solutions are strongly required especially when they have to be implemented on Field Programmable Gate Array (FPGA)-based digital systems. This brief describes a novel methodology to easily design a TRNG on FPGA devices. It exploits the runtime capability of the Digital Clock Manager (DCM) hardware primitives to tune the phase shift between two clock signals. The presented auto-tuning strategy automatically sets the phase difference of two clock signals in order to force on one or more flip-flops (FFs) to enter the metastability region, used as a randomness source. Moreover, a novel use of the fast carry-chain hardware primitive is proposed to further increase the randomness of the generated bits. Finally, an effective on-chip post-processing scheme that does not reduce the TRNG throughput is described. The proposed TRNG architecture has been implemented on the Xilinx Zynq XC7Z020 System on Chip (SoC). It passed all the National Institute of Standards and Technology (NIST) SP 800–22 statistical tests with a maximum throughput of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$300 \times 10^{6}$ </tex-math></inline-formula> bit per second. The latter is considerably higher than the throughput of other previously published DCM-based TRNGs.
Gaussian Activated Neural Radiance Fields for High Fidelity Reconstruction and Pose Estimation

Despite Neural Radiance Fields (NeRF) showing compelling results in photorealistic novel views synthesis of real-world scenes, most existing approaches require accurate prior camera poses. Although approaches for jointly recovering the radiance field and camera pose exist, they rely on a cumbersome coarse-to-fine auxiliary positional embedding to ensure good performance. We present Gaussian Activated Neural Radiance Fields (GARF), a new positional embedding-free neural radiance field architecture – employing Gaussian activations – that is competitive with the current state-of-the-art in terms of high fidelity reconstruction and pose estimation.
Systematic Literature Review: Blockchain Security in NFT Ownership

In the current era, blockchain technology has been used to store non-fungible NFT digital asset ownership data, which means that the characteristics of each token are unique and different from the others. Because an NFT is a digital asset that cannot be exchanged, an NFT can have unauthorized copies and can be traded by irresponsible people. In this study, we have made observations regarding how the protection of NFT property rights is maintained in the NFT and Blockchain security ecosystem. This paper has also explained how the NFT ecosystem works especially in terms of copyright and NFT fraud. In this study, we use the Systematic Literature Review to get more information about the NFT ecosystem and its security. From the results of this paper, it is concluded that there are security problems in the field of copyright infringement, data theft, and plagiarism that cannot be handled by Blockchain. The security issues must be circumvented by NFT traders themselves to prevent NFT security issues previously described.
Circular LBP Prior-Based Enhanced GAN for Image Style Transfer

Image style transfer (IST) has drawn broad attention recently. At present, convolutional neural network (CNN)-based methods and generative adversarial network (GAN)-based methods have been broadly utilized in IST. However, the texture of images obtained by most methods presents a lower definition, which leads to insufficient details of IST. To this end, the authors present a new IST method based on an enhanced GAN with a prior circular local binary pattern (LBP). They utilize circular LBP in a GAN generator as a texture prior to improve the detailed textures of the generated style images. Meanwhile, they integrate a dense connection residual block and an attention mechanism into the generator to further improve high-frequency feature extraction. In addition, the total variation (TV) regularizer is integrated into the loss function to smooth the training results and restrain the noise. The qualitative and quantitative experimental results demonstrate that the metric quality of the generated images can achieve better effects by the proposed strategy compared with other popular approaches.
Area-Efficient Number Theoretic Transform Architecture for Homomorphic Encryption

Homomorphic encryption (HE) has emerged as an ideal cryptographic technology for meaningful computations on encrypted data. Not only does HE secure private information even if the ciphertext is leaked, but it also maintains data integrity when inferring cloud-side services. However, homomorphic computations include expensive polynomial arithmetic, especially polynomial multiplication. Prior studies proposed number theoretic transform (NTT) hardware designs to accelerate polynomial multiplication. However, the trade-off between hardware complexity and throughput of NTT designs was not considered carefully. This paper proposes an area-efficient NTT architecture suitable for HE schemes. Center of the proposed NTT architecture is a high-throughput butterfly unit array, which communicates with a single data memory unit through a conflict-free memory access pattern. Additionally, we developed a twiddle factor generator to reduce memory consumption. The proposed NTT architecture was successfully accelerated on the Xilinx FPGA devices. Performing with a large number of moduli, the proposed NTT design achieves higher hardware efficiency than the prior arts. Especially, our NTT design consumes less on-chip memory with efficiency improvement of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$8.8\times $ </tex-math></inline-formula> over the most related work. The implementation results confirm that our design methodology has advantages to deploy many NTT accelerators on an FPGA device for practical HE-based applications.
Blockchain-Based Multisignature Lock for UAC in Metaverse

As an emerging digital concept offering interconnections across multiple platforms, the metaverse provides digital transformations for various aspects of the physical world, facilitated by a few novel technologies, for example, cloud computing offers data support for the digital world. Humans immersed in the metaverse are digital entities who communicate with others or objects, such that ubiquitous access controls (UACs) are indispensable sectors for multiple platforms. However, in the metaverse, UACs have opened a wide scope of bridges for individuals to shuttle the virtual world, which implies that numerous threats exist at the access layer due to a great pool of entries. In this paper, to solve security issues in the UAC setting of the metaverse, we propose a novel blockchain-based multisignature lock for UAC (BMSL-UAC) scheme. All data institutions reconstruct a consortium blockchain system. In addition, our proposed scheme ensures that only authorized users can access an institution’s data. Finally, we abstract the user’s data access behaviors into the transaction information of the consortium blockchain system to realize full life-cycle data management and traceability. To verify the performance of our scheme, a series of experiments are carried out on the Hyperledger, and evaluation results have demonstrated that the resource consumption, delay, and throughput of this scheme are all within a reasonable range.
BNV-Fusion: Dense 3D Reconstruction using Bi-level Neural Volume Fusion

Dense 3D reconstruction from a stream of depth images is the key to many mixed reality and robotic applications. Although methods based on Truncated Signed Distance Function (TSDF) Fusion have advanced the field over the years, the TSDF volume representation is confronted with striking a balance between the robustness to noisy measurements and maintaining the level of detail. We present Bi-level Neural Volume Fusion (BNV-Fusion), which leverages recent advances in neural implicit representations and neural rendering for dense 3D reconstruction. In order to incrementally integrate new depth maps into a global neural implicit representation, we propose a novel bi-level fusion strategy that considers both efficiency and reconstruction quality by design. We evaluate the proposed method on multiple datasets quantitatively and qualitatively, demonstrating a significant improvement over existing methods.
NeuroLens: Augmented Reality-based Contextual Guidance through Surgical Tool Tracking in Neurosurgery

External ventricular drain (EVD) is a common, yet challenging neurosurgical procedure of placing a catheter into the brain ventricular system that requires prolonged training for surgeons to improve the catheter placement accuracy. In this paper, we introduce NeuroLens, an Augmented Reality (AR) system that provides neurosurgeons with guidance that aides them in completing an EVD catheter placement. NeuroLens builds on prior work in AR-assisted EVD to present a registered hologram of a patient’s ventricles to the surgeons, and uniquely incorporates guidance on the EVD catheter’s trajectory, angle of insertion, and distance to the target. The guidance is enabled by tracking the EVD catheter. We evaluate NeuroLens via a study with 33 medical students, in which we analyzed students’ EVD catheter insertion accuracy and completion time, eye gaze patterns, and qualitative responses. Our study, in which NeuroLens was used to aid students in inserting an EVD catheter into a realistic phantom model of a human head, demonstrated the potential of NeuroLens as a tool that will aid and educate novice neurosurgeons. On average, the use of NeuroLens improved the EVD placement accuracy of year 1 students by 39.4% and of the year 2–4 students by 45.7%. Furthermore, students who focused more on NeuroLens-provided contextual guidance achieved better results.
New CNN-Based Predictor for Reversible Data Hiding

In this letter, we propose a convolutional neural network (CNN) based predictor for reversible data hiding (RDH). Firstly, a new image division strategy is presented, which can divide the cover image into four independent parts. Via using it, any pixel in each part can be predicted by all its 8-neighbor pixels to generate the preprocessed images. Then, the preprocessed image is fed into a carefully designed CNN-based prediction model to output the predicted image, which is used to build the prediction-error histogram for RDH. Experimental results demonstrate that a sharply distributed prediction-error histogram (i.e., small prediction errors) can be easily obtained by our proposed CNN-based predictor. Furthermore, combining with the classical prediction-error expansion (PEE) embedding strategy, a series of new RDH algorithms with higher visual quality can be formed in contrast to the state-of-the-art RDH schemes.
A Smart IoT Enabled End-to-End 3D Object Detection System for Autonomous Vehicles

Integration of advanced signal processing, image processing, deep learning, edge computing, and the Internet of Things (IoT) into vehicles allows intelligent automated vehicles to navigate autonomously in different environments. It is crucial for reliable and safe driving that an autonomous vehicle can accurately, effectively, and efficiently recognize, perceive, and observe the surrounding environments. Autonomous vehicles comprise advanced sensor technologies such as RGB cameras and LiDaR that produce an extensive data set in the form of RGB images and 3D measurement points, also recognized as a point cloud. It is necessary to understand and interpret collected data information efficiently and to identify other road users, such as pedestrians and vehicles. Thus, we introduced a smart IoT-enabled deep learning based end-to-end 3D object detection system that works in real-time, emphasizing autonomous driving situations. The detection model is based on YOLOv3; firstly, the model is utilized for 2D object detection and then modified for 3D object detection purposes. The presented model uses point cloud, and RGB image data as input and outputs detected bounding boxes with confidence scores and class labels. Experiments are carried out on the Lyft data set; results reveal that the YOLOv3 model achieves high accuracy and outperforms from other state-of-the-art detection models in terms of effectiveness and accuracy. The overall accuracy of the model is 96% and 97% for 2D and 3D object detection, respectively.
Controllable Dynamic Multi-Task Architectures

Multi-task learning commonly encounters competition for resources among tasks, specifically when model capac-ity is limited. This challenge motivates models which al-low control over the relative importance of tasks and total compute cost during inference time. In this work, we pro-pose such a controllable multi-task network that dynami-cally adjusts its architecture and weights to match the de-sired task preference as well as the resource constraints. In contrast to the existing dynamic multi-task approaches that adjust only the weights within a fixed architecture, our approach affords the flexibility to dynamically control the total computational cost and match the user-preferred task importance better. We propose a disentangled training of two hype rnetwo rks, by exploiting task affinity and a novel branching regularized loss, to take input prefer-ences and accordingly predict tree-structured models with adapted weights. Experiments on three multi-task bench-marks, namely PASCAL-Context, NYU-v2, and CIFAR-100, show the efficacy of our approach. Project page is available at https://www.nec-labs.com/-mas/DYMU.
Synthetic Object Recognition Dataset for Industries

Smart robots in factories highly depend on Computer Vision (CV) tasks, e.g. object detection and recognition, to perceive their surroundings and react accordingly. These CV tasks can be performed after training deep learning (DL) models on large annotated datasets. In an industrial setting, acquiring and annotating such datasets is challenging because it is time-consuming, prone to human error, and limited by several privacy and security regulations. In this study, we propose a synthetic industrial dataset for object detection purposes created using NVIDIA Omniverse. The dataset consists of S industrial assets in 32 scenarios and 200,000 photo-realistic rendered images that are annotated with accurate bounding boxes. For evaluation purposes, multiple object detectors were trained with synthetic data to infer on real images captured inside a factory. Accuracy values higher than 50% and up to 100% were reported for most of the considered assets.
GIFM: An Image Restoration Method With Generalized Image Formation Model for Poor Visible Conditions

Recently, image restoration has attracted considerable attention from researchers, and these methods generally restore degraded images based on the atmospheric scattering model (ATSM) and retinex model (RM). The two models only take into the single attenuation process during imaging, thereby introducing undesirable results. To deal with this issue, we propose an image restoration method based on a generalized image formation model (GIFM). First, unlike the existing image restoration methods, we rebuild a novel image formation model, which describes the light attenuation process that includes the light source-scene path and scene-sensor path. Second, we construct an objective optimization function to decompose a degraded image into a color distorted component and color corrected component, and an augmented Lagrange multiplier-based alternating direction minimization algorithm is provided to solve the optimization problem. Finally, we fully consider the advantages of the small-scale neighborhood and large-scale neighborhood in image restoration, and an image itself brightness-based weighted fusion strategy is proposed to balance brightness enhancement and contrast improvement. Extensive experiments on three image enhancement datasets show that our GIFM achieves better results than state-of-the-art methods. Experiments further suggest that our GIFM performs well for image restoration of extreme scenes, keypoint detection, object detection, and image segmentation.
M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining

Despite the potential of multi-modal pre-training to learn highly discriminative feature representations from complementary data modalities, current progress is being slowed by the lack of large-scale modality-diverse datasets. By leveraging the natural suitability of E-commerce, where different modalities capture complementary semantic information, we contribute a large-scale multi-modal pretraining dataset M5Product. The dataset comprises 5 modalities (image, text, table, video, and audio), covers over 6,000 categories and 5,000 attributes, and is 500× larger than the largest publicly available dataset with a similar number of modalities. Furthermore, M5Product contains incomplete modality pairs and noise while also having a long-tailed distribution, resembling most real-world problems. We further propose Self-harmonized ContrAstive LEarning (SCALE), a novel pretraining framework that integrates the different modalities into a unified model through an adaptive feature fusion mechanism, where the importance of each modality is learned directly from the modality embeddings and impacts the inter-modality contrastive learning and masked tasks within a multi-modal transformer model. We evaluate the current multi-modal pre-training state-of-the-art approaches and benchmark their ability to learn from unlabeled data when faced with the large number of modalities in the M5Product dataset. We conduct extensive experiments on four downstream tasks and demonstrate the superiority of our SCALE model, providing insights into the importance of dataset scale and diversity. Dataset and codes are available at <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> https://xiaodongsuper.github.io/M5Product_dataset/.