Reviewerly/exHarmony huggingface dataset:
Augmented Normalizing Flow for Point Cloud Geometry Coding

With the increased popularity of immersive media, point clouds have become one of the popular data representations for presenting 3D scenes. The huge amount of point cloud data poses a great challenge on their storage and real-time transmission, which calls for efficient point cloud compression. This paper presents a novel point cloud geometry compression technique based on learning end-to-end an augmented normalizing flow (ANF) model to represent the occupancy status of voxelized data points. The higher expressive power of ANF than variational autoencoders (V AE) is leveraged for the first time to represent binary occupancy status. Compared to two coding standards developed by MPEG, namely G-PCC (geometry-based point cloud compression) and V-PCC (video-based point cloud compression), our method achieves more than 80% and 30% bitrate reduction, respectively. Compared to several learning-based methods, our method also yields better performance.
Cross-Layer Feature based Multi-Granularity Visual Classification

In contrast to traditional fine-grained visual clas-sification, multi-granularity visual classification is no longer limited to identifying the different sub-classes belonging to the same super-class (e.g., bird species, cars, and aircraft models). Instead, it gives a sequence of labels from coarse to fine (e.g., Passeriformes → Corvidae → Fish Crow), which is more convenient in practice. The key to solving this task is how to use the relationships between the different levels of labels to learn feature representations that contain different levels of granularity. Interestingly, the feature pyramid structure naturally implies different granularity of feature representation, with the shallow layers representing coarse-grained features and the deep layers representing fine-grained features. Therefore, in this paper, we exploit this property of the feature pyramid structure to decouple features and obtain feature representations corre-sponding to different granularities. Specifically, we use shallow features for coarse-grained classification and deep features for fine-grained classification. In addition, to enable fine-grained features to enhance the coarse-grained classification, we propose a feature reinforcement module based on the feature pyramid structure, where deep features are first upsampled and then combined with shallow features to make decisions. Experimental results on three widely used fine-grained image classification datasets such as CUB-200-2011, Stanford Cars, and FGVC-Aircraft validate the method's effectiveness. Code available at https://github.com/PRIS-CV/CGVC.
Recurrent Network with Enhanced Alignment and Attention-Guided Aggregation for Compressed Video Quality Enhancement

Recently, various compressed video quality enhancement technologies have been proposed to overcome the visual artifacts. Most existing methods are based on optical flow or deformable alignment to explore the spatiotemporal information across frames. However, inaccurate motion estimation and training instability of deformable convolution would be detrimental to the reconstruction performance. In this paper, we design a bi-directional recurrent network equipping with enhanced deformable alignment and attention-guided aggregation to promote information flows among frames. For the alignment, a pair of scale and shift parameters are learned to modulate optical flows into new offsets for deformable convolution. Furthermore, an attention aggregation strategy oriented at preference is designed for temporal information fusion. The strategy synthesizes global information of inputs to modulate features for effective fusion. Extensive experiments have proved that the proposed method achieves great performance in terms of quantitative performance and qualitative effect.
Learned Lossless JPEG Transcoding via Joint Lossy and Residual Compression

As a commonly-used image compression format, JPEG has been broadly applied in the transmission and storage of images. To further reduce the compression cost while maintaining the quality of JPEG images, lossless transcoding technology has been proposed to recompress the compressed JPEG image in the DCT domain. Previous works, on the other hand, typically reduce the redundancy of DCT coefficients and optimize the probability prediction of entropy coding in a hand-crafted manner that lacks generalization ability and flexibility. To tackle the above challenge, we propose the learned lossless JPEG transcoding framework via Joint Lossy and Residual Compression. Instead of directly optimizing the entropy estimation, we focus on the redundancy that exists in the DCT coefficients. To the best of our knowledge, we are the first to utilize the learned end-to-end lossy transform coding to reduce the redundancy of DCT coefficients in a compact representational domain. We also introduce residual compression for lossless transcoding, which adaptively learns the distribution of residual DCT coefficients before compressing them using context-based entropy coding. Our proposed transcoding architecture shows significant superiority in the compression of JPEG images thanks to the collaboration of learned lossy transform coding and residual entropy coding. Extensive experi-ments on multiple datasets have demonstrated that our proposed framework can achieve about 21.49% bits saving in average based on JPEG compression, which outperforms the typical lossless transcoding framework JPEG-XL by 3.51%.
An Efficient Content-aware Downsampling-based Video Compression Framework

Recently, deep learning-based video compression algorithms have achieved competitive performance in Bjøntegaard delta (BD) rate, especially those adopting super-resolution networks as post-processing modules in downsampling-based video compression (DBC) frameworks. However, limited by the non-differentiable characteristics of traditional codecs, DBC frameworks mainly focus on improving the performance of super-resolution modules while ignoring optimizing downscaling modules. It is crucial to improve video compression performance without introducing additional modifications to the decoder client in practical application scenarios. We propose a context-aware processing network (CPN) compatible with standard codecs with no computational burden introduced to the client, which preserves the critical information and essential structures during downscaling. The proposed CPN works as a precoder cascaded by standard codecs to improve the compression performance on the server before encoding and transmission. Besides, a surrogate codec is employed to simulate the degradation process of the standard codecs and backpropagate the gradient to optimize the CPN. Experimental results show that the proposed method outperforms latest pre-processing networks and achieves considerable performance compared with the latest DBC frameworks.
Spectral Analysis of Aerial Light Field for Optimization Sampling and Rendering of Unmanned Aerial Vehicle

The aerial light field (ALF) can render higher quality images of large-scale 3D scenes. In this paper, we apply the ALF technology to study the image captured and novel view rendering of unmanned aerial vehicle (UAV), which exists some problems, such as large scene and depth of field. First, we design an ALF sampling model using spectral analysis of Fourier theory. Based on the ALF sampling model, the exact expression of ALF spectrum is derived. By the spectral support of ALF, we analyze the influence of pitch angle on light field sampling and its bandwidth. Particularly, the bandwidth of ALF can be applied to determine the minimum sampling rate for UAV. Additionally, we design a reconstruction filter that is related to pitch angle to render novel views of UAV. Finally, our experiments show that our sampling and rendering methods can improve the rendering quality of UAV novel view rendering.
Clothing Retrieval from Channel Attention to KN Loss Learning

Due to the high diversity of clothing types and names, using text to search may not find the desired clothes. Clothing retrieval via images can help users find their favorite clothes more conveniently. So, it is widely used in many online stores and platforms, and gradually replaces text search. Clothing retrieval performance is often affected due to factors such as shooting angle, occlusion, and lighting. Therefore, we propose a low-parameter clothing retrieval model without using any additional annotations or labels to aid learning. In the model, we use the classification capability of the Capsule Network with multiple residual convolution modules to learn features, and the VGG16 network is used as a clothing feature extractor to help the model learn features such as clothing color and texture. We add different attention mechanisms to the model for feature enhancement and design a new embedding method. We also adjust the loss function to sample multiple negative classes to improve the similarity clustering effect, and add weights to strengthen it. From the experimental results, one can find that our proposed method achieves better retrieval accuracy than several state-of-the-art models.
Application of Named Entity Recognition by Self-Attention BiLSTM-CRF to Chinese Prescription Document

Reporting the contents of the prescription is a time-consuming and labor-intensive task for pharmacists, and it is easy to make mistakes in this kind of work that requires repeated confirmation under the time pressure of customers. Because medicines usually have an impact on the human body, if the wrong medicine is accidentally provided to the customer, it may even lead to death. Therefore, in our work, we train a bidirectional LSTM-CRF network combined with an attention mechanism to perform the NER task on prescription notes and extract key prescription declaration information. The semi-automatic labeling module was implemented using a rule-based approach, and a total of 636 different prescription notes were trained and tested, including hospitals, clinics, and medical centers. The identification accuracy rate is more than 90 % in each field of declaration information. Provides a UI interface for subsequent identification to facilitate pharmacists to modify the identification content and declare key information.
Design and Implementation of Campus Pedestrian Detection based on Unmanned Robot “Sweeper”

Based on the unmanned autonomous sanitation robot “Sweeper” introduced by Shanghai Sanda University, the target detection technology is studied by adopting the idea of blob analysis to improve the target detection ability and ground cleaning function of “Sweeper” with more than 300 images of campus pedestrians. After HSV thresholding binarization, image morphology processing and image contour detection fitting, erosion operation with expansion operation and different rectangular structured element adjustment, the noise reduction of the surrounding environment and the extraction of campus pedestrian contours were achieved and campus pedestrian image detection is obtained. The test shows that the system can achieve 100% success rate for single pedestrian detection in any environment, and 65%-75% for multiple pedestrian images, and the detection success rate of stable natural environment will be better than that of complex natural environment.
Dual Stream Variational Autoencoder for Video Anomaly Detection in Single Scene Videos

Video anomaly detection (VAD) is a subject in computer vision that has a variety of applications such as civil protection, surveillance systems, intrusion detection, etc. Anomaly detection remains an ill-defined problem, despite its diverse applications. The proposed model for VAD is a Dual-Stream Variational Auto-Encoder (DSVAE), which consists of two stacked Variational Auto-Encoders (VAE) models. One model is a shallow generative model, i.e., Fully connected VAE (FCVAE) and the other is Skip connected VAE (SCVAE). The FCVAE model tries to learn the overall features of the model and rejects some of the unwanted features. The SCVAE attempts to extract the spatial and temporal features of the image frames in detail. SCVAE also uses the skip connection to connect the features from the encoder and decoder to minimize the information loss. The model is trained only on the normal video clips, and it tries to minimize the reconstruction error. Then the trained model is tested on the test videos. The reconstruction error is very high for testing video clips, resulting in a low regularity scores and a high anomaly. A properly set threshold regularity score segregates the anomaly frame from the normal one. Anomaly frames usually have low regularity score than the threshold value. The proposed DSVAE model is trained and tested on one of the widely used publicly available datasets, i.e., UCSD ped2, and the obtained performance results are found to be promising.
Frequency-aware Learned Image Compression for Quality Scalability

Spatial frequency analysis and transforms serve a central role in most engineered image and video lossy codecs, but are rarely employed in neural network (NN)-based approaches. We propose a novel NN-based image coding framework that utilizes forward wavelet transforms to decompose the input signal by spatial frequency. Our encoder generates separate bitstreams for each latent representation of low and high frequencies. This enables our decoder to selectively decode bitstreams in a quality-scalable manner. Hence, the decoder can produce an enhanced image by using an enhancement bitstream in addition to the base bitstream. Furthermore, our method is able to enhance only a specific region of interest (ROI) by using a corresponding part of the enhancement latent representation. Our experiments demonstrate that the proposed method shows competitive rate-distortion performance compared to several non-scalable image codecs. We also showcase the effectiveness of our two-level quality scalability, as well as its practicality in ROI quality enhancement.
CdCLR: Clip-Driven Contrastive Learning for Skeleton-Based Action Recognition

In this study, we propose a Clip-Driven Contrastive Learning for Skeleton-Based Action Recognition (CdCLR). In-stead of considering sequences as instances, CdCLR extracts clips from the sequences as new instances. Aim to implement inherent supervision-guided contrastive learning through joint optimal training of sequences discrimination, clips discrimination, and order verification. Mining abundant positive/negative pairs inside sequence while learning inter-and intra-sequence semantic repre-sentations. Extensive experiments on the NTU RGB+D 60, UCLA and iMiGUE datasets present that CdCLR exhibits superior performance under various evaluation protocols and reaches state-of-the-art. Our code is available at https://github.com/Erich-G/CdCLRI.
MSCI: A Multi-Source Composite Image Database for Compression Distortion Quality Assessment

With the rapid development of multi-sensor fusion technology in various industrial fields, many composite images closely related to human life have been produced. To meet the rapidly growing needs of various image-based applications, we have established the first multi-source composite image (MSCI) database for image quality assessment (IQA). Our MSCI database contains 80 reference images and 1600 distorted images, generated by four advanced compression standards with five distortion levels. In particular, these five distortion levels are determined based on the first five just noticeable difference (JND) levels. Moreover, we verify the IQA performance of some representative methods on our MSCI database. The experimental results show that the performance of the existing methods on the MSCI database needs to be further improved.
HSV Semantic Segmentation on Partially Facility and Phanerophyte SunShine-Shadowing Road

The previous work reported the importance of visual intelligence on Self-Driving Sweeping Bot (SDSB) to reach the pedestrian security and the sweeping efficiency, in which the machine vision on object detection and semantic segmentation in terms of vehicle, pedestrian, rubbishes detection and road segmentation would be further investigated. Subsequentially, this work proposed an optimizing HSV encoding framework embedded with the morphology operation to examine the semantic segmentation on campus sweeping road. Furthermore, considering the complex weather effects caused the road performed with partially sunshine shadowing effects by facilities and giant tree plants, such that led to the inconsistent color presentation in a road. The experimental results indicated that the proposed framework given the promising results to three categorized sunshine-shadowing images groups.
Confidential Ethereum Smart Contracts

As decentralized finance (DeFi) applications take off and pose to revolutionize the world's financial infrastructure, the current public blockchains fall short in financial privacy. That is a significant barrier for traditional financial institutions to enter DeFi. The Oasis Ethereum ParaTime proposes a new privacy-first blockchain network to execute Ethereum smart contracts and trade Ethereum digital assets. It provides full transparency into smart contract code and logic. The smart contract is executed on replicated virtual machines on a decentralized set of validator nodes, and the result is guaranteed to be correct through the consensus of those nodes. Yet, the contract data and state are confidential even to the validators and node operators. The transactions and results are only revealed to users who submit the transactions. In this article, we will explore the design of the Oasis Ethereum ParaTime, and its potential applications. We will also discuss an early testnet we launched on the Oasis Network and its future plans.
Quality Assessment of Screen Content Images Based on Multi-Pathway Convolutional Neural Network

In this paper, considering the retinal structure of human eye, and the composition characteristics of screen content images (SCIs), a multi-pathway convolutional neural network (CNN) with picture-text competition is proposed for SCIs quality assessment. According to the visual mechanism of human retina, we design a retinal structure simulation module, which uses multiple parallel convolution pathways to simulate the parallel transmission of visual signals by bipolar cells and uses a multi-pathway feature fusion (MPFF) module to allocate the weight for each channel to simulate horizontal cells' regulation of the information transmission. In addition, we design an adaptive feature extraction and competition module (AFEC) to directly extract the features of textural and pictorial regions and distribute the weight. Furthermore, the attention module combined with deformable convolution and channel attention can accurately extract image edge features and reduce redundancy of information. Experimental results show that the proposed method is superior to the mainstream methods.
Hybrid Regularizations for Multi-Aspect Category Sentiment Analysis

Aspect level sentiment classification aims to identify the sentiment polarity towards a particular aspect in a sentence. Previous attention-based methods generate an aspect-specific representation for each aspect and employ it to classify the sentiment polarity. However, normalized attention scores scatter over every word in the sentence, resulting in two issues. First, the attention may inherently introduce noise and downgrade the performance. Second, the opinion words may be “diluted” by other words, while the opinion feature should dominate for sentiment analysis. The issues become more severe in multi-aspect sentences. In this paper, we address the above two issues via hybrid regularizations, i.e., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">aspect-level</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">task-level regularizations</i> . Concretely, the aspect-level regularizations constrain the attention weights to alleviate noise. Among them, orthogonal regularization is designed for multi-aspect sentences and sparse regularization is for single-aspect sentences. To extract sentiment-dominant features, task-level regularization is proposed by introducing an orthogonal auxiliary task, i.e., aspect category detection. This regularization can allocate task-oriented context information for specific downstream tasks. Extensive experimental results on three public datasets demonstrate the effectiveness of the proposed approach in both single-task and multi-task scenarios.
Asymmetric Learned Image Compression With Multi-Scale Residual Block, Importance Scaling, and Post-Quantization Filtering

Recently, deep learning-based image compression has made significant progresses, and has achieved better rate-distortion (R-D) performance than the latest traditional method, H.266/VVC, in both MS-SSIM metric and the more challenging PSNR metric. However, a major problem is that the complexities of many leading learned schemes are too high. In this paper, we propose an efficient and effective image coding framework, which achieves similar R-D performance with lower complexity than the state of the art. First, we develop an improved multi-scale residual block (MSRB) that can expand the receptive field and capture global information more efficiently, which further reduces the spatial correlation of the latent representations. Second, an importance scaling network is introduced to directly scale the latents to achieve content-adaptive bit allocation without sending side information, which is more flexible than previous importance map methods. Third, we apply a post-quantization filter (PQF) to reduce the quantization error, motivated by the Sample Adaptive Offset (SAO) filter in video coding. Moreover, our experiments show that the performance of the system is less sensitive to the complexity of the decoder. Therefore, we design an asymmetric paradigm, in which the encoder employs three stages of MSRBs to improve the learning capacity, whereas the decoder only uses one stage of MSRB, which reduces the decoder complexity and still yields satisfactory performance. Experimental results show that compared to the state-of-the-art method, the encoding and decoding time of the proposed method are about 17 times faster, and the R-D performance is only reduced by about 1% on both Kodak and Tecnick-40 datasets, which is still better than H.266/VVC(4:4:4) and other leading learning-based methods. Our source code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/fengyurenpingsheng</uri> .
Learning from the NN-based Compressed Domain with Deep Feature Reconstruction Loss

To speedup the image classification process which conventionally takes the reconstructed images as input, compressed domain methods choose to use the compressed images without decompression as input. Correspondingly, there will be a certain decline about the accuracy. Our goal in this paper is to raise the accuracy of compressed domain classification method using compressed images output by the NN-based image compression networks. Firstly, we design a hybrid objective loss function which contains the reconstruction loss of deep feature map. Secondly, one image reconstruction layer is inte-grated into the image classification network for up-sampling the compressed representation. These methods greatly help increase the compressed domain image classification accuracy and need no extra computational complexity. Experimental results on the benchmark ImageNet prove that our design outperforms the latest work ResNet-41 with a large accuracy gain, about 4.49% on the top-1 classification accuracy. Besides, the accuracy lagging behinds the method using reconstructed images is also reduced to 0.47 %. Moreover, our designed classification network has the lowest computational complexity and model complexity.