Reviewerly/exHarmony huggingface dataset:
Highly efficient secure linear algebra for private machine learning classifications over malicious clients in the post-quantum world

Over the last decade there has a been widespread usage of Machine Learning (ML) classifiers in cases such accurate disease diagnosis at clinics, credit card fraud detection in banks, cyber-attacks prevention of computer systems in different industries, etc. However, privacy and security concerns and law regulations have been an obstacle to the usage of ML classifiers. To this end, this paper addresses the scenario where a server has a private trained ML model, and one or more clients have private queries that they wish to classify using the server's model. During the process, the server learns nothing, while the clients learn only their final classifications and nothing else. Several ML classification algorithms, such as Deep Neural Networks, Support Vector Machines, Logistic Regression, different flavors of Naïve Bayes, etc., can be expressed in terms of linear algebra operations. To this end, initially, as building blocks, several novel secure linear algebra operations are proposed. On top of them novel secure ML classification algorithms are proposed for the aforementioned classifiers under strict security, privacy and efficiency constraints and their security is proven under the semi-honest model. Since the used underlying cryptographic primitives are shown to be resilient to quantum computer attacks, the proposed algorithms are also suitable for the post-quantum world. Furthermore, the proposed algorithms are non-interactive and, based on where the bulk of the operations are done, they have the flexibility to be server or client centric. Theoretical analysis and extensive experimental evaluations over benchmark datasets show that the proposed secure linear algebra operations, hence the secure ML algorithms build on top of them, outperform the state-of-the-art schemes in terms of computation and communication costs as well as on security and privacy characteristics. Moreover, and to the best of the authors’ knowledge, for the first time in literature the security of the proposed algorithms is proven when dealing with multiple malicious clients during classifications.
AutoSurg-Research and Implementation of Automatic Target Resection Key Technologies Via Toumai Surgical Robot System

Targeted resection is a surgical operation that occurs at a high frequency in laparoscopic surgery. By automating such surgical tasks, the quality of surgery can be improved and the cognitive burden of doctors can be reduced. This study is based on the Toumai surgical robot system, which locates the target position and instrument position in real time by visual key point localization algorithm and depth localization algorithm, designs a state machine applicable to the automatic resection operation, and plans the motion trajectory of the robotic arm end effector in real time to finally complete the resection of the target point. The experimental results show that this automatic surgical robot system is accurate and efficient.
SketchScene: Scene Sketch To Image Generation With Diffusion Models

Sketch is an abstract visual representation that can be recovered as natural photographs in the human mind. Many researchers are drawn to work on translating abstract sketches to natural photographs. Since conventional sketch-to-image models are designed to generate images with a single object as the subject, generating scene image with multiple classes of objects is a tricky problem. To tackle this challenge, we propose the first scene sketch-to-image generation method based on diffusion models. Our model uses an encoder to summarize the contour and class features of the scene sketch into a latent variable, and a decoder to reconstruct scene images from it. In scene sketch-to-image generation tasks, our method outperforms the state-of-the-art methods. Experiments also show that our model beats other methods in zero-shot general sketch-to-image generation. It demonstrates our model’s potential for full-domain image generation.
Hierarchical and Contrastive Representation Learning for Knowledge-Aware Recommendation

Incorporating knowledge graph into recommendation is an effective way to alleviate data sparsity. Most existing knowledge-aware methods usually perform recursive embedding propagation by enumerating graph neighbors. However, the number of nodes’ neighbors grows exponentially as the hop number increases, forcing the nodes to be aware of vast neighbors under this recursive propagation for distilling the high-order semantic relatedness. This may induce more harmful noise than useful information into recommendation, leading the learned node representations to be indistinguishable from each other, that is, the well-known over-smoothing issue. To relieve this issue, we propose a Hierarchical and CONtrastive representation learning framework for knowledge-aware recommendation named HiCON. Specifically, for avoiding the exponential expansion of neighbors, we propose a hierarchical message aggregation mechanism to interact separately with low-order neighbors and meta-path-constrained high-order neighbors. Moreover, we also perform cross-order contrastive learning to enforce the representations to be more discriminative. Extensive experiments on three datasets show the remarkable superiority of HiCON over state-of-the-art approaches. The code is available now <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Semi-Supervised Top-k Feature Selection with a General Optimization Framework

Feature selection is widely used in multimedia applications to determine informative features from high-dimensional data. Due to the explosive growth of the data size and the expensive cost of obtaining labeled data, it is increasingly demanded to utilize both labeled and unlabeled data for feature selection. In this paper, we introduce the l2,0-norm in semi-supervised feature selection, which is able to select exact k informative features. Due to the non-convexity of l2,0-norm, we further devise an efficient coordinate-descent-based algorithm to solve the l2,0-norm constraint, which facilitates the application of l2,0-norm to more complex applications, including but not limited to the proposed model in this study. We experimentally verify the effectiveness of the proposed l2,0-norm-based semi-supervised method and the efficiency of the proposed optimization algorithm.
Dual Episodic Sampling and Momentum Consistency Regularization for Unsupervised Few-shot Learning

Unsupervised Few-shot Learning (UFSL) is a practical approach to adapting knowledge learned from unlabeled data of base classes to novel classes with limited labeled data. Nevertheless, most existing UFSL methods may not learn generalizable features in latter training epochs due to the simplicity of meta-learning tasks constructed by data augmentation. To address this issue, we propose two novel components, namely Dual Episodic Sampling (DES) and Momentum Consistency Regularization (MCR) for UFSL. In the DES, two types of sampling strategies are used to construct harder training tasks with multiple augmentations to generate each pseudo-class of increased diversity. The MCR constrains the consistency of the backbone encoder with its momentum counterpart to learn better generalized features for novel classes. Experimental results on four datasets verify the superiority of our method for unsupervised few-shot image classification.
Exploring the Influence of View and Camera Path Selection for Dynamic Mesh Quality Assessment

With the development of 3D mesh processing and applications, the quality assessment of dynamic mesh sequences attracts more and more attention. One prevalent strategy for performing the subjective experiment and designing objective quality metrics is to convert the 3D dynamic mesh into 2D images or videos via projection and to collect subjective scores or calculate objective indexes based on these images or videos. In this paper, we study the influence of the view, or camera path selection for the projection, for both subjective and objective dynamic mesh quality assessment, and compare the performance of image-based metrics and point-based metrics corresponding to the collected subjective scores. First, we use the dynamic mesh sequences proposed by MPEG as anchors and generate videos corresponding to different coding configurations and different camera paths. Then, we conduct subjective experiments to collect the ground truth of mean opinion scores. Besides, we calculate the state-of-the-art objective metric scores for each sequence. We analyze the differences between subjective scores with respect to different camera paths and the correlation between subjective scores and objective metrics. The results show that different camera paths tend to generate close subjective perceptions and that the selection of views can influence some objective metrics.
OAFormer: Occlusion Aware Transformer for Camouflaged Object Detection

Camouflaged object usually have a similar appearance or color to their surrounding environment, so it’s difficult to be detected, especially in heavily obscured situations. To deal with this challenge, in this paper, we propose a novel occlusion aware transformer network (OAFormer) to accurately identify the occluded camouflaged object. In OAFormer, a hierarchical location guidance module (HLGM) is designed to locate the potential locations of camouflaged objects. Then, in order to perceive the structural consistency of the occluded object, we design a neighborhood searching module (NSM) to focus on local pixel details of concealed objects. Besides, for each NSM, we take advantages of transformer blocks to capture long-distance dependencies. So our model can easily capture the complete camouflaged object. In the end, we utilize the auxiliary supervision strategy to promote the learning ability of our model. Compared with other state-of-the-art methods, the proposed OAFormer achieves higher accuracy on four challenging datasets. Code and models are available at: https://github.com/xinyang920/OAFormer.
MA-NeRF: Motion-Assisted Neural Radiance Fields for Face Synthesis from Sparse Images

We address the problem of photorealistic 3D face avatar synthesis from sparse images. Existing Parametric models for face avatar reconstruction struggle to generate details that originate from inputs. Meanwhile, although current NeRF-based avatar methods provide promising results for novel view synthesis, they fail to generalize well for unseen expressions. We improve from NeRF and propose a novel framework that, by leveraging the parametric 3DMM models, can reconstruct a high-fidelity drivable face avatar and successfully handle the unseen expressions. At the core of our implementation are structured displacement feature and semantic-aware learning module. Our structured displacement feature will introduce the motion prior as an additional constraints and help perform better for unseen expressions, by constructing displacement volume. Besides, the semantic-aware learning incorporates multi-level prior, e.g., semantic embedding, learnable latent code, to lift the performance to a higher level. Thorough experiments have been doen both quantitatively and qualitatively to demonstrate the design of our framework, and our method achieves much better results than the current state-of-the-arts.
Generalized Compressed Video Restoration by Multi-Scale Temporal Fusion and Hierarchical Quality Score Estimation

Learning-based methods have achieved excellent performance for compressed video restoration (CVR) in recent years. However, existing networks aggregate multi-frame information inefficiently and are usually developed for specific quantization parameters (QPs), which are not convenient for practical usage. Moreover, current works only consider compressed video restoration in Constant QP (CQP) setting, but do not discuss the performance of the model in more realistic scenarios, e.g., Constant Rate Factor (CRF) and Constant Bitrate (CBR). In this paper, we propose a generalized quality-aware compressed video restoration network, namely QCRN. Specifically, to achieve multi-frame aggregation efficiently, we propose a multi-scale deformable temporal fusion. Meanwhile, QCRN decouples the global quality and local quality representations from input via the hierarchical quality score estimator, and then employs them to adjust the feature enhancement. Extensive experiments on compressed videos in various settings demonstrate that our proposed QCRN achieves favorable performance against state-of-the-art methods in terms of both quantitative metrics and visual quality.
Multi-object Video Generation from Single Frame Layouts

In this paper, we study video synthesis with emphasis on simplifying the generation conditions. Most existing video synthesis models or datasets are designed to address complex motions of a single object, lacking the ability of comprehensively understanding the spatio-temporal relationships among multiple objects. Besides, current methods are usually conditioned on intricate annotations (e.g. video segmentations) to generate new videos, being fundamentally less practical. These motivate us to generate multi-object videos conditioning exclusively on object layouts from a single frame. To solve above challenges and inspired by recent research on image generation from layouts, we have proposed a novel video generative framework capable of synthesizing global scenes with local objects, via implicit neural representations and layout motion self-inference. Our framework is a non-trivial adaptation from image generation methods, and is new to this field. In addition, our model has been evaluated on two widely-used video recognition benchmarks, demonstrating effectiveness compared to the baseline model.
Effective Occlusion Suppression Network via Grouped Pose Estimation for Occluded Person Re-Identification

The occluded target person often lead to incorrect matching results, so eliminating the interference of background clutter is a key matter. To deal with the challenging occluded person re-identification (Re-ID) tasks, this paper proposes a new grouped pose estimation occlusion generation (GPEOG) network to fully use the person grouped keypoint information and the stable non-occluded features, and make the extracted features more robust and discriminative. The local branch uses the proposed Dynamic Adaptive module (DAM) to automatically adjust the patch size during training process and provides adaptive local features. The occlusion-suppression generation branch mainly includes the proposed effective occlusion suppression (EOS) module. It judges whether the person’s keypoint groups are occluded, and then generates the masks of the corresponding parts to suppress occlusion. Extensive experiments show that the proposed method has obvious advantages compared with several state-of-the-art methods.
Meta-ILF: In-Loop Filter with Customized Weights For VVC Intra Coding

In-Loop filter (ILF) is an essential module in video coding for suppressing compression artifacts and thus improving the quality of reconstructed images. As the state-of-the-art video coding standard, Versatile Video Coding (H.266/VVC) employs three in-loop filters, including deblocking filter, sample adaptive offset, and adaptive loop filter. Recently, many neural network-based in-loop filters have been proposed and shown great success in image restoration. In this paper, we propose a meta-learning-based method called Meta-ILF, which performs filtering with customized weights to enhance the quality of VVC intra-coded images. Meta-ILF consists of a meta-network and a filter network. For each reconstructed image block, the meta-network generates the customized weights first. Then, the filter network uses the customized weights to infer the enhanced reconstruction. By dynamically customizing the network weights for each reconstructed block, meta-ILF can better cope with the diverse compression artifacts. To test the performance, Meta-ILF is integrated into VVC reference software VTM-11.0. The experimental results demonstrate that Meta-ILF can reach an average of 6.77% Bjøntegaard Delta rate (BD-rate) improvement over VVC with all intra configuration.
DR-Net: A Multi-View Face Synthesis Network Driven by Dual Representation

With the rapid growth of virtual human techniques and the popularity of 3D face modeling, multi-view face synthesis has received considerable attention in the computer vision community. Though learning-based methods have achieved photo-realistic effect, the lack of geometry guidance remains a great hurdle to their results in high fidelity of identity. To relieve it, we propose a novel network named Dual-Representation-Net (DR-Net) driven by both learning and geometry information. In addition, we utilize face parsing to lead the learning of regional attention, in this way, the two synthesizing representations can collaborate and compete dynamically under different face angles, which improves the stability of face synthesis by assimilating the benefits of both kinds of information. What is more, we want to emphasize that DR-Net can serve as a general framework for multi-view face synthesis by jointly exploiting learning and geometry information. That is, our geometry module and learning module are actually substitutable, which can be replaced by any other SOTA methods. This makes our method both flexible and scalable. Extensive experiments verify that our approach is superior to other counterparts in both synthesis quality and identity preservation. Moreover, with the geometric information, we can decouple the synthesis of face and background to obtain more robust results on in-the-wild data.
Understanding and Improving Perceptual Quality of Volumetric Video Streaming

Volumetric video is fully three-dimensional and provides users with highly immersive and interactive experience. However, it is difficult to stream volumetric video over the Internet due to sheer video size and limited network bandwidth. Existing solutions suffered from poor perceptual quality and low coding efficiency. In this paper, we first conduct a comprehensive user study to understand the effectiveness of popular perceptual quality metrics for volumetric video. It is observed that those metrics cannot well capture the impact of user viewing behaviors. Considering the findings that users are more sensitive to the distortion of 2D image rendered from 3D point cloud, a new metric called Volu-FMAF is proposed to better represent perceptual quality of volumetric video. Next, we propose a novel neural-based volumetric video streaming framework RenderVolu and design a distortion-aware rendered image super-resolution network, called RenDA-Net, to further improve user perceptual quality. Last, we conduct extensive experiments with real datasets to validate our proposed method, and the results show that our method can boost the perceptual quality of volumetric video by 171% to 190%, and achieves a speedup of 108x in terms of decoding efficiency compared to the state-of-the-art approaches.
Edge-Aware Mirror Network for Camouflaged Object Detection

Existing edge-aware camouflaged object detection (COD) methods normally output the edge prediction in the early stage. However, edges are important and fundamental factors in the following segmentation task. Due to the high visual similarity between camouflaged targets and the surroundings, edge prior predicted in early stage usually introduces erroneous foreground-background and contaminates features for segmentation. To tackle this problem, we propose a novel Edge-aware Mirror Network (EAMNet), which models edge detection and camouflaged object segmentation as a cross refinement process. More specifically, EAMNet has a two-branch architecture, where a segmentation-induced edge aggregation module and an edge- induced integrity aggregation module are designed to cross-guide the segmentation branch and edge detection branch. A guided-residual channel attention module which leverages the residual connection and gated convolution finally better extracts structural details from low-level features. Quantitative and qualitative experiment results show that EAMNet outperforms existing cutting-edge baselines on three widely used COD datasets. Codes are available at https://github.com/sdy1999/EAMNet.
Multi-Level Memory-Augmented Appearance-Motion Correspondence Framework for Video Anomaly Detection

Frame prediction based on AutoEncoder plays a significant role in unsupervised video anomaly detection. Ideally, the models trained on the normal data could generate larger prediction errors of anomalies. However, the correlation between appearance and motion information is underutilized, which makes the models lack an understanding of normal patterns. Moreover, the models do not work well due to the uncontrollable generalizability of deep AutoEncoder. To tackle these problems, we propose a multi-level memory-augmented appearance-motion correspondence framework. The latent correspondence between appearance and motion is explored via appearance-motion semantics alignment and semantics replacement training. Besides, we also introduce a Memory-Guided Suppression Module, which utilizes the difference from normal prototype features to suppress the reconstruction capacity caused by skip-connection, achieving the tradeoff between the good reconstruction of normal data and the poor reconstruction of abnormal data. Experimental results show that our framework outperforms the state-of-the-art methods, achieving AUCs of 99.6%, 93.8%, and 76.3% on UCSD Ped2, CUHK Avenue, and ShanghaiTech datasets.
Attention-Aware Anime Line Drawing Colorization

Automatic colorization of anime line drawing has attracted much attention in recent years since it can substantially benefit the animation industry. User-hint based methods are the mainstream approach for line drawing colorization, while reference-based methods offer a more intuitive approach. Nevertheless, although reference-based methods can improve feature aggregation of the reference image and the line drawing, the colorization results are not compelling in terms of color consistency or semantic correspondence. In this paper, we introduce an attention-based model for anime line drawing colorization, in which a channel-wise and spatial-wise Convolutional Attention module is used to improve the ability of the encoder for feature extraction and key area perception, and a Stop-Gradient Attention module with cross-attention and self-attention is used to tackle the cross-domain long-range dependency problem. Extensive experiments show that our method outperforms other SOTA methods, with more accurate line structure and semantic color information.
FreConv: Frequency Branch-and-Integration Convolutional Networks

Recent researches indicate that utilizing the frequency information of input data can enhance the performance of networks. However, the existing popular convolutional structure is not designed specifically for utilizing the frequency information contained in datasets. In this paper, we propose a novel and effective module, named FreConv (frequency branch-and-integration convolution), to replace the vanilla convolution. FreConv adopts a dual-branch architecture to extract and integrate high- and low-frequency information. In the high-frequency branch, a derivative-filter-like architecture is designed to extract the high-frequency information while a light extractor is employed in the low-frequency branch because the low-frequency information is usually redundant. FreConv is able to exploit the frequency information of input data in a more reasonable way to enhance feature representation ability and reduce the memory and computational cost significantly. Without any bells and whistles, experimental results on various tasks demonstrate that FreConv-equipped networks consistently outperform state-of-the-art baselines.