Reviewerly/exHarmony huggingface dataset:
One-Stage Visual Grounding via Semantic-Aware Feature Filter

Visual grounding has attracted much attention with the popularity of vision language. Existing one-stage methods are far ahead of two-stage methods in speed. However, these methods fuse the textual feature and visual feature map by simply concatenation, which ignores the textual semantics and limits these models' ability in cross-modal understanding. To overcome this weakness, we propose a semantic-aware framework that utilizes both queries' structured knowledge and context-sensitive representations to filter the visual feature maps to localize the referents more accurately. Our framework contains an entity filter, an attribute filter, and a location filter. These three filters filter the input visual feature map step by step according to each query's aspects respectively. A grounding module further regresses the bounding boxes to localize the referential object. Experiments on various commonly used datasets show that our framework achieves a real-time inference speed and outperforms all state-of-the-art methods.
Weakly-Supervised Temporal Action Localization via Cross-Stream Collaborative Learning

Weakly supervised temporal action localization (WTAL) is a challenging task as only video-level category labels are available during training stage. Without precise temporal annotations, most approaches rely on complementary RGB and optical flow features to predict the start and end frame of each action category in a video. However, existing approaches simply resort to either concatenation or weighted sum to learn how to take advantages of these two modalities for accurate action localization, which ignore the substantial variance between such two modalities. In this paper, we present Cross-Stream Collaborative Learning (CSCL) to address these issues. The proposed CSCL introduce a cross-stream weighting module to identify which modality is more robust during training and take advantage of the robust modality to guide the weaker one. Furthermore, we suppress the snippets which has high action-ness scores in both modalities to further exploiting the complementary property between two modalities. In addition, we bring the concept of co-training for WTAL and take both modalities into account for pseudo label generation to help training a stronger model. Extensive experiments conducted on THUMOS14 and ActivityNet dataset demonstrate that CSCL achieves a favorable performance against state-of-the-arts methods.
Efficient algorithms for victim item selection in privacy-preserving utility mining

High-utility itemset mining has evolved as an essential and captivating research topic. It aims to extract the patterns/itemsets having high utility value; hence, they are called high utility itemsets (HUIs). From a business perspective, a utility can be the benefit associated with the sale of a particular item or the usefulness or satisfaction that a customer experiences from a product. The economic utilities are helpful to evaluate the drivers behind a customer’s purchase decision. The advances in information technology have enabled us to access the datasets related to various domains like health care, stock market, market-basket, education and bioinformatics. Companies strive to increase the utility value of their products and share their customer’s transactions data to extract high utility patterns to achieve global customer insights. However, this can lead to massive security and privacy risk if their competitors misuse the patterns that can disclose their confidential information. Privacy-preserving utility mining (PPUM) is a branch of privacy-preserving data mining (PPDM) that presents various algorithms which intend to hide sensitive high utility itemsets (SHUIs) and maintain a balance between utility-maximizing and privacy-preserving. In this paper, two SHUIs hiding algorithms, MinMax and Weighted, are proposed with three variants of each algorithm. Experiments on various datasets show that proposed algorithms perform better than the existing SHUIs hiding algorithms as fewer distortions of non-sensitive knowledge occur. This study uses six performance evaluating metrics to assess the proposed algorithms against compared algorithms.
From premise to practice of social consensus: How to agree on common knowledge in blockchain-enabled supply chains

Attaining “Common Knowledge” is essential in the formation of deals and agreements, but is hard to achieve over point-to-point communication networks such as the Internet. While blockchain technologies provide a helpful mechanism in the form of technical consensus, originally created to resolve the double-spending problem, little does it address impediments to proper contract formation that have to do with the quality of exogenous data, asymmetry of information, or even its suitability (the colloquial garbage-in/garbage-out problem). In the face of the limitations of technical consensus, we explore how the strengths of technical consensus can be utilized by introducing another dimension of consensus – what we have called social consensus – which directly addresses the formation of true (if retrospective) Common Knowledge essential to arms-length agreement between multiple actors. We have eschewed the notion that social consensus as an epistemological concern, and instead focus on mechanisms by which environments of multiple actors form and sustain bodies of information that enable them to go about their activities with a reasonable level of confidence that others in the supply chain will ‘do their bit,’ backed by a ‘failsafe’ mechanism enforced by cryptography which holds the actors unharmed in case the envisioned finality is not obtained. These bodies of information create what is essentially a retrospective form of common knowledge, and a lesser variant, mutual knowledge. We propose an implementation from the preferred mechanism of a multisig primitive, which builds and improves on the Gnosis multisig, in the use case of decentralized blockchain-powered supply chains consisting of multiple parties, human and machine.
DSANet: Dynamic Segment Aggregation Network for Video-Level Representation Learning

Long-range and short-range temporal modeling are two complementary and crucial aspects of video recognition. Most of the state-of-the-arts focus on short-range spatio-temporal modeling and then average multiple snippet-level predictions to yield the final video-level prediction. Thus, their video-level prediction does not consider spatio-temporal features of how video evolves along the temporal dimension. In this paper, we introduce a novel Dynamic Segment Aggregation (DSA) module to capture relationship among snippets. To be more specific, we attempt to generate a dynamic kernel for a convolutional operation to aggregate long-range temporal information among adjacent snippets adaptively. The DSA module is an efficient plug-and-play module and can be combined with the off-the-shelf clip-based models (i.e., TSM, I3D) to perform powerful long-range modeling with minimal overhead. The final video architecture, coined as DSANet. We conduct extensive experiments on several video recognition benchmarks (i.e., Mini-Kinetics-200, Kinetics-400, Something-Something V1 and ActivityNet) to show its superiority. Our proposed DSA module is shown to benefit various video recognition models significantly. For example, equipped with DSA modules, the top-1 accuracy of I3D ResNet-50 is improved from 74.9% to 78.2% on Kinetics-400. Codes are available at https://github.com/whwu95/DSANet.
Lesion-Inspired Denoising Network: Connecting Medical Image Denoising and Lesion Detection

Deep learning has achieved notable performance in the denoising task of low-quality medical images and the detection task of lesions, respectively. However, existing low-quality medical image denoising approaches are disconnected from the detection task of lesions. Intuitively, the quality of denoised images will influence the lesion detection accuracy that in turn can be used to affect the denoising performance. To this end, we propose a play-and-plug medical image denoising framework, namely Lesion-Inspired Denoising Network (LIDnet), to collaboratively improve both denoising performance and detection accuracy of denoised medical images. Specifically, we propose to insert the feedback of downstream detection task into existing denoising framework by jointly learning a multi-loss objective. Instead of using perceptual loss calculated on the entire feature map, a novel region-of-interest (ROI) perceptual loss induced by the lesion detection task is proposed to further connect these two tasks. To achieve better optimization for overall framework, we propose a customized collaborative training strategy for LIDnet. On consideration of clinical usability and imaging characteristics, three low-dose CT images datasets are used to evaluate the effectiveness of the proposed LIDnet. Experiments show that, by equipping with LIDnet, both of the denoising and lesion detection performance of baseline methods can be significantly improved.
Advantages and Bottlenecks of Quantum Machine Learning for Remote Sensing

This article aims to explore the potential of current approaches for quantum image classification in the context of remote sensing. After a brief outline of quantum computers and an analysis of the current bottlenecks, it shows for the first time experiments with quantum neural networks on a reference Earth observation (EO) dataset: EuroSAT. Moreover, it establishes the proof of concept of quantum computing for EO: the models trained and run on a quantum simulator are on par with classical ones. We make the open-source code available for further developments <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> QNN4EO repository: https://github.com/ESA-PhiLab/QNN4EO..
Learning Forgery Region-Aware and ID-Independent Features for Face Manipulation Detection

Over the past several years, to solve the problem of malicious abuse of facial manipulation technology, face manipulation detection has obtained considerable attention. Although existing works achieved impressive performance on a hold-out test set, their methods suffered a significant performance drop on data from a different distribution than the training set used. In this paper, we conduct an in-depth analysis on existing typical models about poor generalization capability and propose a novel method for face manipulation detection, which can alleviate overfitting and improve the generalization ability by learning forgery region aware and ID-independent features. Specifically, a forgery region guided self-attention module (FR) is introduced to make the model focus on the forgery region and a landmark guided dropout module (LM) is designed to randomly remove features of structured informative regions for destroying identity features. These two regularization modules are then added to the basic classification network, e.g., Xception. With the help of the joint learning framework, both forgery region-aware and ID-independent features are learned for face manipulation detection with better generalization ability. Extensive experiments demonstrate that the proposed method can not only outperform competing state-of-the-art methods on the FaceForensics&#x002B;&#x002B; dataset but also achieve superior generalization performance on the Celeb-DF dataset.
Birds of a Feather Flock Together

RDMA-capable networks are gaining traction with datacenter deployments due to their high throughput, low latency, CPU efficiency, and advanced features, such as remote memory operations. However, efficiently utilizing RDMA capability in a common setting of high fan-in, fan-out asymmetric network topology is challenging. For instance, using RDMA programming features comes at the cost of connection scalability, which does not scale with increasing cluster size. To address that, several works forgo some RDMA features by only focusing on conventional RPC APIs.
Stereo Video Super-Resolution via Exploiting View-Temporal Correlations

Stereo Video Super-Resolution (StereoVSR) aims to generate high-resolution video steams from two low-resolution videos under stereo settings. Existing video super-resolution and stereo image super-resolution techniques can be extended to tackle the StereoVSR task, yet they cannot make full use of the multi-view and temporal information to achieve satisfactory performance. In this paper, we propose a novel Stereo Video Super-Resolution Network (SVSRNet) to fulfill the StereoVSR task via exploiting view-temporal correlations. First, we devise a view-temporal attention module (VTAM) to integrate the information of cross-time-cross-view for constructing high-resolution stereo videos. Second, we propose a spatial-temporal fusion module (STFM), which aggregates the information across time in intra-view to emphasize important features for subsequent restoration. In addition, we design a view-temporal consistency loss function to enforce consistency constraint of superresolved stereo videos. Comprehensive experimental results demonstrate that our method generates superior results.
Secure Outsourced Blockchain-Based Medical Data Sharing System Using Proxy Re-Encryption

The security and privacy of electronic health records (EHRs) have received considerable attention from healthcare workers and researchers. To ensure security, various encryption and decryption schemes as well as key management protocols have been developed. However, owing to sharing and scalability issues, additional security technologies have been proposed. Nonetheless, these technologies cause other problems, such as efficiency issues. Blockchain-based EHR management systems have been proposed to overcome computational overhead. However, because most blockchain systems are installed by outsourcing companies, EHRs may be leaked to the company. Hence, we herein propose a blockchain-based EHR management scheme with proxy re-encryption. In this scheme, we set a proxy server that re-encrypts the ciphertext between file servers, thereby solving EHR sharing issues. Furthermore, because the server is separated from the blockchain system, the outsourcing company cannot manipulate the server or access the records. In addition, the blockchain assists in access control by using smart contracts, thereby enabling secure and efficient EHR sharing. By performing security analysis, we prove that our proposed scheme solves the aforementioned security problems. In addition, we experimentally demonstrate the efficient operation of the proposed system.
Lossy Point Cloud Geometry Compression via Region-Wise Processing

Point cloud geometry (PCG) is used to precisely represent arbitrary-shaped 3D objects and scenes, is of great interest to vast applications which puts forward the pressing desire of high-efficiency PCG compression for transmission and storage. Existing PCG coding mostly relies on the octree model by which point-wise processing is applied without exploring nonlocal regional geometry similarity across the entire 3D surface. This work, instead, suggests the region-wise processing to leverage the region similarity to exploit inter-region redundancy for efficient lossy point cloud geometry compression. Towards this goal, a given PCG is first segmented into numerous local regions each of which comprises a portion of point cloud surface, and can be represented by a surface vector that describes the geometry shape numerically in a projected principal space. Subsequently, these regions are grouped into several discriminative clusters, assuring that inter-cluster similarity is minimized and intra-cluster similarity is maximized simultaneously, where the similarity is calculated using the regional surface vectors. In each cluster, we set a reference region having the largest similarity score to the others, which enables the non-reference region prediction from the reference one using alignment transform. In the end, we encode the reference regions directly using the lossless mode of the Geometry-based Point Cloud Compression (G-PCC), while corresponding non-reference regions are signaled using associated transform parameters. Compared with the state-of-the-art G-PCC using octree model, our region-wise approach can offer remarkable coding efficiency improvement, e.g., 32.4% and 22.0% Bjontegaard-delta rate (BD-Rate) gains for respective point-to-point ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$D1$ </tex-math></inline-formula> ) and point-to-plane ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$D2$ </tex-math></inline-formula> ) distortion evaluations, across a variety of common test sequences used in standard committee.
Probabilistic Human-like Gesture Synthesis from Speech using GRU-based WGAN

Gestures are crucial for increasing the human-likeness of agents and robots to achieve smoother interactions with humans. The realization of an effective system to model human gestures, which are matched with the speech utterances, is necessary to be embedded in these agents. In this work, we propose a GRU-based autoregressive generation model for gesture generation, which is trained with a CNN-based discriminator in an adversarial manner using a WGAN-based learning algorithm. The model is trained to output the rotation angles of the joints in the upper body, and implemented to animate a CG avatar. The motions synthesized by the proposed system are evaluated via an objective measure and a subjective experiment, showing that the proposed model outperforms a baseline model which is trained by a state-of-the-art GAN-based algorithm, using the same dataset. This result reveals that it is essential to develop a stable and robust learning algorithm for training gesture generation models. Our code can be found in https://github.com/wubowen416/gesture-generation.
A study on data augmentation in voice anti-spoofing

In this paper we perform an in depth study of how data augmentation techniques improve synthetic or spoofed audio detection. Specifically, we propose methods to deal with channel variability, different audio compressions, different bandwidths and unseen spoofing attacks. These challenges, have all been shown to significantly degrade the performance of audio based systems and anti spoofing systems. Our results are based on the ASVspoof 2021 challenge, in the Logical Access (LA) and Deep Fake (DF) categories. Our study is Data-Centric, meaning that the models are fixed and we significantly improve the results by manipulating the data. We introduce two forms of data augmentation - compression augmentation for the DF part, and compression and channel augmentation for the LA part. In addition, we introduce a double sided log spectrogram feature design that improves the results significantly by centering the sub-bands of interest, where the discriminating spoofing artifacts can be localized. Furthermore, a new type of online data augmentation, SpecAverage, is introduced. This method includes masking the audio features with their average value in order to improve generalization. Our best single system and fusion schemes both achieve state of the art performance in the DF category, with an EER of 15.46% and 14.27%, respectively. Our best system for the LA task reduced the best baseline EER by 50% and the min t-DCF by 16%. Our techniques to deal with spoofed data from a wide variety of distributions can be replicated and can help anti spoofing and speech based systems enhance their results.
Efficient Privacy Preserving Edge Intelligent Computing Framework for Image Classification in IoT

To extract knowledge from the large data collected by edge devices, a traditional cloud-based approach that requires data upload may not be feasible due to communication bandwidth limitations as well as privacy and security concerns of end-users. A novel privacy-preserving edge intelligent computing framework for image classification in IoT is proposed to address these challenges. Specifically, the autoencoder will be trained unsupervised at each edge device individually, and then the obtained latent vectors transmitted to the edge server for the training of a classifier. This framework would reduce the communication overhead and protect end-users’ data. Compared to federated learning, the training of the classifier in the proposed framework is not subject to the constraints of the edge devices, and the autoencoder can be trained independently at each edge device without any server involvement. Compared to collaborative intelligence such as SplitNN, the proposed method does not suffer from high communication cost as noticed in SplitNN. Furthermore, the privacy of the end-users’ data is protected by transmitting latent vectors and without the additional cost of encryption. Experimental results provide insights on the image classification performance vs. various design parameters such as the data compression ratio of the autoencoder and the model complexity.
Gender Classification Using Proposed CNN-Based Model and Ant Colony Optimization

Pedestrian gender classification is one of the key assignments of pedestrian study, and it finds practical applications in content-based image retrieval, population statistics, human–computer interaction, health care, multimedia retrieval systems, demographic collection, and visual surveillance. In this research work, gender classification was carried out using a deep learning approach. A new 64-layer architecture named 4-BSMAB derived from deep AlexNet is proposed. The proposed model was trained on CIFAR-100 dataset utilizing SoftMax classifier. Then, features were obtained from applied datasets with this pre-trained model. The obtained feature set was optimized with ant colony system (ACS) optimization technique. Various classifiers of SVM and KNN were used to perform gender classification utilizing the optimized feature set. Comprehensive experimentation was performed on gender classification datasets, and proposed model produced better results than the existing methods. The suggested model attained highest accuracy, i.e., 85.4%, and 92% AUC on MIT dataset, and best classification results, i.e., 93% accuracy and 96% AUC, on PKU-Reid dataset. The outcomes of extensive experiments carried out on existing standard pedestrian datasets demonstrate that the proposed framework outperformed existing pedestrian gender classification methods, and acceptable results prove the proposed model as a robust model.
X-GGM: Graph Generative Modeling for Out-of-distribution Generalization in Visual Question Answering

Encouraging progress has been made towards Visual Question Answering (VQA) in recent years, but it is still challenging to enable VQA models to adaptively generalize to out-of-distribution (OOD) samples. Intuitively, recompositions of existing visual concepts (i.e., attributes and objects) can generate unseen compositions in the training set, which will promote VQA models to generalize to OOD samples. In this paper, we formulate OOD generalization in VQA as a compositional generalization problem and propose a graph generative modeling-based training scheme (X-GGM) to handle the problem implicitly. X-GGM leverages graph generative modeling to iteratively generate a relation matrix and node representations for the predefined graph that utilizes attribute-object pairs as nodes. Furthermore, to alleviate the unstable training issue in graph generative modeling, we propose a gradient distribution consistency loss to constrain the data distribution with adversarial perturbations and the generated distribution. The baseline VQA model (LXMERT) trained with the X-GGM scheme achieves state-of-the-art OOD performance on two standard VQA OOD benchmarks, i.e., VQA-CP v2 and GQA-OOD. Extensive ablation studies demonstrate the effectiveness of X-GGM components.