Reviewerly/exHarmony huggingface dataset:
Tree Structure-Aware Few-Shot Image Classification via Hierarchical Aggregation

In this paper, we mainly focus on the problem of how to learn additional feature representations for few-shot image classification through pretext tasks (e.g., rotation or color permutation and so on). This additional knowledge generated by pretext tasks can further improve the performance of few-shot learning (FSL) as it differs from human-annotated supervision (i.e., class labels of FSL tasks). To solve this problem, we present a plug-in Hierarchical Tree Structure-aware (HTS) method, which not only learns the relationship of FSL and pretext tasks, but more importantly, can adaptively select and aggregate feature representations generated by pretext tasks to maximize the performance of FSL tasks. A hierarchical tree constructing component and a gated selection aggregating component is introduced to construct the tree structure and find richer transferable knowledge that can rapidly adapt to novel classes with a few labeled images. Extensive experiments show that our HTS can significantly enhance multiple few-shot methods to achieve new state-of-the-art performance on four benchmark datasets. The code is available at: https://github.com/remiMZ/HTS-ECCV22 .
A versatile dataset of agile open source software projects

Agile software development is nowadays a widely adopted practise in both open-source and industrial software projects. Agile teams typically heavily rely on issue management tools to document new issues and keep track of outstanding ones, in addition to storing their technical details, effort estimates, assignment to developers, and more. Previous work utilised the historical information stored in issue management systems for various purposes; however, when researchers make their empirical data public, it is usually relevant solely to the study's objective. In this paper, we present a more holistic and versatile dataset containing a wealth of information on more than half a million issues from 44 open-source Agile software, making it well-suited to several research avenues, and cross-analyses therein, including effort estimation, issue prioritization, issue assignment and many more. We make this data publicly available on GitHub to facilitate ease of use, maintenance, and extensibility.
Selective Residual M-Net for Real Image Denoising

Image restoration is a low-level vision task which is to restore degraded images to noise-free images. With the success of deep neural networks, the convolutional neural networks surpass the traditional restoration methods and become the mainstream in the computer vision area. To advance the performanceof denoising algorithms, we propose a blind real image denoising network (SRMNet) by employing a hierarchical architecture improved from U-Net. Specifically, we use a selective kernel with residual block on the hierarchical structure called M-Net to enrich the multi-scale semantic information. Furthermore, our SRMNet has competitive performance results on two synthetic and two real-world noisy datasets in terms of quantitative metrics and visual quality. The source code and pretrained model are available at https://github.com/TentativeGitHub/SRMNet.
Automatic Sentiment Annotation of Idiomatic Expressions for Sentiment Analysis Task

Users of social media may use words and phrases literally to convey their views or opinion clearly.However, some people choose to utilise idioms or proverbs that are implicit and indirect in order to make a stronger impression on the audience or perhaps to catch their attention by utilising a funny, sarcastic, or metaphorical phrases.Idioms and proverbs are examples of figurative expressions with a thematically coherent totality that cannot be understood literally.In a previous work, the extension of IBM's Sentiment Lexicon of Idiomatic Expressions was proposed to include around 9,000 idioms; both lexicons are manually annotated by crowdsourcing service.Therefore, in this research, we provide knowledge-based expansion approach to avoid human annotation of idioms.For sentiment classification, the proposed method has the advantage that it does not require any fine-tuning for the BERT model.Experimental comparisons show that the automated idiom enrichment and annotation are very beneficial for the performance of the sentiment classifier.The expanded annotated lexicon will be made available to the general public.
Moz$$\mathbb {Z}_{2^k}$$arella: Efficient Vector-OLE and Zero-Knowledge Proofs over $$\mathbb {Z}_{2^k}$$

Zero-knowledge proof systems are usually designed to support computations for circuits over $$\mathbb {F}_2$$ or $$\mathbb {F}_p$$ for large p, but not for computations over $$\mathbb {Z}_{2^k}$$ , which all modern CPUs operate on. Although $$\mathbb {Z}_{2^k}$$ -arithmetic can be emulated using prime moduli, this comes with an unavoidable overhead. Recently, Baum et al. (CCS 2021) suggested a candidate construction for a designated-verifier zero-knowledge proof system that natively runs over $$\mathbb {Z}_{2^k}$$ . Unfortunately, their construction requires preprocessed random vector oblivious linear evaluation (VOLE) to be instantiated over $$\mathbb {Z}_{2^k}$$ . Currently, it is not known how to efficiently generate such random VOLE in large quantities. In this work, we present a maliciously secure, VOLE extension protocol that can turn a short seed-VOLE over $$\mathbb {Z}_{2^k}$$ into a much longer, pseudorandom VOLE over the same ring. Our construction borrows ideas from recent protocols over finite fields, which we non-trivially adapt to work over $$\mathbb {Z}_{2^k}$$ . Moreover, we show that the approach taken by the QuickSilver zero-knowledge proof system (Yang et al. CCS 2021) can be generalized to support computations over $$\mathbb {Z}_{2^k}$$ . This new VOLE-based proof system, which we call QuarkSilver, yields better efficiency than the previous zero-knowledge protocols suggested by Baum et al. Furthermore, we implement both our VOLE extension and our zero-knowledge proof system, and show that they can generate 13–50 million VOLEs per second for $${64}\,{\textrm{bit}}$$ to $${256}\,{\textrm{bit}}$$ rings, and evaluate $${1.3}\,\textrm{million}$$ $${64}\,{\textrm{bit}}$$ multiplications per second in zero-knowledge.
Learning to Imagine: Diversify Memory for Incremental Learning using Unlabeled Data

Deep neural network (DNN) suffers from catastrophic forgetting when learning incrementally, which greatly limits its applications. Although maintaining a handful of samples (called “exemplars”) of each task could alleviate forgetting to some extent, existing methods are still limited by the small number of exemplars since these exemplars are too few to carry enough task-specific knowledge, and therefore the forgetting remains. To overcome this problem, we propose to “imagine” diverse counterparts of given exemplars referring to the abundant semantic-irrelevant information from unlabeled data. Specifically, we develop a learnable feature generator to diversify exemplars by adaptively generating diverse counterparts of exemplars based on semantic information from exemplars and semantically-irrelevant information from unlabeled data. We introduce semantic contrastive learning to enforce the generated samples to be semantic consistent with exemplars and perform semantic-decoupling contrastive learning to encourage diversity of generated samples. The diverse generated samples could effectively prevent DNN from forgetting when learning new tasks. Our method does not bring any extra inference cost and outperforms state-of-the-art methods on two benchmarks CIFAR-100 and ImageNet-Subset by a clear margin.
A Highly Robust Reversible Watermarking Scheme Using Embedding Optimization and Rounded Error Compensation

The robust reversible watermarking (RRW) requires high robustness and capacity on the condition of reversibility and imperceptibility, which still remains a big challenge nowadays. In this paper, we propose a two-stage RRW scheme that improves robustness and capacity through embedding optimization and rounded error compensation. The first stage inserts a robust watermark into the selected Pseudo-Zernike moments (PZMs) by using an adaptive normalization method and an optimized embedding strategy. Specifically, the adaptive normalization method achieves both an invariance to pixel amplitude variation and a balance between robustness and imperceptibility, and the optimized embedding strategy reduces embedding distortions remarkably. The watermarked PZMs are inversely transformed to generate the robustly watermarked image, in which rounded errors caused in the inverse transformation is compensated elaborately and thus a larger capacity can be obtained at the same embedding distortion. The second stage embeds a reversible watermark consisting of errors between the robust watermark embedded image and the original one, aiming at achieving the reversibility in case of no attacks. Extensive experimental simulations show that the proposed scheme provides strong robustness against common signal processing, including AWGN, salt-and-pepper noise, JPEG, JPEG2000, median filtering, mean filtering, geometrical transformations involving rotation and scaling, and a compressive sensing attack exemplified by two-dimensional compressive sensing, which outperforms the state-of-the-art schemes. Our code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/yichao-tang/PZMs-RRW</uri> .
Pose Forecasting in Industrial Human-Robot Collaboration

Pushing back the frontiers of collaborative robots in industrial environments, we propose a new Separable-Sparse Graph Convolutional Network (SeS-GCN) for pose forecasting. For the first time, SeS-GCN bottlenecks the interaction of the spatial, temporal and channel-wise dimensions in GCNs, and it learns sparse adjacency matrices by a teacher-student framework. Compared to the state-of-the-art, it only uses 1.72% of the parameters and it is $$\sim $$ 4 times faster, while still performing comparably in forecasting accuracy on Human3.6M at 1 s in the future, which enables cobots to be aware of human operators. As a second contribution, we present a new benchmark of Cobots and Humans in Industrial COllaboration (CHICO ). CHICO includes multi-view videos, 3D poses and trajectories of 20 human operators and cobots, engaging in 7 realistic industrial actions. Additionally, it reports 226 genuine collisions, taking place during the human-cobot interaction. We test SeS-GCN on CHICO for two important perception tasks in robotics: human pose forecasting, where it reaches an average error of 85.3 mm (MPJPE) at 1 sec in the future with a run time of 2.3 ms, and collision detection, by comparing the forecasted human motion with the known cobot motion, obtaining an F1-score of 0.64.
Learning Invisible Markers for Hidden Codes in Offline-to-online Photography

QR (quick response) codes are widely used as an offline-to-online channel to convey information (e.g., links) from publicity materials (e.g., display and print) to mobile devices. However, QR codes are not favorable for taking up valuable space of publicity materials. Recent works propose invisible codes/hyperlinks that can convey hidden information from offline to online. However, they require markers to locate invisible codes, which fails the purpose of invisible codes to be visible because of the markers. This paper proposes a novel invisible information hiding architecture for display/print-camera scenarios, consisting of hiding, locating, correcting, and recovery, where invisible markers are learned to make hidden codes truly invisible. We hide information in a sub-image rather than the entire image and include a localization module in the end-to-end framework. To achieve both high visual quality and high recovering robustness, an effective multi-stage training strategy is proposed. The experimental results show that the proposed method outperforms the state-of-the-art information hiding methods in both visual quality and robustness. In addition, the automatic localization of hidden codes significantly reduces the time of manually correcting geometric distortions for photos, which is a revolutionary innovation for information hiding in mobile applications.
Constructing Balance from Imbalance for Long-Tailed Image Recognition

Long-tailed image recognition presents massive challenges to deep learning systems since the imbalance between majority (head) classes and minority (tail) classes severely skews the data-driven deep neural networks. Previous methods tackle with data imbalance from the viewpoints of data distribution, feature space, and model design, etc. In this work, instead of directly learning a recognition model, we suggest confronting the bottleneck of head-to-tail bias before classifier learning, from the previously omitted perspective of balancing label space. To alleviate the head-to-tail bias, we propose a concise paradigm by progressively adjusting label space and dividing the head classes and tail classes, dynamically constructing balance from imbalance to facilitate the classification. With flexible data filtering and label space mapping, we can easily embed our approach to most classification models, especially the decoupled training methods. Besides, we find the separability of head-tail classes varies among different features with different inductive biases. Hence, our proposed model also provides a feature evaluation method and paves the way for long-tailed feature learning. Extensive experiments show that our method can boost the performance of state-of-the-arts of different types on widely-used benchmarks. Code is available at https://github.com/silicx/DLSA .
ContextFL: Context-aware Federated Learning by Estimating the Training and Reporting Phases of Mobile Clients

Federated Learning (FL) suffers from Low-quality model training in mobile edge computing, due to the dynamic environment of mobile clients. To the best of our knowledge, most FL frameworks follow the reactive client scheduling, in which the FL parameter server selects participants according to the currently-observed state of clients. Thus, the participants selected by the reactive-manner methods are very likely to fail while training a round of FL. To this end, we propose a proactive Context-aware Federated Learning (ContextFL) mechanism, which consists of two primary modules. Firstly, the state prediction module enables each client device to predict the conditions of both local training and reporting phases of FL locally. Secondly, the decision-making algorithm module is devised using the contextual Multi-Armed Bandit (cMAB) framework, which can help the parameter server select the most appropriate group of mobile clients. Finally, we carried out trace-driven FL experiments using real-world mobility datasets collected from volunteers. The evaluation results demonstrate that the proposed ContextFL mechanism outperforms other baselines in terms of the convergence stability of the global FL model and the ratio of valid participants.
Self-supervised Learning for Real-World Super-Resolution from Dual Zoomed Observations

In this paper, we consider two challenging issues in reference-based super-resolution (RefSR), (i) how to choose a proper reference image, and (ii) how to learn real-world RefSR in a self-supervised manner. Particularly, we present a novel self-supervised learning approach for real-world image SR from observations at dual camera zooms (SelfDZSR). Considering the popularity of multiple cameras in modern smartphones, the more zoomed (telephoto) image can be naturally leveraged as the reference to guide the SR of the lesser zoomed (short-focus) image. Furthermore, SelfDZSR learns a deep network to obtain the SR result of short-focus image to have the same resolution as the telephoto image. For this purpose, we take the telephoto image instead of an additional high-resolution image as the supervision information and select a center patch from it as the reference to super-resolve the corresponding short-focus image patch. To mitigate the effect of the misalignment between short-focus low-resolution (LR) image and telephoto ground-truth (GT) image, we design an auxiliary-LR generator and map the GT to an auxiliary-LR while keeping the spatial position unchanged. Then the auxiliary-LR can be utilized to deform the LR features by the proposed adaptive spatial transformer networks (AdaSTN), and match the Ref features to GT. During testing, SelfDZSR can be directly deployed to super-solve the whole short-focus image with the reference of telephoto image. Experiments show that our method achieves better quantitative and qualitative performance against state-of-the-arts. Codes are available at https://github.com/cszhilu1998/SelfDZSR .
An Efficient Training Approach for Very Large Scale Face Recognition

Face recognition has achieved significant progress in deep learning era due to the ultra-large-scale and well- labeled datasets. However, training on the outsize datasets is time-consuming and takes up a lot of hardware resource. Therefore, designing an efficient training approach is in- dispensable. The heavy computational and memory costs mainly result from the million-level dimensionality of the fully connected (FC) layer. To this end, we propose a novel training approach, termed Faster Face Classification (F <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> C), to alleviate time and cost without sacrificing the performance. This method adopts Dynamic Class Pool (DCP) for storing and updating the identities' features dy-namically, which could be regarded as a substitute for the FC layer. DCP is efficiently time-saving and cost-saving, as its smaller size with the independence from the whole face identities together. We further validate the proposed F <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> C method across several face benchmarks and private datasets, and display comparable results, meanwhile the speed is faster than state-of-the-art FC-based methods in terms of recognition accuracy and hardware costs. More-over, our method is further improved by a well-designed dual data loader including indentity-based and instance- based loaders, which makes it more efficient for updating DCP parameters.
MTFormer: Multi-task Learning via Transformer and Cross-Task Reasoning

In this paper, we explore the advantages of utilizing transformer structures for addressing multi-task learning (MTL). Specifically, we demonstrate that models with transformer structures are more appropriate for MTL than convolutional neural networks (CNNs), and we propose a novel transformer-based architecture named MTFormer for MTL. In the framework, multiple tasks share the same transformer encoder and transformer decoder, and lightweight branches are introduced to harvest task-specific outputs, which increases the MTL performance and reduces the time-space complexity. Furthermore, information from different task domains can benefit each other, and we conduct cross-task reasoning. We propose a cross-task attention mechanism for further boosting the MTL results. The cross-task attention mechanism brings little parameters and computations while introducing extra performance improvements. Besides, we design a self-supervised cross-task contrastive learning algorithm for further boosting the MTL performance. Extensive experiments are conducted on two multi-task learning datasets, on which MTFormer achieves state-of-the-art results with limited network parameters and computations. It also demonstrates significant superiorities for few-shot learning and zero-shot learning.
3D Moments from Near-Duplicate Photos

We introduce 3D Moments, a new computational photography effect. As input we take a pair of near-duplicate photos, i.e., photos of moving subjects from similar viewpoints, common in people's photo collections. As output, we produce a video that smoothly interpolates the scene motion from the first photo to the second, while also producing camera motion with parallax that gives a heightened sense of 3D. To achieve this effect, we represent the scene as a pair of feature-based layered depth images augmented with scene flow. This representation enables motion interpolation along with independent control of the camera viewpoint. Our system produces photorealistic space-time videos with motion parallax and scene dynamics, while plausibly recovering regions occluded in the original views. We conduct extensive experiments demonstrating superior performance over baselines on public datasets and in-the-wild photos. Project page: https://3d-moments.github.io/.
Generalizable Medical Image Segmentation via Random Amplitude Mixup and Domain-Specific Image Restoration

For medical image analysis, segmentation models trained on one or several domains lack generalization ability to unseen domains due to discrepancies between different data acquisition policies. We argue that the degeneration in segmentation performance is mainly attributed to overfitting to source domains and domain shift. To this end, we present a novel generalizable medical image segmentation method. To be specific, we design our approach as a multi-task paradigm by combining the segmentation model with a self-supervision domain-specific image restoration (DSIR) module for model regularization. We also design a random amplitude mixup (RAM) module, which incorporates low-level frequency information of different domain images to synthesize new images. To guide our model be resistant to domain shift, we introduce a semantic consistency loss. We demonstrate the performance of our method on two public generalizable segmentation benchmarks in medical images, which validates our method could achieve the state-of-the-art performance. (Code is available at https://github.com/zzzqzhou/RAM-DSIR ).
Accountable Attribute-Based Data-Sharing Scheme Based on Blockchain for Vehicular Ad Hoc Network

Vehicular ad hoc network (VANET), as one of the bases of intelligent transport systems, plays an essential role in improving road traffic safety. Nevertheless, in such a complicated, distributed, and highly mobile network structure, how to achieve secure data sharing is a great challenge. The ciphertext-policy attribute-based encryption (CP-ABE) is a potential method to realize one-to-many data sharing for VANET. However, the key abuse problems of users and attribute authorities (AAs) incur many security concerns for VANET. Both issues are extremely important because the attribute keys directly affect users’ access to shared data. To solve the above issues, we propose an accountable attribute-based data-sharing scheme with the blockchain technology (AT-DS-VAHN, in short). For AAs key abuse, we use the consortium blockchain maintained by AAs to achieve distributed key storage and distribution. The attribute keys generated by each AA and its key distribution records are recorded on the blockchain in the form of transactions. Based on the traceability of blockchain, the key abuse behavior of AAs can be caught and prosecuted. For user key abuse, we achieve white-box traceability and efficient user revocation. Based on the principle of traceable-then-revocable, malicious users can be tracked and then revoked directly from the system without complex operations. Besides, to reduce the computation burden on users, our proposal supports online/offline encryption and verifiable outsourced decryption. Security and efficiency analyses show that our proposal is secure and efficient, with high practicability and reliability for VANET.