Reviewerly/exHarmony huggingface dataset:
FastCaps: A Design Methodology for Accelerating Capsule Network on Field Programmable Gate Arrays

Capsule Network (CapsNet) has shown significant improvement in understanding the variation in images along with better generalization ability compared to traditional Convolutional Neural Network (CNN). CapsNet preserves spatial relationship among extracted features and apply dynamic routing to efficiently learn the internal connections between capsules. However, due to the capsule structure and the complexity of the routing mechanism, it is non-trivial to accelerate CapsNet performance in its original form on Field Programmable Gate Array (FPGA). Most of the existing works on CapsNet have achieved limited acceleration as they implement only the dynamic routing algorithm on FPGA, while considering all the processing steps synergistically is important for real-world applications of Capsule Networks. Towards this, we propose a novel two-step approach that deploys a full-fledged CapsNet on FPGA. First, we prune the network using a novel Look-Ahead Kernel Pruning (LAKP) methodology that uses the sum of look-ahead scores of the model parameters. Next, we simplify the nonlinear operations, reorder loops, and parallelize operations of the routing algorithm to reduce CapsNet hardware complexity. To the best of our knowledge, this is the first work accelerating a full-fledged CapsNet on FPGA. Experimental results on the MNIST and F-MNIST datasets (typical in Capsule Network community) show that the proposed LAKP approach achieves an effective compression rate of 99.26% and 98.84%, and achieves a throughput of 82 FPS and 48 FPS on Xilinx PYNQ-Z1 FPGA, respectively. Furthermore, reducing the hardware complexity of the routing algorithm increases the throughput to 1351 FPS and 934 FPS respectively. As corroborated by our results, this work enables highly performance-efficient deployment of CapsNets on low-cost FPGA that are popular in modern edge devices.
Combining datasets to improve model fitting

For many use cases, combining information from different datasets can be of interest to improve a machine learning model's performance, especially when the number of samples from at least one of the datasets is small. An additional challenge in such cases is that the features from these datasets are not identical, even though there are some commonly shared features among the datasets. To tackle this, we propose a novel framework called Combine datasets based on Imputation (ComImp). In addition, we propose PCA-ComImp, a variant of ComImp that utilizes Principle Component Analysis (PCA), where dimension reduction is conducted before combining datasets. This is useful when the datasets have a large number of features that are not shared across them. Furthermore, our framework can also be utilized for data preprocessing by imputing missing data, i.e., filling in the missing entries while combining different datasets. To illustrate the performance and practicability of the proposed methods and their potential usages, we conduct experiments for various tasks (regression, classification) and for different data types (tabular data, time series data) when the datasets to be combined have missing data. We also investigate how the devised methods can be used with transfer learning to provide even further model training improvement. Our results indicate that can provide extra improvement when being used in combination with transfer learning.
An Anchor-Free Detector to Detect Small Objects

Owing to the lack of sufficient pixels and details, small object detection has long been a challenging and significant task in the field of detection. The traditional anchor-free detectors use vanilla sampling methods and only utilize very little regional information. This may lose a lot of information and can not utilize contextual information due to the relatively small receptive field. Therefore, we propose SCMBox to solve these problems. Compared to the current anchor-free techniques, our anchor-free approach includes several novel contributions. The entire network structure avoid using strided convolution layers and pooling layers, which makes the approach suitable for small objects with low resolution. The quality of feature upsampling directly influences the detection quality in small object detection. To preserve the global structure of the network as much as possible, we employed a highly effective upsampling operator that can make use of more contextual information. We proposed a modified receptive field block (MRFB) to increase the receptive field. We use Wasserstein distance in the loss function of the box regression to reduce the susceptibility of small objects to position variation. Our method achieves excellent performance on the standard MS-COCO object detection benchmark and Vis-Drone2021 dataset and outperforms the state-of-the-art methods.
Improving Visual Question Answering by Multimodal Gate Fusion Network

Visual question answering (VQA) is a difficult multimodal task that requires answering questions about images. It requires a fine-grained level of understanding of both the visual content of the image and the textual content of the question. However, most of the existing models perform weakly in filtering noisy information and are unable to fuse features from multiple modalities effectively. To resolve the above restriction, we propose a novel multimodal gate fusion network (MGFN), which consists of an attention-on-attention interaction module (AoAIM) and a multimodal gate fusion module (MGFM). The role of AoAIM is to capture intra-modal and inter-modal dependencies and to filter out some irrelevant attention. The proposed MGFM can effectively fuse textual and visual features based on the relative importance of textual and visual modalities. We have performed many ablation experiments on the VQA-v2 dataset to validate the effectiveness of AoAIM and MGFM. The ablation experiments demonstrate that both AoAIM and MGFM play a key role in improving the performance of the model. By embedding these two modules, MGFN performs better than the previous state-of-the-art (SOTA) model on the VQA-v2 dataset. Particularly, the MGFN achieves an overall accuracy of 71.68% on the test-dev set and 72.12% on the test-std set.
Distilling Stereo Networks for Performant and Efficient Leaner Networks

Knowledge distillation has been quite popular in vision for tasks like classification and segmentation however not much work has been done for distilling state-of-the-art stereo matching methods despite their range of applications. One of the reasons for its lack of use in stereo matching networks is due to the inherent complexity of these networks, where a typical network is composed of multiple two- and three-dimensional modules. In this work, we systematically combine the insights from state-of-the-art stereo methods with general knowledge-distillation techniques to develop a joint framework for stereo networks distillation with competitive results and faster inference. Moreover, we show, via a detailed empirical analysis, that distilling knowledge from the stereo network requires careful design of the complete distillation pipeline starting from backbone to the right selection of distillation points and corresponding loss functions. This results in the student networks that are not only leaner and faster but give excellent performance. For instance, our student network while performing better than the performance oriented methods like PSMNet [1], CFNet [2], and LEAStereo [3]) on benchmark SceneFlow dataset, is 8 x, 5 x, and 8 x faster respectively. Furthermore, compared to speed oriented methods having inference time less than 100ms, our student networks perform better than all the tested methods. In addition, our student network also shows better generalization capabilities when tested on unseen datasets like ETH3D and Middlebury <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Code: https://github.com/cogsys-tuebingen/Distilling-Stereo-Networks.
Improved Word Sense Disambiguation via Prompt-based Contextual Word Representation

Word sense disambiguation (WSD), which entails figuring out a target word's meaning based on its context, has become an increasingly significant task. However, none of them has been able to achieve an F1-score higher than 85% since Wordnet is used as the sense inventory. The primary cause of this problem is most likely the over-fine granularity of the lexical sense inventory, which makes it challenging for even skilled human annotators to distinguish certain senses of the word. Furthermore, there is a lack of sufficient annotation data for some senses. To address this issue, we build a more practical dataset based on current English-English dictionaries and then create a prompt-based contextual word representation (PCWR) that can better make use of the embedded semantic information in the pre-trained model and enhance the performance of the WSD model in low-resource scenarios. Additionally, we validate the performance of our method on the FEWS dataset in both the zero-shot and few-shot cases.
Relation-Aware Graph Attention Network for Multi-Behavior Recommendation

In practical recommendation scenarios, the types of user behaviors are usually diverse (e.g., click, add-to-cart, purchase), and different types of user behaviors can provide different aspects of user preference information. However, most existing methods only consider a single type of user behavior for modeling, which is not sufficient to fully learn complex user preferences. Besides multiple behavior types, the heterogeneous preference strength of users for items under the same behavior is also a factor that is often overlooked by most methods. Moreover, different types of behaviors may be correlated due to various factors, inadequate exploration of the implicit relationships between different types of behaviors may lead to the loss of potential information across behaviors. To solve the above problems, we propose a novel multi-behavior model with relation-aware graph attention network (RGAN), which is built on a graph-based neural architecture to explore high-order user-item relations. Specifically, we design a relation-aware attention propagation layer and an inter-behavior dependency encoder to capture heterogeneous collaborative signals from type-specific and inter-type behavior relations, respectively. During behavior integration, our proposed model automatically learns which types of behaviors are more important for assisting target behavior prediction. Extensive experiments conducted on three real-world datasets demonstrate that the RGAN model consistently outper-forms many state-of-the-art baselines, in terms of HR@n and NDCG@n.
Deep Inversion Method for Attacking Lifelong Learning Neural Networks

Artificial neural networks suffer from catastrophic forgetting when knowledge needs to be learned from multi-batch or streaming data. In response to this problem, researchers have proposed a variety of lifelong learning methods to avoid catastrophic forgetting. However, current methods usually do not consider the possibility of malicious attacks. Meanwhile, in real lifelong learning scenarios, batch data or streaming data usually come from an incompletely trusted environment. Attackers can easily manipulate data or inject malicious samples into the training data set. As a result, the reliability of neural networks decreases. Recently, researches of lifelong learning attacks need to obtain real samples of the attacked classes, whether using backdoor attacks or data poisoning attacks. In this paper, we focus on an attack setting that is more suitable for lifelong learning scenario. This setting has two main features. The first is the setting does not require real samples of the attacked classes, and the second is it allows attacks to be performed on tasks that exclude the attacked classes. For this scenario, we propose a lifelong learning attack model based on deep inversion. In the scenario where EWC is used as the benchmark lifelong learning model, our experiments show that 1) in the data poisoning attack, the target accuracy can be significantly decreased by adding 0.5% of poisoned samples; 2) The backdoor attack with high accuracy can be achieved by adding 1% of backdoor samples.
Out-of-Scope Intent Detection with Supervised Deep Metric Learning

Detecting Out-of-Scope(OOS) intents in dialogue systems is a challenging technique with practical applications. As for OOS intent detection, it not only ensures the accuracy of classifying known intents but detecting OOS intents is also crucial. Current related models are limited in learning decision boundaries or setting the threshold of confidence score, which all neglect that a well-formed intent representation is a key point. Meanwhile, text extractors trained by traditional cross-entropy loss merely focus on reducing the error rate of the class to which the sample is classified. In this paper, we propose an effective feature extraction method based on deep metric learning to construct the triplet network with prior knowledge. With the constructed triplet loss, mining hard samples, which refers to the far-apart intents between the same class and close intent representations among different classes, can further obtain discriminative intent representations. In addition, we also introduce adversarial training to make intent representations more robust. Experiments on three public datasets prove the effectiveness of our proposed method of learning discriminative intent representations.
Spatially Exclusive Pasting: A General Data Augmentation for the Polyp Segmentation

Automated polyp segmentation technology plays an important role in diagnosing intestinal diseases, such as tumors and precancerous lesions. Previous works have typically trained convolution-based U-Net or Transformer-based neural network architectures with labeled data. However, the available public polyp segmentation datasets are too small to train the network sufficiently, suppressing each network's potential performance. To alleviate this issue, we propose a universal data augmentation technology, called SEP, to synthesize more data from the existing datasets. Specifically, we paste the polyp area with surroundings into the same image's background in a spatial-exclusive manner to obtain a combinatorial number of new images. SEP is a preprocessing method that can be incorporated into any existing network. For a fair comparison, we design a modular platform to facilitate conducting controlled experiments on various networks and datasets. Extensive results show that the proposed method enhances the data efficiency and achieves consistent improvements over baselines. Finally, we hit a new state of the art in this task.
Towards Sharper Risk Bounds for Agnostic Multi-objective Learning

Many real-world machine learning tasks have mul-tiple objectives, such as multi-object detection and product recommendation, which can not be optimized directly through a single objective function. Fortunately, multi-objective learning can be used to solve this problem efficiently by some vector-valued algorithms. Recently, researchers find that the performance of multi-objective learning will be impaired when the mixture weights are unknown, where a fixed algorithm is difficult to select the optimal model in the hypothesis space. Thus, agnostic multi-objective learning has been proposed, which provides an effective approach to solve the problem of simultaneously optimizing multiple objectives with unknown mixture weights. In this way, a proper model will be selected because the agnostic multi-objective learning can improve the worst case of the hypothesis space. However, the current generalization error bounds for agnostic multi-objective learning can not converge faster than <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathcal{O}(1/\sqrt{n})$</tex> , which limits the generalization guarantee. In this paper, we provide a sharper excess risk bound for agnostic multi-objective learning with convergence rate of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathcal{O}(1/n)$</tex> , which is much faster than the existing results and matches the best theoretical results of centralized learning. Based on our theory, we then propose a novel algorithm to improve the generalization performance of agnostic multi-objective learning.
IFA-Net: Isomerous Feature-aware Network for Single-view 3D Reconstruction

Single-view 3D reconstruction has long been an intractable and fundamental problem in computer vision. Objects with complex topological structures are difficult to be accurately reconstructed, which makes the existing methods suffer from blurred shape boundaries between multiple components in the object. Recently, convolutional neural network and vision transformer have begun to appear in the field of 3D reconstruction and have been widely used with excellent performance. However, the existing transformer-based methods mainly focus on the global long-term context dependency, and ignore the local details of the part space features, resulting in poor reconstruction of the detail part. In this paper, we propose a novel dual-branch network architecture, called IFA-Net, to capture local spatial perception information and retain global structural features for single-view 3D reconstruction. In addition, we propose an isomerous feature-aware module, which enables the dynamic fusion of different resolution features under the two branches. Thus, high-fidelity and detail-rich 3D object reconstruction can be achieved. Extensive experimental results demonstrate that our method is able to produce high-quality voxels, particularly with diverse topologies, as compared with the state-of-the-art methods.
FLCAP: Federated Learning with Clustered Adaptive Pruning for Heterogeneous and Scalable Systems

In this paper, we address the challenge of efficient local training on heterogeneous data in Federated Learning, where devices have limited computation and communication resources. The proposed algorithm generates personalized masks for groups of nodes by applying structured or unstructured pruning methods. We find a trade-off between global generalization and per-device personalization by optimizing the personalized masks to local data distributions through clustering methods and by aggregating the parameters in a unique model. Our results demonstrate effectiveness of the method for training personalized and sparsified models on non-IID distributions. The maximal accuracy achieved by our Clustered Adaptive Pruning method outperforms the Federated Average algorithm, while using a small percentage of the original set of coefficients per dataset. In particular, for MNIST, CIFAR10, and FMNIST datasets, our method uses approximately two-thirds, one-third, and less than one-third of the total number of parameters required for Federated Average training, respectively.
DietCNN: Multiplication-free Inference for Quantized CNNs

The rising demand for networked embedded systems with machine intelligence has been a catalyst for sustained attempts by the research community to implement Convolutional Neural Networks (CNN) based inferencing on embedded resource-limited devices. Redesigning a CNN by removing costly multiplication operations has already shown promising results in terms of reducing inference energy usage. This paper proposes a new method for replacing multiplications in a CNN by table look-ups. Unlike existing methods that completely modify the CNN operations, the proposed methodology preserves the semantics of the major CNN operations. Conforming to the existing mechanism of the CNN layer operations ensures that the reliability of a standard CNN is preserved. It is shown that the proposed multiplication-free CNN, based on a single activation codebook, can achieve 4.7x, 5.6x, and 3.5x reduction in energy per inference in an FPGA implementation of MNIST-LeNet-5, CIFAR10-VGG-11, and Tiny ImageNet-ResNet-18 respectively. Our results show that the DietCNN approach significantly improves the resource consumption and latency of deep inference for smaller models, often used in embedded systems. Our code is available at: https://github.com/swadeykgp/DietCNN
SFMNet: Self-guided Feature Mining Network for Vehicle Re-identification

Vehicle Re-identification (Re-ID) is very important in intelligent transportation and video surveillance. Most previous works mainly extract discriminative features from the visual appearance of vehicles. However, the irrelevant background in the image will affect the extracted features, and it is also important to extract fine-grained features because the vehicles have the same style. Previous works extracted robust features by manual annotation, which was inefficient. To solve this problem, this paper proposes a self-guided feature mining network (SFMNet) that can eliminate background intervention without annotation, while mining fine-grained feature information of vehicles. In order to eliminate background intervention and mine fine-grained features without resorting to annotations, we have carefully designed two novel modules. (i) The noise patch filter (NPF) module can identify the image background without annotation, and filter out the background to eliminate the intervention of the background on the image features. (ii) The salient feature extraction (SFE) module uses self-attention as a guide without vehicle component annotation to mine fine-grained features and enhance discriminative visual cue features. Extensive experiments demonstrate the effectiveness of our method, and we achieve state-of-the-art results on three publicly available datasets, including Veri-776, VehicleID, and VERI-WILD.
Swarm Behavior Recognition: Martingales Protected Paradigm

This paper introduces a martingales-protected paradigm for machine learning algorithms to solve swarm behavior classification problems. While regular machine learning algorithms, such as Neural Networks, Random Forests, and Decision Trees try to induce a general decision function for a learning task, the martingales-protected machine learning algorithms consider a set of fuzzy decision functions. The paper analyzes why the martingales-protected paradigm is well suited for most existing machine learning algorithms. These theoretical findings are supported by experiments on three swarm test collections. The experiments show substantial improvements over regular machine learning algorithms and present an empirical comparison of the new method and the classical learning approach for predictive modeling of real-life swarm behavior data. These comparisons suggest that the proposed martingale-protected method consistently performs better predictive performance than classical predictive modeling.