OpenRaiser/Intern-Atlas huggingface dataset:
Rethinking the backbone architecture for tiny object detection
Venue: arXiv cs.CV 2023
Year: 2023

Tiny object detection has become an active area of research because images with tiny targets are common in several important real-world scenarios. However, existing tiny object detection methods use standard deep neural networks as their backbone architecture. We argue that such backbones are inappropriate for detecting tiny objects as they are designed for the classification of larger objects, and do not have the spatial resolution to identify small targets. Specifically, such backbones use max-pooling or a large stride at early stages in the architecture. This produces lower resolution feature-maps that can be efficiently processed by subsequent layers. However, such low-resolution feature-maps do not contain information that can reliably discriminate tiny objects. To solve this problem we design 'bottom-heavy' versions of backbones that allocate more resources to processing higher-resolution features without introducing any additional computational burden overall. We also investigate if pre-training these backbones on images of appropriate size, using CIFAR100 and ImageNet32, can further improve performance on tiny object detection. Results on TinyPerson and WiderFace show that detectors with our proposed backbones achieve better results than the current state-of-the-art methods.
Method Chaining Redux: An Empirical Study of Method Chaining in Java, Kotlin, and Python
Venue: arXiv cs.SE 2023
Year: 2023

There are possible benefits and drawbacks to chaining methods together, as is often done in fluent APIs. A prior study investigated how Java developers chain methods in over 2.7k open-source projects. That study observed, for the dataset analyzed, that the use of method chaining in Java is popular and seems to be increasing over time. That study however was limited to a smaller sample of Java projects, and it is also not clear if the results generalize to other languages. In this work, we first replicate the prior results by building a similar dataset and our own analysis scripts. We then extend those results by analyzing a much larger dataset of 89k Java projects and generalizing to other programming languages by analyzing 26k Kotlin projects and 98k Python projects. The results show chaining is more popular in Java and Kotlin than Python, chaining use in Kotlin is not growing, and Python sees more use in non-testing code.
Cascading Hierarchical Networks with Multi-task Balanced Loss for Fine-grained hashing
Venue: arXiv cs.CV 2023
Year: 2023

With the explosive growth in the number of fine-grained images in the Internet era, it has become a challenging problem to perform fast and efficient retrieval from large-scale fine-grained images. Among the many retrieval methods, hashing methods are widely used due to their high efficiency and small storage space occupation. Fine-grained hashing is more challenging than traditional hashing problems due to the difficulties such as low inter-class variances and high intra-class variances caused by the characteristics of fine-grained images. To improve the retrieval accuracy of fine-grained hashing, we propose a cascaded network to learn compact and highly semantic hash codes, and introduce an attention-guided data augmentation method. We refer to this network as a cascaded hierarchical data augmentation network. We also propose a novel approach to coordinately balance the loss of multi-task learning. We do extensive experiments on some common fine-grained visual classification datasets. The experimental results demonstrate that our proposed method outperforms several state-of-art hashing methods and can effectively improve the accuracy of fine-grained retrieval. The source code is publicly available: https://github.com/kaiba007/FG-CNET.
Model Stitching: Looking For Functional Similarity Between Representations
Venue: arXiv cs.LG 2023
Year: 2023

Model stitching (Lenc & Vedaldi 2015) is a compelling methodology to compare different neural network representations, because it allows us to measure to what degree they may be interchanged. We expand on a previous work from Bansal, Nakkiran & Barak which used model stitching to compare representations of the same shapes learned by differently seeded and/or trained neural networks of the same architecture. Our contribution enables us to compare the representations learned by layers with different shapes from neural networks with different architectures. We subsequently reveal unexpected behavior of model stitching. Namely, we find that stitching, based on convolutions, for small ResNets, can reach high accuracy if those layers come later in the first (sender) network than in the second (receiver), even if those layers are far apart.
How (Implicit) Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part II: the Multi-D Case of Two Layers with Random First Layer
Venue: arXiv cs.LG 2023
Year: 2023

Randomized neural networks (randomized NNs), where only the terminal layer's weights are optimized constitute a powerful model class to reduce computational time in training the neural network model. At the same time, these models generalize surprisingly well in various regression and classification tasks. In this paper, we give an exact macroscopic characterization (i.e., a characterization in function space) of the generalization behavior of randomized, shallow NNs with ReLU activation (RSNs). We show that RSNs correspond to a generalized additive model (GAM)-typed regression in which infinitely many directions are considered: the infinite generalized additive model (IGAM). The IGAM is formalized as solution to an optimization problem in function space for a specific regularization functional and a fairly general loss. This work is an extension to multivariate NNs of prior work, where we showed how wide RSNs with ReLU activation behave like spline regression under certain conditions and if the input is one-dimensional.
Team Coordination on Graphs with State-Dependent Edge Cost
Venue: arXiv cs.MA 2023
Year: 2023

This paper studies a team coordination problem in a graph environment. Specifically, we incorporate "support" action which an agent can take to reduce the cost for its teammate to traverse some edges that have higher costs otherwise. Due to this added feature, the graph traversal is no longer a standard multi-agent path planning problem. To solve this new problem, we propose a novel formulation by posing it as a planning problem in the joint state space: the joint state graph (JSG). Since the edges of JSG implicitly incorporate the support actions taken by the agents, we are able to now optimize the joint actions by solving a standard single-agent path planning problem in JSG. One main drawback of this approach is the curse of dimensionality in both the number of agents and the size of the graph. To improve scalability in graph size, we further propose a hierarchical decomposition method to perform path planning in two levels. We provide complexity analysis as well as a statistical analysis to demonstrate the efficiency of our algorithm.
Did You Train on My Dataset? Towards Public Dataset Protection with Clean-Label Backdoor Watermarking
Venue: arXiv cs.CR 2023
Year: 2023

The huge supporting training data on the Internet has been a key factor in the success of deep learning models. However, this abundance of public-available data also raises concerns about the unauthorized exploitation of datasets for commercial purposes, which is forbidden by dataset licenses. In this paper, we propose a backdoor-based watermarking approach that serves as a general framework for safeguarding public-available data. By inserting a small number of watermarking samples into the dataset, our approach enables the learning model to implicitly learn a secret function set by defenders. This hidden function can then be used as a watermark to track down third-party models that use the dataset illegally. Unfortunately, existing backdoor insertion methods often entail adding arbitrary and mislabeled data to the training set, leading to a significant drop in performance and easy detection by anomaly detection algorithms. To overcome this challenge, we introduce a clean-label backdoor watermarking framework that uses imperceptible perturbations to replace mislabeled samples. As a result, the watermarking samples remain consistent with the original labels, making them difficult to detect. Our experiments on text, image, and audio datasets demonstrate that the proposed framework effectively safeguards datasets with minimal impact on original task performance. We also show that adding just 1% of watermarking samples can inject a traceable watermarking function and that our watermarking samples are stealthy and look benign upon visual inspection.
Optimization Framework for Green Networking
Venue: arXiv cs.NI 2023
Year: 2023

Reducing energy consumption - and especially carbon emissions - is one of the most important challenges facing humankind. ICT (Information and Communication Technology) is a powerful tool to reduce emissions as it offers alternatives to activities that are costly in energy: video streaming saves energy vs driving to a movie theater, for instance. Still, the carbon footprint of ICT in general and networking in particular have been growing, despite better energy efficiency per transmitted bit, due to the sheer growth in Internet usage and traffic. The information and communication technology (ICT) sector is currently estimated to create 2.7% of all global CO2 emissions and expected to continue to increase. Hence, monitoring and reducing the CO2 emissions from ICT is increasingly important. Networks are responsible for around 13% of ICT energy consumption, a third of which in turn is attributable to backbone (core) networks. As such, it is important to offer new mechanisms to reduce the energy footprint of networks. In this paper, we present a framework to include energy considerations (as a proxy for carbon footprint) into the management plane of a network. We apply this framework to optimize the network topology so as to minimize the energy spending while at the same time providing a satisfactory Quality of Experience (QoE) to the end users. We present this framework along high-level considerations, as its deployment and evaluation is left for future work.
Chance-Constrained Multi-Robot Motion Planning under Gaussian Uncertainties
Venue: arXiv cs.RO 2023
Year: 2023

We consider a chance-constrained multi-robot motion planning problem in the presence of Gaussian motion and sensor noise. Our proposed algorithm, CC-K-CBS, leverages the scalability of kinodynamic conflict-based search (K-CBS) in conjunction with the efficiency of the Gaussian belief trees used in the Belief-A framework, and inherits the completeness guarantees of Belief-A's low-level sampling-based planner. We also develop three different methods for robot-robot probabilistic collision checking, which trade off computation with accuracy. Our algorithm generates motion plans driving each robot from its initial state to its goal while accounting for the evolution of its uncertainty with chance-constrained safety guarantees. Benchmarks compare computation time to conservatism of the collision checkers, in addition to characterizing the performance of the planner as a whole. Results show that CC-K-CBS can scale up to 30 robots.
Smart-Tree: Neural Medial Axis Approximation of Point Clouds for 3D Tree Skeletonization
Venue: arXiv cs.CV 2023
Year: 2023

This paper introduces Smart-Tree, a supervised method for approximating the medial axes of branch skeletons from a tree point cloud. Smart-Tree uses a sparse voxel convolutional neural network to extract the radius and direction towards the medial axis of each input point. A greedy algorithm performs robust skeletonization using the estimated medial axis. Our proposed method provides robustness to complex tree structures and improves fidelity when dealing with self-occlusions, complex geometry, touching branches, and varying point densities. We evaluate Smart-Tree using a multi-species synthetic tree dataset and perform qualitative analysis on a real-world tree point cloud. Our experimentation with synthetic and real-world datasets demonstrates the robustness of our approach over the current state-of-the-art method. The dataset and source code are publicly available.
Dynamics-Aware Loss for Learning with Label Noise
Venue: arXiv cs.LG 2023
Year: 2023

Label noise poses a serious threat to deep neural networks (DNNs). Employing robust loss functions which reconcile fitting ability with robustness is a simple but effective strategy to handle this problem. However, the widely-used static trade-off between these two factors contradicts the dynamics of DNNs learning with label noise, leading to inferior performance. Therefore, we propose a dynamics-aware loss (DAL) to solve this problem. Considering that DNNs tend to first learn beneficial patterns, then gradually overfit harmful label noise, DAL strengthens the fitting ability initially, then gradually improves robustness. Moreover, at the later stage, to further reduce the negative impact of label noise and combat underfitting simultaneously, we let DNNs put more emphasis on easy examples than hard ones and introduce a bootstrapping term. Both the detailed theoretical analyses and extensive experimental results demonstrate the superiority of our method. Our source code can be found in https://github.com/XiuchuanLi/DAL.
Nonuniform magnetic domain-wall synapses enabled by population coding
Venue: arXiv cs.ET 2023
Year: 2023

Traditional artificial intelligence implemented in software is usually executed on accurate digital computers. Nevertheless, the nanoscale devices for the implementation of neuromorphic computing may not be ideally identical, and the performance is reduced by nonuniform devices. In biological brains, information is usually encoded by a cluster of neurons such that the variability of nerve cells does not influence the accuracy of human cognition and movement. Here, we introduce the population encoding strategy in neuromorphic computing and demonstrate that this strategy can overcome the problems caused by nonuniform devices. Using magnetic memristor device based on current-induced domain-wall motion as an example, we show that imperfect storage devices can be applied in a hardware network to perform principal component analysis (PCA), and the accuracy of unsupervised classification is comparable to that of conventional PCA using ideally accurate synaptic weights. Our results pave the way for hardware implementation of neuromorphic computing and lower the criteria for the uniformity of nanoscale devices.
Efficient Multi-stage Inference on Tabular Data
Venue: arXiv cs.LG 2023
Year: 2023

Many ML applications and products train on medium amounts of input data but get bottlenecked in real-time inference. When implementing ML systems, conventional wisdom favors segregating ML code into services queried by product code via Remote Procedure Call (RPC) APIs. This approach clarifies the overall software architecture and simplifies product code by abstracting away ML internals. However, the separation adds network latency and entails additional CPU overhead. Hence, we simplify inference algorithms and embed them into the product code to reduce network communication. For public datasets and a high-performance real-time platform that deals with tabular data, we show that over half of the inputs are often amenable to such optimization, while the remainder can be handled by the original model. By applying our optimization with AutoML to both training and inference, we reduce inference latency by 1.3x, CPU resources by 30%, and network communication between application front-end and ML back-end by about 50% for a commercial end-to-end ML platform that serves millions of real-time decisions per second.
Adaptive Experimentation at Scale: A Computational Framework for Flexible Batches
Venue: arXiv cs.LG 2023
Year: 2023

Standard bandit algorithms that assume continual reallocation of measurement effort are challenging to implement due to delayed feedback and infrastructural/organizational difficulties. Motivated by practical instances involving a handful of reallocation epochs in which outcomes are measured in batches, we develop a computation-driven adaptive experimentation framework that can flexibly handle batching. Our main observation is that normal approximations, which are universal in statistical inference, can also guide the design of adaptive algorithms. By deriving a Gaussian sequential experiment, we formulate a dynamic program that can leverage prior information on average rewards. Instead of the typical theory-driven paradigm, we leverage computational tools and empirical benchmarking for algorithm development. In particular, our empirical analysis highlights a simple yet effective algorithm, Residual Horizon Optimization, which iteratively solves a planning problem using stochastic gradient descent. Our approach significantly improves statistical power over standard methods, even when compared to Bayesian bandit algorithms (e.g., Thompson sampling) that require full distributional knowledge of individual rewards. Overall, we expand the scope of adaptive experimentation to settings that are difficult for standard methods, involving limited adaptivity, low signal-to-noise ratio, and unknown reward distributions.
Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation
Venue: arXiv cs.CV 2023
Year: 2023

Adversarial Robustness Distillation (ARD) is a promising task to solve the issue of limited adversarial robustness of small capacity models while optimizing the expensive computational costs of Adversarial Training (AT). Despite the good robust performance, the existing ARD methods are still impractical to deploy in natural high-security scenes due to these methods rely entirely on original or publicly available data with a similar distribution. In fact, these data are almost always private, specific, and distinctive for scenes that require high robustness. To tackle these issues, we propose a challenging but significant task called Data-Free Adversarial Robustness Distillation (DFARD), which aims to train small, easily deployable, robust models without relying on data. We demonstrate that the challenge lies in the lower upper bound of knowledge transfer information, making it crucial to mining and transferring knowledge more efficiently. Inspired by human education, we design a plug-and-play Interactive Temperature Adjustment (ITA) strategy to improve the efficiency of knowledge transfer and propose an Adaptive Generator Balance (AGB) module to retain more data information. Our method uses adaptive hyperparameters to avoid a large number of parameter tuning, which significantly outperforms the combination of existing techniques. Meanwhile, our method achieves stable and reliable performance on multiple benchmarks.
Covert Communication over Two Types of Additive Noise Channels
Venue: arXiv cs.IT 2023
Year: 2023

We extend previous results on covert communication over the additive white Gaussian noise channel to two other types of additive noise channels. The first is the Gaussian channel with memory, where the noise sequence is a Gaussian vector with an arbitrary invertible covariance matrix. We show that the fundamental limit for covert communication over such a channel is the same as over the channel with white, i.e., memoryless, Gaussian noise. The second type of channel we consider is one with memoryless generalized Gaussian noise. For such a channel we prove a general upper bound on the dominant term in the maximum number of nats that can be covertly communicated over n channel uses. When the shape parameter p of the generalized Gaussian noise distribution is in the interval (0, 1], we also prove a matching lower bound.
Data Augmentation For Label Enhancement
Venue: arXiv cs.LG 2023
Year: 2023

Label distribution (LD) uses the description degree to describe instances, which provides more fine-grained supervision information when learning with label ambiguity. Nevertheless, LD is unavailable in many real-world applications. To obtain LD, label enhancement (LE) has emerged to recover LD from logical label. Existing LE approach have the following problems: (\textbf{i}) They use logical label to train mappings to LD, but the supervision information is too loose, which can lead to inaccurate model prediction; (\textbf{ii}) They ignore feature redundancy and use the collected features directly. To solve (\textbf{i}), we use the topology of the feature space to generate more accurate label-confidence. To solve (\textbf{ii}), we proposed a novel supervised LE dimensionality reduction approach, which projects the original data into a lower dimensional feature space. Combining the above two, we obtain the augmented data for LE. Further, we proposed a novel nonlinear LE model based on the label-confidence and reduced features. Extensive experiments on 12 real-world datasets are conducted and the results show that our method consistently outperforms the other five comparing approaches.
Unlocking Layer-wise Relevance Propagation for Autoencoders
Venue: arXiv cs.LG 2023
Year: 2023

Autoencoders are a powerful and versatile tool often used for various problems such as anomaly detection, image processing and machine translation. However, their reconstructions are not always trivial to explain. Therefore, we propose a fast explainability solution by extending the Layer-wise Relevance Propagation method with the help of Deep Taylor Decomposition framework. Furthermore, we introduce a novel validation technique for comparing our explainability approach with baseline methods in the case of missing ground-truth data. Our results highlight computational as well as qualitative advantages of the proposed explainability solution with respect to existing methods.
Recommendation Systems in Libraries: an Application with Heterogeneous Data Sources
Venue: arXiv cs.IR 2023
Year: 2023

The Reading&Machine project exploits the support of digitalization to increase the attractiveness of libraries and improve the users' experience. The project implements an application that helps the users in their decision-making process, providing recommendation system (RecSys)-generated lists of books the users might be interested in, and showing them through an interactive Virtual Reality (VR)-based Graphical User Interface (GUI). In this paper, we focus on the design and testing of the recommendation system, employing data about all users' loans over the past 9 years from the network of libraries located in Turin, Italy. In addition, we use data collected by the Anobii online social community of readers, who share their feedback and additional information about books they read. Armed with this heterogeneous data, we build and evaluate Content Based (CB) and Collaborative Filtering (CF) approaches. Our results show that the CF outperforms the CB approach, improving by up to 47\% the relevant recommendations provided to a reader. However, the performance of the CB approach is heavily dependent on the number of books the reader has already read, and it can work even better than CF for users with a large history. Finally, our evaluations highlight that the performances of both approaches are significantly improved if the system integrates and leverages the information from the Anobii dataset, which allows us to include more user readings (for CF) and richer book metadata (for CB).