OpenRaiser/Intern-Atlas huggingface dataset:
Labeling Where Adapting Fails: Cross-Domain Semantic Segmentation with Point Supervision via Active Selection
Venue: arXiv cs.CV 2022
Year: 2022

Training models dedicated to semantic segmentation requires a large amount of pixel-wise annotated data. Due to their costly nature, these annotations might not be available for the task at hand. To alleviate this problem, unsupervised domain adaptation approaches aim at aligning the feature distributions between the labeled source and the unlabeled target data. While these strategies lead to noticeable improvements, their effectiveness remains limited. To guide the domain adaptation task more efficiently, previous works attempted to include human interactions in this process under the form of sparse single-pixel annotations in the target data. In this work, we propose a new domain adaptation framework for semantic segmentation with annotated points via active selection. First, we conduct an unsupervised domain adaptation of the model; from this adaptation, we use an entropy-based uncertainty measurement for target points selection. Finally, to minimize the domain gap, we propose a domain adaptation framework utilizing these target points annotated by human annotators. Experimental results on benchmark datasets show the effectiveness of our methods against existing unsupervised domain adaptation approaches. The propose pipeline is generic and can be included as an extra module to existing domain adaptation strategies.
Benchmark of DNN Model Search at Deployment Time
Venue: arXiv cs.IR 2022
Year: 2022

Deep learning has become the most popular direction in machine learning and artificial intelligence. However, the preparation of training data, as well as model training, are often time-consuming and become the bottleneck of the end-to-end machine learning lifecycle. Reusing models for inferring a dataset can avoid the costs of retraining. However, when there are multiple candidate models, it is challenging to discover the right model for reuse. Although there exist a number of model sharing platforms such as ModelDB, TensorFlow Hub, PyTorch Hub, and DLHub, most of these systems require model uploaders to manually specify the details of each model and model downloaders to screen keyword search results for selecting a model. We are lacking a highly productive model search tool that selects models for deployment without the need for any manual inspection and/or labeled data from the target domain. This paper proposes multiple model search strategies including various similarity-based approaches and non-similarity-based approaches. We design, implement, and evaluate these approaches on multiple model inference scenarios, including activity recognition, image recognition, text classification, natural language processing, and entity matching. The experimental evaluation showed that our proposed asymmetric similarity-based measurement, adaptivity, outperformed symmetric similarity-based measurements and non-similarity-based measurements in most of the workloads.
Intelligent Omni-Surfaces: Reflection-Refraction Circuit Model, Full-Dimensional Beamforming, and System Implementation
Venue: arXiv cs.IT 2022
Year: 2022

The intelligent omni-surface (IOS) is a dynamic metasurface that has recently been proposed to achieve full-dimensional communications by realizing the dual function of anomalous reflection and anomalous refraction. Existing research works provide only simplified models for the reflection and refraction responses of the IOS, which do not explicitly depend on the physical structure of the IOS and the angle of incidence of the electromagnetic (EM) wave. Therefore, the available reflection-refraction models are insufficient to characterize the performance of full-dimensional communications. In this paper, we propose a complete and detailed circuit-based reflection-refraction model for the IOS, which is formulated in terms of the physical structure and equivalent circuits of the IOS elements, as well as we validate it against full-wave EM simulations. Based on the proposed circuit-based model for the IOS, we analyze the asymmetry between the reflection and transmission coefficients. Moreover, the proposed circuit-based model is utilized for optimizing the hybrid beamforming of IOS-assisted networks and hence improving the system performance. To verify the circuit-based model, the theoretical findings, and to evaluate the performance of full-dimensional beamforming, we implement a prototype of IOS and deploy an IOS-assisted wireless communication testbed to experimentally measure the beam patterns and to quantify the achievable rate. The obtained experimental results validate the theoretical findings and the accuracy of the proposed circuit-based reflection-refraction model for IOSs.
Transformer with Fourier Integral Attentions
Venue: arXiv cs.LG 2022
Year: 2022

Multi-head attention empowers the recent success of transformers, the state-of-the-art models that have achieved remarkable success in sequence modeling and beyond. These attention mechanisms compute the pairwise dot products between the queries and keys, which results from the use of unnormalized Gaussian kernels with the assumption that the queries follow a mixture of Gaussian distribution. There is no guarantee that this assumption is valid in practice. In response, we first interpret attention in transformers as a nonparametric kernel regression. We then propose the FourierFormer, a new class of transformers in which the dot-product kernels are replaced by the novel generalized Fourier integral kernels. Different from the dot-product kernels, where we need to choose a good covariance matrix to capture the dependency of the features of data, the generalized Fourier integral kernels can automatically capture such dependency and remove the need to tune the covariance matrix. We theoretically prove that our proposed Fourier integral kernels can efficiently approximate any key and query distributions. Compared to the conventional transformers with dot-product attention, FourierFormers attain better accuracy and reduce the redundancy between attention heads. We empirically corroborate the advantages of FourierFormers over the baseline transformers in a variety of practical applications including language modeling and image classification.
AdaVITS: Tiny VITS for Low Computing Resource Speaker Adaptation
Venue: arXiv cs.SD 2022
Year: 2022

Speaker adaptation in text-to-speech synthesis (TTS) is to finetune a pre-trained TTS model to adapt to new target speakers with limited data. While much effort has been conducted towards this task, seldom work has been performed for low computational resource scenarios due to the challenges raised by the requirement of the lightweight model and less computational complexity. In this paper, a tiny VITS-based TTS model, named AdaVITS, for low computing resource speaker adaptation is proposed. To effectively reduce parameters and computational complexity of VITS, an iSTFT-based wave construction decoder is proposed to replace the upsampling-based decoder which is resource-consuming in the original VITS. Besides, NanoFlow is introduced to share the density estimate across flow blocks to reduce the parameters of the prior encoder. Furthermore, to reduce the computational complexity of the textual encoder, scaled-dot attention is replaced with linear attention. To deal with the instability caused by the simplified model, instead of using the original text encoder, phonetic posteriorgram (PPG) is utilized as linguistic feature via a text-to-PPG module, which is then used as input for the encoder. Experiment shows that AdaVITS can generate stable and natural speech in speaker adaptation with 8.97M model parameters and 0.72GFlops computational complexity.
PiDRAM: An FPGA-based Framework for End-to-end Evaluation of Processing-in-DRAM Techniques
Venue: arXiv cs.AR 2022
Year: 2022

DRAM-based main memory is used in nearly all computing systems as a major component. One way of overcoming the main memory bottleneck is to move computation near memory, a paradigm known as processing-in-memory (PiM). Recent PiM techniques provide a promising way to improve the performance and energy efficiency of existing and future systems at no additional DRAM hardware cost. We develop the Processing-in-DRAM (PiDRAM) framework, the first flexible, end-to-end, and open source framework that enables system integration studies and evaluation of real PiM techniques using real DRAM chips. We demonstrate a prototype of PiDRAM on an FPGA-based platform (Xilinx ZC706) that implements an open-source RISC-V system (Rocket Chip). To demonstrate the flexibility and ease of use of PiDRAM, we implement two PiM techniques: (1) RowClone, an in-DRAM copy and initialization mechanism (using command sequences proposed by ComputeDRAM), and (2) D-RaNGe, an in-DRAM true random number generator based on DRAM activation-latency failures. Our end-to-end evaluation of RowClone shows up to 14.6X speedup for copy and 12.6X initialization operations over CPU copy (i.e., conventional memcpy) and initialization (i.e., conventional calloc) operations. Our implementation of D-RaNGe provides high throughput true random numbers, reaching 8.30 Mb/s throughput. Over the Verilog and C++ basis provided by PiDRAM, implementing the required hardware and software components, implementing RowClone end-to-end takes 198 (565) and implementing D-RaNGe end-to-end takes 190 (78) lines of Verilog (C++) code. PiDRAM is open sourced on Github: https://github.com/CMU-SAFARI/PiDRAM.
Sustaining Security and Safety in ICT: A Quest for Terminology, Objectives, and Limits
Venue: arXiv cs.CR 2022
Year: 2022

Security and safety are intertwined concepts in the world of computing. In recent years, the terms "sustainable security" and "sustainable safety" came into fashion and are being used referring to a variety of systems properties ranging from efficiency to profitability, and sometimes meaning that a product or service is good for people and planet. This leads to confusing perceptions of products where customers might expect a sustainable product to be developed without child labour, while the producer uses the term to signify that their new product uses marginally less power than the previous generation of that products. Even in research on sustainably safe and secure ICT, these different notions of terminology are prevalent. As researchers we often work towards optimising our subject of study towards one specific sustainability metric - let's say energy consumption - while being blissfully unaware of, e.g., social impacts, life-cycle impacts, or rebound effects of such optimisations. In this paper I dissect the idea of sustainable safety and security, starting from the questions of what we want to sustain, and for whom we want to sustain it. I believe that a general "people and planet" answer is inadequate here because this form of sustainability cannot be the property of a single industry sector but must be addressed by society as a whole. However, with sufficient understanding of life-cycle impacts we may very well be able to devise research and development efforts, and inform decision making processes towards the use of integrated safety and security solutions that help us to address societal challenges in the context of the climate and ecological crises, and that are aligned with concepts such as intersectionality and climate justice. Of course, these solutions can only be effective if they are embedded in societal and economic change towards more frugal uses of data and ICT.
Efficient Multi-Purpose Cross-Attention Based Image Alignment Block for Edge Devices
Venue: arXiv cs.CV 2022
Year: 2022

Image alignment, also known as image registration, is a critical block used in many computer vision problems. One of the key factors in alignment is efficiency, as inefficient aligners can cause significant overhead to the overall problem. In the literature, there are some blocks that appear to do the alignment operation, although most do not focus on efficiency. Therefore, an image alignment block which can both work in time and/or space and can work on edge devices would be beneficial for almost all networks dealing with multiple images. Given its wide usage and importance, we propose an efficient, cross-attention-based, multi-purpose image alignment block (XABA) suitable to work within edge devices. Using cross-attention, we exploit the relationships between features extracted from images. To make cross-attention feasible for real-time image alignment problems and handle large motions, we provide a pyramidal block based cross-attention scheme. This also captures local relationships besides reducing memory requirements and number of operations. Efficient XABA models achieve real-time requirements of running above 20 FPS performance on NVIDIA Jetson Xavier with 30W power consumption compared to other powerful computers. Used as a sub-block in a larger network, XABA also improves multi-image super-resolution network performance in comparison to other alignment methods.
Content Distribution based on Joint V2I and V2V Scheduling in mmWave Vehicular Networks
Venue: arXiv cs.IT 2022
Year: 2022

With the explosive growth of vehicle applications, vehicular networks based on millimeter wave (mmWave) bands have attracted interests from both academia and industry. mmWave communications are able to utilize the huge available bandwidth to provide multiple Gbps transmission rates among vehicles. In this paper, we address the content distribution scheduling problem in mmWave vehicular networks. It has been challenging for all vehicles in the same network to complete content downloading due to the limited communication resources of roadside units (RSUs) and the high mobility of vehicles. We propose a joint vehicle-to-infrastructure (V2I) and vehicle-tovehicle (V2V) scheduling scheme to minimize the total number of content distribution time slots from a global optimization perspective. In the V2I phase, the RSU serially transmits integrity content to vehicles, which are selected according to the vehicular network topology and transmission scheduling scheme. In the V2V phase, full-duplex communications and concurrent transmissions are exploited to achieve content sharing between vehicles and improve transmission efficiency. Performance evaluations demonstrate that our proposed scheme reduces the number of time slots and significantly improves system throughput when compared with other schemes, especially under large-size file transfers and a large number of vehicles.
Temporal Characterization of VR Traffic for Network Slicing Requirement Definition
Venue: arXiv cs.NI 2022
Year: 2022

Over the past few years, the concept of VR has attracted increasing interest thanks to its extensive industrial and commercial applications. Currently, the 3D models of the virtual scenes are generally stored in the VR visor itself, which operates as a standalone device. However, applications that entail multi-party interactions will likely require the scene to be processed by an external server and then streamed to the visors. However, the stringent Quality of Service (QoS) constraints imposed by VR's interactive nature require Network Slicing (NS) solutions, for which profiling the traffic generated by the VR application is crucial. To this end, we collected more than 4 hours of traces in a real setup and analyzed their temporal correlation. More specifically, we focused on the CBR encoding mode, which should generate more predictable traffic streams. From the collected data, we then distilled two prediction models for future frame size, which can be instrumental in the design of dynamic resource allocation algorithms. Our results show that even the state-of-the-art H.264 CBR mode can have significant fluctuations, which can impact the NS optimization. We then exploited the proposed models to dynamically determine the Service Level Agreement (SLA) parameters in an NS scenario, providing service with the required QoS while minimizing resource usage.
LDoS attack detection method based on traffic time-frequency characteristics
Venue: arXiv cs.CR 2022
Year: 2022

For the traditional denial-of-service attack detection methods have complex algorithms and high computational overhead, which are difficult to meet the demand of online detection; and the experimental environment is mostly a simulation platform, which is difficult to deploy in real network environment, we propose a real network environment-oriented LDoS attack detection method based on the time-frequency characteristics of traffic data. All the traffic data flowing through the Web server is obtained through the acquisition storage system, and the detection data set is constructed using pre-processing; the simple features of the flow fragments are used as input, and the deep neural network is used to learn the time-frequency domain features of normal traffic features and generate reconstructed sequences, and the LDoS attack is discriminated based on the differences between the reconstructed sequences and the input data in the time-frequency domain. The experimental results show that the proposed method can accurately detect the attack features in the flow fragments in a very short time and achieve high detection accuracy for complex and diverse LDoS attacks; since only the statistical features of the packets are used, there is no need to parse the packet data, which can be adapted to different network environments.
On the Hardness of Dominant Strategy Mechanism Design
Venue: arXiv cs.GT 2022
Year: 2022

We study the communication complexity of dominant strategy implementations of combinatorial auctions. We start with two domains that are generally considered "easy": multi-unit auctions with decreasing marginal values and combinatorial auctions with gross substitutes valuations. For both domains we have fast algorithms that find the welfare-maximizing allocation with communication complexity that is poly-logarithmic in the input size. This immediately implies that welfare maximization can be achieved in ex-post equilibrium with no significant communication cost, by using VCG payments. In contrast, we show that in both domains the communication complexity of any dominant strategy implementation that achieves the optimal welfare is polynomial in the input size. We then move on to studying the approximation ratios achievable by dominant strategy mechanisms. For multi-unit auctions with decreasing marginal values, we provide a dominant-strategy communication FPTAS. For combinatorial auctions with general valuations, we show that there is no dominant strategy mechanism that achieves an approximation ratio better than $m^{1-ε}$ that uses $poly(m,n)$ bits of communication, where $m$ is the number of items and $n$ is the number of bidders. In contrast, a \emph{randomized} dominant strategy mechanism that achieves an $O(\sqrt m)$ approximation with $poly(m,n)$ communication is known. This proves the first gap between computationally efficient deterministic dominant strategy mechanisms and randomized ones. En route, we answer an open question on the communication cost of implementing dominant strategy mechanisms for more than two players, and also solve some open problems in the area of simultaneous combinatorial auctions.
Insertion of real agents behaviors in CARLA autonomous driving simulator
Venue: arXiv cs.RO 2022
Year: 2022

The role of simulation in autonomous driving is becoming increasingly important due to the need for rapid prototyping and extensive testing. The use of physics-based simulation involves multiple benefits and advantages at a reasonable cost while eliminating risks to prototypes, drivers and vulnerable road users. However, there are two main limitations. First, the well-known reality gap which refers to the discrepancy between reality and simulation that prevents simulated autonomous driving experience from enabling effective real-world performance. Second, the lack of empirical knowledge about the behavior of real agents, including backup drivers or passengers and other road users such as vehicles, pedestrians or cyclists. Agent simulation is usually pre-programmed deterministically, randomized probabilistically or generated based on real data, but it does not represent behaviors from real agents interacting with the specific simulated scenario. In this paper we present a preliminary framework to enable real-time interaction between real agents and the simulated environment (including autonomous vehicles) and generate synthetic sequences from simulated sensor data from multiple views that can be used for training predictive systems that rely on behavioral models. Our approach integrates immersive virtual reality and human motion capture systems with the CARLA simulator for autonomous driving. We describe the proposed hardware and software architecture, and discuss about the so-called behavioural gap or presence. We present preliminary, but promising, results that support the potential of this methodology and discuss about future steps.
Self-Supervised Learning as a Means To Reduce the Need for Labeled Data in Medical Image Analysis
Venue: arXiv cs.CV 2022
Year: 2022

One of the largest problems in medical image processing is the lack of annotated data. Labeling medical images often requires highly trained experts and can be a time-consuming process. In this paper, we evaluate a method of reducing the need for labeled data in medical image object detection by using self-supervised neural network pretraining. We use a dataset of chest X-ray images with bounding box labels for 13 different classes of anomalies. The networks are pretrained on a percentage of the dataset without labels and then fine-tuned on the rest of the dataset. We show that it is possible to achieve similar performance to a fully supervised model in terms of mean average precision and accuracy with only 60\% of the labeled data. We also show that it is possible to increase the maximum performance of a fully-supervised model by adding a self-supervised pretraining step, and this effect can be observed with even a small amount of unlabeled data for pretraining.
Attention-embedded Quadratic Network (Qttention) for Effective and Interpretable Bearing Fault Diagnosis
Venue: arXiv cs.LG 2022
Year: 2022

Bearing fault diagnosis is of great importance to decrease the damage risk of rotating machines and further improve economic profits. Recently, machine learning, represented by deep learning, has made great progress in bearing fault diagnosis. However, applying deep learning to such a task still faces a major problem. A deep network is notoriously a black box. It is difficult to know how a model classifies faulty signals from the normal and the physics principle behind the classification. To solve the interpretability issue, first, we prototype a convolutional network with recently-invented quadratic neurons. This quadratic neuron empowered network can qualify the noisy bearing data due to the strong feature representation ability of quadratic neurons. Moreover, we independently derive the attention mechanism from a quadratic neuron, referred to as qttention, by factorizing the learned quadratic function in analogue to the attention, making the model with quadratic neurons inherently interpretable. Experiments on the public and our datasets demonstrate that the proposed network can facilitate effective and interpretable bearing fault diagnosis.