Reviewerly/exHarmony huggingface dataset:
The Manifold Hypothesis for Gradient-Based Explanations

When do gradient-based explanation algorithms provide perceptually-aligned explanations? We propose a criterion: the feature attributions need to be aligned with the tangent space of the data manifold. To provide evidence for this hypothesis, we introduce a framework based on variational autoencoders that allows to estimate and generate image manifolds. Through experiments across a range of different datasets – MNIST, EMNIST, CIFAR10, X-ray pneumonia and Diabetic Retinopathy detection – we demonstrate that the more a feature attribution is aligned with the tangent space of the data, the more perceptually-aligned it tends to be. We then show that the attributions provided by popular post-hoc methods such as Integrated Gradients and SmoothGrad are more strongly aligned with the data manifold than the raw gradient. Adversarial training also improves the alignment of model gradients with the data manifold. As a consequence, we suggest that explanation algorithms should actively strive to align their explanations with the data manifold. An extended version of this paper is available at https://arxiv.org/abs/2206.07387.
AdaMTL: Adaptive Input-dependent Inference for Efficient Multi-Task Learning

Modern Augmented reality applications require performing multiple tasks on each input frame simultaneously. Multi-task learning (MTL) represents an effective approach where multiple tasks share an encoder to extract representative features from the input frame, followed by task-specific decoders to generate predictions for each task. Generally, the shared encoder in MTL models needs to have a large representational capacity in order to generalize well to various tasks and input data, which has a negative effect on the inference latency. In this paper, we argue that due to the large variations in the complexity of the input frames, some computations might be unnecessary for the output. Therefore, we introduce AdaMTL, an adaptive framework that learns task-aware inference policies for the MTL models in an input-dependent manner Specifically, we attach a task-aware lightweight policy network to the shared encoder and co-train it alongside the MTL model to recognize unnecessary computations. During runtime, our task-aware policy network decides which parts of the model to activate depending on the input frame and the target computational complexity. Extensive experiments on the PASCAL dataset demonstrate that AdaMTL reduces the computational complexity by 43% while improving the accuracy by 1.32% compared to single-task models. Combined with SOTA MTL methodologies, AdaMTL boosts the accuracy by 7.8% while improving the efficiency by 3.1 ×. When deployed on Vuzix M4000 smart glasses, AdaMTL reduces the inference latency and the energy consumption by up to 21.8% and 37.5%, respectively, compared to the static MTL model. Our code is publicly available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Physics-Informed Machine Learning for Optical Fiber Communications: Opportunities and Challenges

A comprehensive evaluation and discussion of physics-informed machine learning (PIML) which revolutionized multiple fields is conducted from the perspective of optical fiber communications. Despite the challenges, PIML provides a promising solution with a fresh perspective.
Mobile traffic classification through burst traffic statistical features

Mobile traffic classification is a topic of interest for researchers focused on improving the network capacity or for those seeking to identify potential risks to users’ privacy. In recent years, traffic classification accuracy has significantly improved thanks to machine learning techniques. These techniques allow traffic identification even if it is encrypted, as in mobile networks. In this paper, we show that it is feasible to classify mobile traffic applications with high accuracy using downlink control information (DCI) messages and machine learning. The DCI messages are collected using a sniffer located near the base station. Then we extract the statistical features of the bursts and inter-burst periods of the traffic generated by mobile applications at the physical layer. This strategy uses few features and does not require a big dataset. We have tested our approach on a 4G cellular network testbed and a commercial 4G cellular network. The results show an accuracy greater than 92% and 95% for application and category classification, respectively.
SS-TTA: Test-Time Adaption for Self-Supervised Denoising Methods

Even though image denoising has already been studied for decades, recent progress in deep learning has provided novel and considerably better results for this classical signal reconstruction problem. One of the most significant advances in recent years has been relaxing the requirement of having noise-free (clean) images in the training dataset. By leveraging self-supervised learning, recent methods already reach the reconstruction quality of classical and some supervised schemes. In this paper, we propose SS-TTA, a generic test-time adaptation policy that can be applied on top of various self-supervised denoising methods. Taking a pre-trained self-supervised denoising model and a test image as input, our SS-TTA algorithm improves the denoising performance through a proposed ’inference-guided regularization’ process. Based on experiments with three synthetic and three real noise datasets, SS-TTA improves the denoising results of several state-of-the-art self-supervised methods, outperforms recent test-time adaptation approaches, and shows promising performance with supervised models. Finally, SS-TTA also generalizes to cases where the test-time noise distribution differs from the noise distribution of training images.
Photonic networks with nanoseconds switching and control for distributed machine learning systems

To flexibly accommodate diverse cloud workloads and AI applications, a novel disaggregated DCN architecture based on photonic integrated switches prototype with low latency and high bandwidth to interconnect computing and memory nodes is presented.
FreqHPT: Frequency-aware attention and flow fusion for Human Pose Transfer

Human pose transfer is a challenging task that synthesizes images in various target poses while preserving the original appearance. This is typically achieved through aligning the source texture and supplementing it to the target pose. However, most of previous alignment methods only rely on either attention or flow, thereby failing to fully leverage distinctive strengths of these two methods. Moreover, the receptive field of these methods is generally limited in supplementation, resulting in the lack of global texture consistency. To address this issue, observing that attention and flow exhibit distinct characteristics in terms of their frequency distribution, Frequency-aware Human Pose Transfer (FreqHPT) is proposed in this paper. FreqHPT investigates the complementarity between attention and flow from the frequency perspective for improving texture-preserving pose transfer. To this end, FreqHPT first transforms the features from attention and flow into the wavelet domain and then fuses them over multi-frequency bands in an adaptive manner. Subsequently, FreqHPT globally refines the fused features in the Fourier space for texture supplement, enhancing the overall semantic consistency. Extensive experiments on the DeepFashion dataset demonstrate the superiority of FreqHPT in generating texture-preserving and realistic pose transfer images.
Structured Specification Framework for the Attacks, Weaknesses, and Vulnerabilities of Vehicle E&amp;E systems

We propose a new structured notation for vulnerability suitable for distributed developments for advanced vehicle electrical and electronics systems. In highly complex attacks, attackers synthesize opportunistically found vulnerabilities and normal behaviors of the elements of the target system to compromise the target. They combine the normal behaviors of elements and attack the vulnerabilities of elements to construct a chain of attack behaviors. We introduced the "attack behavior block" and "normal behavior block" as the basic units for our structured specification to describe such attacks. We demonstrate that a directed graph of both blocks can describe an attack action. In addition, each organization participating in distributed development has a limited target system scope. To provide a consistent view of cyberattacks for each organization, we introduced an "assumed attack behavior block" for information sharing from the upper to a lower process and a "synthetic attack behavior block" for information sharing in the opposite direction. We examined the applicability of our notation by applying this notation to a publicly available cyber-attack case.
ST-RoomNet: Learning Room Layout Estimation From Single Image Through Unsupervised Spatial Transformations

Room layout estimation is an important task for the 3D reconstruction of indoor scenes and augmented reality applications. The layout of the room is usually estimated by predicting the keypoints of the room corners, room planer segmentation (floor, ceiling, right, left, and front walls), or line detection. In this paper, we propose a novel way to estimate the room layout from monocular RGB images using spatial transformation networks (STN). Since it is commonly known that the room has a cuboid layout, we train a convolutional neural network to predict unsupervised perspective transformation parameters that can transform a reference cuboid layout to the required room layout based on the deep features of the input room image. We show that the proposed method is simple and efficient in learning the room layout without the need to perform segmentation, line detection, or keypoint estimation. We test the proposed method on two challenging benchmarks; LSUN Room Layout and Hedau dataset showing that our method can achieve pixel accuracy errors of 5.24% on LSUN and 7.10% on Hedau at a speed of (10∼15 fps) outperforming the state-of-the-art methods in room layout estimation task. Paper’s code is available at: https://github.com/lab231/ST-RoomNet/.
Zero-shot Unsupervised Transfer Instance Segmentation

Segmentation is a core computer vision competency, with applications spanning a broad range of scientifically and economically valuable domains. To date, however, the prohibitive cost of annotation has limited the deployment of flexible segmentation models. In this work, we propose Zero-shot Unsupervised Transfer Instance Segmentation (ZUTIS), a framework that aims to meet this challenge. The key strengths of ZUTIS are: (i) no requirement for instance-level or pixel-level annotations; (ii) an ability of zero-shot transfer, i.e., no assumption on access to a target data distribution; (iii) a unified framework for semantic and instance segmentations with solid performance on both tasks compared to state-or-the art unsupervised methods. While comparing to previous work, we show ZUTIS achieves a gain of 2.2 mask AP on COCO-20K and 14.5 mIoU on ImageNet-S with 919 categories for instance and semantic segmentations, respectively. Code will be made publicly available. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Extremely Low Latency Interactive Streaming over an 802.11 Wireless Link

Globally, interactive streaming services including AR/VR streaming or cloud gaming are getting more and more attention. To support a smooth and natural experience during the service, a client’s actions should be quickly reflected on the screen in 20 ms while the video frames are generated from the server. To achieve this, the end-to-end latency should be shrunk as much as possible. While most of the works are focusing on reducing the encoding/decoding time or the volume of the video data at the application layer assuming the other layers are reliable, we try to give attention to the other layers. Considering that most streaming users prefer the use of wireless devices, it is important to study the performance of low latency streaming over wireless networks. In this work, we show that the use of general 802.11 (Wi-Fi) wireless networks might cause a long latency which impedes smooth and natural experience during extremely low latency streaming services. Especially with a poor wireless condition, the number of link-layer retransmissions increases resulting in long latency. To relieve the problem, we limit the number of link-layer retransmissions and complement its side-effect by using application layer forward error correction (AL-FEC). From the extensive evaluation on a real wireless testbed, we show that the proposed method could achieve constant low latency service even with a poor wireless condition.
Advanced LiDAR Translation for Huge Domain Gap to Handle Adverse Weather Change

Perception task is a key technology for autonomous vehicles. Among the perception tasks, image perception has achieved powerful performance in contrast to perception tasks with other sensors. This is because of a lack of data. When comparing large amounts of image data, other sensors’ data is not sufficient, particularly in LiDAR data. To deal with the lack of data problem, many translation methods have been proposed. Although most of them focus only on image translation and LiDAR translation between simulated and real data, [1] proposed by us is the first LiDAR translation method that can convert sunny data to data of adverse weather conditions while maintaining 3D point cloud information. It is possible to generate fake LiDAR data for weather changes from sunny data. However, there is a big limitation in that noisy points are generated for a huge domain gap, such as the translation between sunny and heavily foggy conditions. Thus, we propose an advanced LiDAR translation for a huge domain gap to handle adverse weather change.
BeautyREC: Robust, Efficient, and Component-Specific Makeup Transfer

In this work, we propose a Robust, Efficient, and Component-specific makeup transfer method (abbreviated as BeautyREC). A unique departure from prior methods that leverage global attention, simply concatenate features, or implicitly manipulate features in latent space, we propose a component-specific correspondence to directly transfer the makeup style of a reference image to the corresponding components (e.g., skin, lips, eyes) of a source image, making elaborate and accurate local makeup transfer. As an auxiliary, the long-range visual dependencies of Transformer are introduced for effective global makeup transfer. Instead of the commonly used cycle structure that is complex and unstable, we employ a content consistency loss coupled with a content encoder to implement efficient single-path makeup transfer. The key insights of this study are modeling component-specific correspondence for local makeup transfer, capturing long-range dependencies for global makeup transfer, and enabling efficient makeup transfer via a single-path structure.We also contribute BeautyFace, a makeup transfer dataset to supplement existing datasets. This dataset contains 3,000 faces, covering more diverse makeup styles, face poses, and races. Each face has annotated parsing map. Extensive experiments demonstrate the effectiveness of our method against state-of-the-art methods. Besides, our method is appealing as it is with only 1M parameters, outperforming the state-of-the-art methods (BeautyGAN: 8.43M, PSGAN: 12.62M, SCGAN: 15.30M, CPM: 9.24M, SSAT: 10.48M).
VideoMatt: A Simple Baseline for Accessible Real-Time Video Matting

Recently, real-time video matting has received growing attention from academia and industry as a new research area on the rise. However, most current state-of-the-art solutions are trained and evaluated on private or inaccessible matting datasets, which makes it hard for future researchers to conduct fair comparisons among different models. Moreover, most methods are built upon image matting models with various tricks across frames to boost matting quality. For real-time video matting models, simple and effective temporal modeling methods must be explored better. As a result, we first composite a new video matting benchmark that is purely based on publicly accessible datasets for training and testing. We further empirically investigate various temporal modeling methods and compare their performance in matting accuracy and inference speed. We name our method as VideoMatt: a simple and strong real-time video matting baseline model based on a newly-composited accessible benchmark. Extensive experiments show that our VideoMatt variants reach better trade-offs between inference speed and matting quality compared with other state-of-the-art methods for real-time trimap-free video matting. We release the VideoMatt benchmark at https://drive.google.com/file/d/1QT4KHeGW3YrtBs1_7zovdCwCAofQ_GIj/view?usp=sharing.
Category Differences Matter: A Broad Analysis of Inter-Category Error in Semantic Segmentation

In current evaluation schemes of semantic segmentation, metrics are calculated in such a way that all predicted classes should equally be identical to their ground truth, paying less attention to the various manifestations of the false predictions within the object category. In this work, we propose the Critical Error Rate (CER) as a supplement to the current evaluation metrics, focusing on the error rate, which reflects predictions that fall outside of the category from the ground truth. We conduct a series of experiments evaluating the behavior of different network architectures in various evaluation setups, including domain shift, the introduction of novel classes, and a mixture of these. We demonstrate the essential criteria for network generalization with those experiments. Furthermore, we ablate the impact of utilizing various class taxonomies for the evaluation of out-of-category error.
SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentaries

Soccer is more than just a game - it is a passion that transcends borders and unites people worldwide. From the roar of the crowds to the excitement of the commentators, every moment of a soccer match is a thrill. Yet, with so many games happening simultaneously, fans cannot watch them all live. Notifications for main actions can help, but lack the engagement of live commentary, leaving fans feeling disconnected. To fulfill this need, we propose in this paper a novel task of dense video captioning focusing on the generation of textual commentaries anchored with single times-tamps. To support this task, we additionally present a challenging dataset consisting of almost 37k timestamped commentaries across 715.9 hours of soccer broadcast videos. Additionally, we propose a first benchmark and baseline for this task, highlighting the difficulty of temporally anchoring commentaries yet showing the capacity to generate meaningful commentaries. By providing broadcasters with a tool to summarize the content of their video with the same level of engagement as a live game, our method could help satisfy the needs of the numerous fans who follow their team but cannot necessarily watch the live game. We believe our method has the potential to enhance the accessibility and understanding of soccer content for a wider audience, bringing the excitement of the game to more people.
Dynamic Inference Acceleration of 3D Point Cloud Deep Neural Networks Using Point Density and Entropy

This paper introduces a density- and entropy-adaptive inference acceleration method for 3D point cloud based deep neural networks. Based on the entropy of each input frame, the method first determines the number of points to be inferred. Then we apply a novel density calculation method to sample the points in the order of density of each point. Experiments on two representative 3D scene flow estimation models with the KITTI dataset show that the proposed scheme reduces inference latency by 32%c each within 0.01m of the estimation error.
An Extended Study of Human-like Behavior under Adversarial Training

Neural networks have a number of shortcomings. Amongst the severest ones is the sensitivity to distribution shifts which allows models to be easily fooled into wrong predictions by small perturbations to inputs that are often imperceivable to humans and do not have to carry semantic meaning. Adversarial training poses a partial solution to address this issue by training models on worst-case perturbations. Yet, recent work has also pointed out that the reasoning in neural networks is different from humans. Humans identify objects by shape, while neural nets mainly employ texture cues. Exemplarily, a model trained on photographs will likely fail to generalize to datasets containing sketches. Interestingly, it was also shown that adversarial training seems to favorably increase the shift toward shape bias. In this work, we revisit this observation and provide an extensive analysis of this effect on various architectures, the common ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> - and ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</inf> -training, and Transformer-based models. Further, we provide a possible explanation for this phenomenon from a frequency perspective.
FLOW: A Scalable Multi-Model Federated Learning Framework on the Wheels

The highly mobility nature of connected vehicles poses significant challenges in the research area of federated learning, and to the best of our knowledge, the existing federated learning approaches do not consider the problem of training multi-model for constantly on-the-wheel moving vehicles. To bridge this gap, we design and implement FLOW, a scalable multi-model federated learning framework for highly mobile connected vehicles, which includes three essential components: (1) a dynamic client vehicle selection algorithm to deal with problems such as signal loss or weak signals, which may prevent some vehicles from participating in the training cluster; (2) a well-designed model allocation algorithm to select appropriate vehicle computing units for specific model training tasks; (3) geofencing not independent and identically distributed (non-i.i.d) data training, which can make models more robust and generalizable to different geographic driving area. Finally, we compare the proposed framework with centralized training and explore the performance of four aggregation protocols. The experiment results demonstrated the effectiveness of FLOW for the real-world applications.
MsSDEdit: Deep Learning Image Enhancement for Automated Bounding Box Annotations in Automotive Monocular Camera Applications

Recent advances in deep learning image enhancement enable new methods to automate 2D bounding box annotations required to train object detectors based on convolutional neural networks used in automotive monocular cameras. Thus, the following paper proposes an approach to automate the generation of 2D bounding box annotated ground truth images. The implementation of the proposed approach makes use of the CARLA simulation environment and deep learning image enhancement based on stochastic differential editing. Images of a virtually modeled camera sensor are captured in the CARLA simulation environment along with ground truth data of relevant objects in the scene. A multi-step stochastic differential editing deep learning approach then transfers all generated images into the feature space of an actual camera sensor. The enhanced images and their respective ground truth data are used to train an object detection convolutional neural network. Compared to training the convolutional neural network with unenhanced simulation images only, the precision of detecting cars in actual camera images increases by 30%, and the recall improves by 20% without adding computational costs during run-time of the object detection. Hence, the proposed approach contributes to allowing further automation of training pipelines for object detection convolutional neural networks.