OpenRaiser/Intern-Atlas huggingface dataset:
Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference
Venue: arXiv cs.LG 2022
Year: 2022

Despite rapid advances in continual learning, a large body of research is devoted to improving performance in the existing setups. While a handful of work do propose new continual learning setups, they still lack practicality in certain aspects. For better practicality, we first propose a novel continual learning setup that is online, task-free, class-incremental, of blurry task boundaries and subject to inference queries at any moment. We additionally propose a new metric to better measure the performance of the continual learning methods subject to inference queries at any moment. To address the challenging setup and evaluation protocol, we propose an effective method that employs a new memory management scheme and novel learning techniques. Our empirical validation demonstrates that the proposed method outperforms prior arts by large margins. Code and data splits are available at https://github.com/naver-ai/i-Blurry.
Layer-wise Adaptive Model Aggregation for Scalable Federated Learning
Venue: arXiv cs.LG 2022
Year: 2022

In Federated Learning, a common approach for aggregating local models across clients is periodic averaging of the full model parameters. It is, however, known that different layers of neural networks can have a different degree of model discrepancy across the clients. The conventional full aggregation scheme does not consider such a difference and synchronizes the whole model parameters at once, resulting in inefficient network bandwidth consumption. Aggregating the parameters that are similar across the clients does not make meaningful training progress while increasing the communication cost. We propose FedLAMA, a layer-wise model aggregation scheme for scalable Federated Learning. FedLAMA adaptively adjusts the aggregation interval in a layer-wise manner, jointly considering the model discrepancy and the communication cost. The layer-wise aggregation method enables to finely control the aggregation interval to relax the aggregation frequency without a significant impact on the model accuracy. Our empirical study shows that FedLAMA reduces the communication cost by up to 60% for IID data and 70% for non-IID data while achieving a comparable accuracy to FedAvg.
A unifying framework for $n$-dimensional quasi-conformal mappings
Venue: arXiv cs.CG 2022
Year: 2022

With the advancement of computer technology, there is a surge of interest in effective mapping methods for objects in higher-dimensional spaces. To establish a one-to-one correspondence between objects, higher-dimensional quasi-conformal theory can be utilized for ensuring the bijectivity of the mappings. In addition, it is often desirable for the mappings to satisfy certain prescribed geometric constraints and possess low distortion in conformality or volume. In this work, we develop a unifying framework for computing $n$-dimensional quasi-conformal mappings. More specifically, we propose a variational model that integrates quasi-conformal distortion, volumetric distortion, landmark correspondence, intensity mismatch and volume prior information to handle a large variety of deformation problems. We further prove the existence of a minimizer for the proposed model and devise efficient numerical methods to solve the optimization problem. We demonstrate the effectiveness of the proposed framework using various experiments in two- and three-dimensions, with applications to medical image registration, adaptive remeshing and shape modeling.
Different Applications and Technologies of Internet of Things (IoT)
Venue: arXiv cs.CY 2022
Year: 2022

Internet of things (IoT) has significantly altered the traditional lifestyle to a highly technologically advanced society. Some of the significant transformations that have been achieved through IoT are smart homes, smart transportation, smart city, and control of pollution. A considerable number of studies have been conducted and continue to be done to increase the use of technology through IoT. Furthermore, the research about IoT has not been done fully in improving the application of technology through IoT. Besides, IoT experiences several problems that need to be considered in order to get the full capability of IoT in changing society. This research paper addresses the key applications of IoT, the architecture of IoT, and the key issues affecting IoT. In addition, the paper highlights how big data analytics is essential in improving the effectiveness of IoT in various applications within society.
FlexTOE: Flexible TCP Offload with Fine-Grained Parallelism
Venue: arXiv cs.NI 2022
Year: 2022

FlexTOE is a flexible, yet high-performance TCP offload engine (TOE) to SmartNICs. FlexTOE eliminates almost all host data-path TCP processing and is fully customizable. FlexTOE interoperates well with other TCP stacks, is robust under adverse network conditions, and supports POSIX sockets. FlexTOE focuses on data-path offload of established connections, avoiding complex control logic and packet buffering in the NIC. FlexTOE leverages fine-grained parallelization of the TCP data-path and segment reordering for high performance on wimpy SmartNIC architectures, while remaining flexible via a modular design. We compare FlexTOE on an Agilio-CX40 to host TCP stacks Linux and TAS, and to the Chelsio Terminator TOE. We find that Memcached scales up to 38% better on FlexTOE versus TAS, while saving up to 81% host CPU cycles versus Chelsio. FlexTOE provides competitive performance for RPCs, even with wimpy SmartNICs. FlexTOE cuts 99.99th-percentile RPC RTT by 3.2$\times$ and 50% versus Chelsio and TAS, respectively. FlexTOE's data-path parallelism generalizes across hardware architectures, improving single connection RPC throughput up to 2.4$\times$ on x86 and 4$\times$ on BlueField. FlexTOE supports C and XDP programs written in eBPF. It allows us to implement popular data center transport features, such as TCP tracing, packet filtering and capture, VLAN stripping, flow classification, firewalling, and connection splicing.
Multi-stable design of triangulated origami structures on cones of revolution
Venue: arXiv cs.CG 2022
Year: 2022

It is well-known that the Kresling pattern of congruent triangles can be arranged either circularly on a cylinder of revolution or in a helical way. In both cases the resulting cylindrical structures are multi-stable. We generalize these arrangements with respect to cones of revolution, where our approach allows to construct structures, which snap between conical realizations whose apex angles serve as design parameters. In this context we also figure out shaky realizations, intervals for self-intersection free realizations and an interesting property related to the cross sectional area. Finally, we analyze these origami structures with respect to their capability to snap by means of the so-called snappability index.
SLURP: Side Learning Uncertainty for Regression Problems
Venue: arXiv cs.CV 2022
Year: 2022

It has become critical for deep learning algorithms to quantify their output uncertainties to satisfy reliability constraints and provide accurate results. Uncertainty estimation for regression has received less attention than classification due to the more straightforward standardized output of the latter class of tasks and their high importance. However, regression problems are encountered in a wide range of applications in computer vision. We propose SLURP, a generic approach for regression uncertainty estimation via a side learner that exploits the output and the intermediate representations generated by the main task model. We test SLURP on two critical regression tasks in computer vision: monocular depth and optical flow estimation. In addition, we conduct exhaustive benchmarks comprising transfer to different datasets and the addition of aleatoric noise. The results show that our proposal is generic and readily applicable to various regression problems and has a low computational cost with respect to existing solutions.
PlaneRecNet: Multi-Task Learning with Cross-Task Consistency for Piece-Wise Plane Detection and Reconstruction from a Single RGB Image
Venue: arXiv cs.CV 2022
Year: 2022

Piece-wise 3D planar reconstruction provides holistic scene understanding of man-made environments, especially for indoor scenarios. Most recent approaches focused on improving the segmentation and reconstruction results by introducing advanced network architectures but overlooked the dual characteristics of piece-wise planes as objects and geometric models. Different from other existing approaches, we start from enforcing cross-task consistency for our multi-task convolutional neural network, PlaneRecNet, which integrates a single-stage instance segmentation network for piece-wise planar segmentation and a depth decoder to reconstruct the scene from a single RGB image. To achieve this, we introduce several novel loss functions (geometric constraint) that jointly improve the accuracy of piece-wise planar segmentation and depth estimation. Meanwhile, a novel Plane Prior Attention module is used to guide depth estimation with the awareness of plane instances. Exhaustive experiments are conducted in this work to validate the effectiveness and efficiency of our method.
Measuring the (Over)use of Service Workers for In-Page Push Advertising Purposes
Venue: arXiv cs.NI 2022
Year: 2022

Rich offline experience, periodic background sync, push notification functionality, network requests control, improved performance via requests caching are only a few of the functionalities provided by the Service Worker (SW) API. This new technology, supported by all major browsers, can significantly improve users' experience by providing the publisher with the technical foundations that would normally require a native application. Albeit the capabilities of this new technique and its important role in the ecosystem of Progressive Web Apps (PWAs), it is still unclear what is their actual purpose on the web, and how publishers leverage the provided functionality in their web applications. In this study, we shed light in the real world deployment of SWs, by conducting the first large scale analysis of the prevalence of SWs in the wild. We see that SWs are becoming more and more popular, with the adoption increased by 26% only within the last 5 months. Surprisingly, besides their fruitful capabilities, we see that SWs are being mostly used for In-Page Push Advertising, in 65.08% of the SWs that connect with 3rd parties. We highlight that this is a relatively new way for advertisers to bypass ad-blockers and render ads on the user's displays natively.
Real-time, low-cost multi-person 3D pose estimation
Venue: arXiv cs.CV 2022
Year: 2022

The process of tracking human anatomy in computer vision is referred to pose estimation, and it is used in fields ranging from gaming to surveillance. Three-dimensional pose estimation traditionally requires advanced equipment, such as multiple linked intensity cameras or high-resolution time-of-flight cameras to produce depth images. However, there are applications, e.g.~consumer electronics, where significant constraints are placed on the size, power consumption, weight and cost of the usable technology. Here, we demonstrate that computational imaging methods can achieve accurate pose estimation and overcome the apparent limitations of time-of-flight sensors designed for much simpler tasks. The sensor we use is already widely integrated in consumer-grade mobile devices, and despite its low spatial resolution, only 4$\times$4 pixels, our proposed Pixels2Pose system transforms its data into accurate depth maps and 3D pose data of multiple people up to a distance of 3 m from the sensor. We are able to generate depth maps at a resolution of 32$\times$32 and 3D localization of a body parts with an error of only $\approx$10 cm at a frame rate of 7 fps. This work opens up promising real-life applications in scenarios that were previously restricted by the advanced hardware requirements and cost of time-of-flight technology.
ML with HE: Privacy Preserving Machine Learning Inferences for Genome Studies
Venue: arXiv cs.CR 2022
Year: 2022

Preserving the privacy and security of big data in the context of cloud computing, while maintaining a certain level of efficiency of its processing remains to be a subject, open for improvement. One of the most popular applications epitomizing said concerns is found to be useful in genome analysis. This work proposes a secure multi-label tumor classification method using homomorphic encryption, whereby two different machine learning algorithms, SVM and XGBoost, are used to classify the encrypted genome data of different tumor types.
Action Planning for Packing Long Linear Elastic Objects into Compact Boxes with Bimanual Robotic Manipulation
Venue: arXiv cs.RO 2022
Year: 2022

In this paper, we propose a new action planning approach to automatically pack long linear elastic objects into common-size boxes with a bimanual robotic system. For that, we developed a hybrid geometric model to handle large-scale occlusions combining an online vision-based method and an offline reference template. Then, a reference point generator is introduced to automatically plan the reference poses for the predesigned action primitives. Finally, an action planner integrates these components enabling the execution of high-level behaviors and the accomplishment of packing manipulation tasks. To validate the proposed approach, we conducted a detailed experimental study with multiple types and lengths of objects and packing boxes.
FakeNewsLab: Experimental Study on Biases and Pitfalls Preventing us from Distinguishing True from False News
Venue: arXiv cs.CY 2022
Year: 2022

Misinformation posting and spreading in Social Media is ignited by personal decisions on the truthfulness of news that may cause wide and deep cascades at a large scale in a fraction of minutes. When individuals are exposed to information, they usually take a few seconds to decide if the content (or the source) is reliable, and eventually to share it. Although the opportunity to verify the rumour is often just one click away, many users fail to make a correct evaluation. We studied this phenomenon with a web-based questionnaire that was compiled by 7,298 different volunteers, where the participants were asked to mark 20 news as true or false. Interestingly, false news is correctly identified more frequently than true news, but showing the full article instead of just the title, surprisingly, does not increase general accuracy. Also, displaying the original source of the news may contribute to mislead the user in some cases, while a genuine wisdom of the crowd can positively assist individuals' ability to classify correctly. Finally, participants whose browsing activity suggests a parallel fact-checking activity, show better performance and declare themselves as young adults. This work highlights a series of pitfalls that can influence human annotators when building false news datasets, which in turn fuel the research on the automated fake news detection; furthermore, these findings challenge the common rationale of AI that suggest users to read the full article before re-sharing.
Unraveling the Hidden Environmental Impacts of AI Solutions for Environment
Venue: arXiv cs.AI 2022
Year: 2022

In the past ten years, artificial intelligence has encountered such dramatic progress that it is now seen as a tool of choice to solve environmental issues and in the first place greenhouse gas emissions (GHG). At the same time the deep learning community began to realize that training models with more and more parameters requires a lot of energy and as a consequence GHG emissions. To our knowledge, questioning the complete net environmental impacts of AI solutions for the environment (AI for Green), and not only GHG, has never been addressed directly. In this article, we propose to study the possible negative impacts of AI for Green. First, we review the different types of AI impacts, then we present the different methodologies used to assess those impacts, and show how to apply life cycle assessment to AI services. Finally, we discuss how to assess the environmental usefulness of a general AI service, and point out the limitations of existing work in AI for Green.
A Study of Multimodal Person Verification Using Audio-Visual-Thermal Data
Venue: arXiv cs.CV 2022
Year: 2022

In this paper, we study an approach to multimodal person verification using audio, visual, and thermal modalities. The combination of audio and visual modalities has already been shown to be effective for robust person verification. From this perspective, we investigate the impact of further increasing the number of modalities by adding thermal images. In particular, we implemented unimodal, bimodal, and trimodal verification systems using state-of-the-art deep learning architectures and compared their performance under clean and noisy conditions. We also compared two popular fusion approaches based on simple score averaging and the soft attention mechanism. The experiment conducted on the SpeakingFaces dataset demonstrates the superior performance of the trimodal verification system. Specifically, on the easy test set, the trimodal system outperforms the best unimodal and bimodal systems by over 50% and 18% relative equal error rates, respectively, under both the clean and noisy conditions. On the hard test set, the trimodal system outperforms the best unimodal and bimodal systems by over 40% and 13% relative equal error rates, respectively, under both the clean and noisy conditions. To enable reproducibility of the experiment and facilitate research into multimodal person verification, we made our code, pretrained models, and preprocessed dataset freely available in our GitHub repository.
Integration of Blockchain and Auction Models: A Survey, Some Applications, and Challenges
Venue: arXiv cs.CR 2022
Year: 2022

In recent years, blockchain has gained widespread attention as an emerging technology for decentralization, transparency, and immutability in advancing online activities over public networks. As an essential market process, auctions have been well studied and applied in many business fields due to their efficiency and contributions to fair trade. Complementary features between blockchain and auction models trigger a great potential for research and innovation. On the one hand, the decentralized nature of blockchain can provide a trustworthy, secure, and cost-effective mechanism to manage the auction process; on the other hand, auction models can be utilized to design incentive and consensus protocols in blockchain architectures. These opportunities have attracted enormous research and innovation activities in both academia and industry; however, there is a lack of an in-depth review of existing solutions and achievements. In this paper, we conduct a comprehensive state-of-the-art survey of these two research topics. We review the existing solutions for integrating blockchain and auction models, with some application-oriented taxonomies generated. Additionally, we highlight some open research challenges and future directions towards integrated blockchain-auction models.
Instance-Conditional Knowledge Distillation for Object Detection
Venue: arXiv cs.CV 2022
Year: 2022

Knowledge distillation has shown great success in classification, however, it is still challenging for detection. In a typical image for detection, representations from different locations may have different contributions to detection targets, making the distillation hard to balance. In this paper, we propose a conditional distillation framework to distill the desired knowledge, namely knowledge that is beneficial in terms of both classification and localization for every instance. The framework introduces a learnable conditional decoding module, which retrieves information given each target instance as query. Specifically, we encode the condition information as query and use the teacher's representations as key. The attention between query and key is used to measure the contribution of different features, guided by a localization-recognition-sensitive auxiliary task. Extensive experiments demonstrate the efficacy of our method: we observe impressive improvements under various settings. Notably, we boost RetinaNet with ResNet-50 backbone from 37.4 to 40.7 mAP (+3.3) under 1x schedule, that even surpasses the teacher (40.4 mAP) with ResNet-101 backbone under 3x schedule. Code has been released on https://github.com/megvii-research/ICD.
Learning What to Memorize: Using Intrinsic Motivation to Form Useful Memory in Partially Observable Reinforcement Learning
Venue: arXiv cs.LG 2022
Year: 2022

Reinforcement Learning faces an important challenge in partial observable environments that has long-term dependencies. In order to learn in an ambiguous environment, an agent has to keep previous perceptions in a memory. Earlier memory based approaches use a fixed method to determine what to keep in the memory, which limits them to certain problems. In this study, we follow the idea of giving the control of the memory to the agent by allowing it to have memory-changing actions. This learning mechanism is supported by an intrinsic motivation to memorize rare observations that can help the agent to disambiguate its state in the environment. Our approach is experimented and analyzed on several partial observable tasks with long-term dependencies and compared with other memory based methods.
Sparsity-Specific Code Optimization using Expression Trees
Venue: arXiv cs.PL 2022
Year: 2022

We introduce a code generator that converts unoptimized C++ code operating on sparse data into vectorized and parallel CPU or GPU kernels. Our approach unrolls the computation into a massive expression graph, performs redundant expression elimination, grouping, and then generates an architecture-specific kernel to solve the same problem, assuming that the sparsity pattern is fixed, which is a common scenario in many applications in computer graphics and scientific computing. We show that our approach scales to large problems and can achieve speedups of two orders of magnitude on CPUs and three orders of magnitude on GPUs, compared to a set of manually optimized CPU baselines. To demonstrate the practical applicability of our approach, we employ it to optimize popular algorithms with applications to physical simulation and interactive mesh deformation.
Learning to run a power network with trust
Venue: arXiv cs.AI 2022
Year: 2022

Artificial agents are promising for real-time power network operations, particularly, to compute remedial actions for congestion management. However, due to high reliability requirements, purely autonomous agents will not be deployed any time soon and operators will be in charge of taking action for the foreseeable future. Aiming at designing assistant for operators, we instead consider humans in the loop and propose an original formulation. We first advance an agent with the ability to send to the operator alarms ahead of time when the proposed actions are of low confidence. We further model the operator's available attention as a budget that decreases when alarms are sent. We present the design and results of our competition "Learning to run a power network with trust" in which we evaluate our formulation and benchmark the ability of submitted agents to send relevant alarms while operating the network to their best.