OpenRaiser/Intern-Atlas huggingface dataset:
End-to-End Attention-based Image Captioning
Venue: arXiv cs.CV 2021
Year: 2021

In this paper, we address the problem of image captioning specifically for molecular translation where the result would be a predicted chemical notation in InChI format for a given molecular structure. Current approaches mainly follow rule-based or CNN+RNN based methodology. However, they seem to underperform on noisy images and images with small number of distinguishable features. To overcome this, we propose an end-to-end transformer model. When compared to attention-based techniques, our proposed model outperforms on molecular datasets.
Perceptual Image Quality Assessment with Transformers
Venue: arXiv cs.CV 2021
Year: 2021

In this paper, we propose an image quality transformer (IQT) that successfully applies a transformer architecture to a perceptual full-reference image quality assessment (IQA) task. Perceptual representation becomes more important in image quality assessment. In this context, we extract the perceptual feature representations from each of input images using a convolutional neural network (CNN) backbone. The extracted feature maps are fed into the transformer encoder and decoder in order to compare a reference and distorted images. Following an approach of the transformer-based vision models, we use extra learnable quality embedding and position embedding. The output of the transformer is passed to a prediction head in order to predict a final quality score. The experimental results show that our proposed model has an outstanding performance for the standard IQA datasets. For a large-scale IQA dataset containing output images of generative model, our model also shows the promising results. The proposed IQT was ranked first among 13 participants in the NTIRE 2021 perceptual image quality assessment challenge. Our work will be an opportunity to further expand the approach for the perceptual IQA task.
Reproducibility of "FDA: Fourier Domain Adaptation forSemantic Segmentation
Venue: arXiv cs.CV 2021
Year: 2021

The following paper is a reproducibility report for "FDA: Fourier Domain Adaptation for Semantic Segmentation" published in the CVPR 2020 as part of the ML Reproducibility Challenge 2020. The original code was made available by the author. The well-commented version of the code containing all ablation studies performed derived from the original code along with WANDB integration is available at <github.com/thefatbandit/FDA> with proper instructions to execute experiments in README.
Technology Report : Robotic Localization and Navigation System for Visible Light Positioning and SLAM
Venue: arXiv cs.RO 2021
Year: 2021

Visible light positioning (VLP) technology is a promising technique as it can provide high accuracy positioning based on the existing lighting infrastructure. However, existing approaches often require dense lighting distributions. Additionally, due to complicated indoor environments, it is still challenging to develop a robust VLP. In this work, we proposed loosely-coupled multi-sensor fusion method based on VLP and Simultaneous Localization and Mapping (SLAM), with light detection and ranging (LiDAR), odometry, and rolling shutter camera. Our method can provide accurate and robust robotics localization and navigation in LED-shortage or even outage situations. The efficacy of the proposed scheme is verified by extensive real-time experiment. The results show that our proposed scheme can provide an average accuracy of 2 cm and the average computational time in low-cost embedded platforms is around 50 ms.
Editable Free-viewpoint Video Using a Layered Neural Representation
Venue: arXiv cs.CV 2021
Year: 2021

Generating free-viewpoint videos is critical for immersive VR/AR experience but recent neural advances still lack the editing ability to manipulate the visual perception for large dynamic scenes. To fill this gap, in this paper we propose the first approach for editable photo-realistic free-viewpoint video generation for large-scale dynamic scenes using only sparse 16 cameras. The core of our approach is a new layered neural representation, where each dynamic entity including the environment itself is formulated into a space-time coherent neural layered radiance representation called ST-NeRF. Such layered representation supports fully perception and realistic manipulation of the dynamic scene whilst still supporting a free viewing experience in a wide range. In our ST-NeRF, the dynamic entity/layer is represented as continuous functions, which achieves the disentanglement of location, deformation as well as the appearance of the dynamic entity in a continuous and self-supervised manner. We propose a scene parsing 4D label map tracking to disentangle the spatial information explicitly, and a continuous deform module to disentangle the temporal motion implicitly. An object-aware volume rendering scheme is further introduced for the re-assembling of all the neural layers. We adopt a novel layered loss and motion-aware ray sampling strategy to enable efficient training for a large dynamic scene with multiple performers, Our framework further enables a variety of editing functions, i.e., manipulating the scale and location, duplicating or retiming individual neural layers to create numerous visual effects while preserving high realism. Extensive experiments demonstrate the effectiveness of our approach to achieve high-quality, photo-realistic, and editable free-viewpoint video generation for dynamic scenes.
SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation
Venue: arXiv cs.CV 2021
Year: 2021

State-of-the-art semantic or instance segmentation deep neural networks (DNNs) are usually trained on a closed set of semantic classes. As such, they are ill-equipped to handle previously-unseen objects. However, detecting and localizing such objects is crucial for safety-critical applications such as perception for automated driving, especially if they appear on the road ahead. While some methods have tackled the tasks of anomalous or out-of-distribution object segmentation, progress remains slow, in large part due to the lack of solid benchmarks; existing datasets either consist of synthetic data, or suffer from label inconsistencies. In this paper, we bridge this gap by introducing the "SegmentMeIfYouCan" benchmark. Our benchmark addresses two tasks: Anomalous object segmentation, which considers any previously-unseen object category; and road obstacle segmentation, which focuses on any object on the road, may it be known or unknown. We provide two corresponding datasets together with a test suite performing an in-depth method analysis, considering both established pixel-wise performance metrics and recent component-wise ones, which are insensitive to object sizes. We empirically evaluate multiple state-of-the-art baseline methods, including several models specifically designed for anomaly / obstacle segmentation, on our datasets and on public ones, using our test suite. The anomaly and obstacle segmentation results show that our datasets contribute to the diversity and difficulty of both data landscapes.
Forming Ensembles at Runtime: A Machine Learning Approach
Venue: arXiv cs.LG 2021
Year: 2021

Smart system applications (SSAs) built on top of cyber-physical and socio-technical systems are increasingly composed of components that can work both autonomously and by cooperating with each other. Cooperating robots, fleets of cars and fleets of drones, emergency coordination systems are examples of SSAs. One approach to enable cooperation of SSAs is to form dynamic cooperation groups-ensembles-between components at runtime. Ensembles can be formed based on predefined rules that determine which components should be part of an ensemble based on their current state and the state of the environment (e.g., "group together 3 robots that are closer to the obstacle, their battery is sufficient and they would not be better used in another ensemble"). This is a computationally hard problem since all components are potential members of all possible ensembles at runtime. In our experience working with ensembles in several case studies the past years, using constraint programming to decide which ensembles should be formed does not scale for more than a limited number of components and ensembles. Also, the strict formulation in terms of hard/soft constraints does not easily permit for runtime self-adaptation via learning. This poses a serious limitation to the use of ensembles in large-scale and partially uncertain SSAs. To tackle this problem, in this paper we propose to recast the ensemble formation problem as a classification problem and use machine learning to efficiently form ensembles at scale.
Action in Mind: A Neural Network Approach to Action Recognition and Segmentation
Venue: arXiv cs.CV 2021
Year: 2021

Recognizing and categorizing human actions is an important task with applications in various fields such as human-robot interaction, video analysis, surveillance, video retrieval, health care system and entertainment industry. This thesis presents a novel computational approach for human action recognition through different implementations of multi-layer architectures based on artificial neural networks. Each system level development is designed to solve different aspects of the action recognition problem including online real-time processing, action segmentation and the involvement of objects. The analysis of the experimental results are illustrated and described in six articles. The proposed action recognition architecture of this thesis is composed of several processing layers including a preprocessing layer, an ordered vector representation layer and three layers of neural networks. It utilizes self-organizing neural networks such as Kohonen feature maps and growing grids as the main neural network layers. Thus the architecture presents a biological plausible approach with certain features such as topographic organization of the neurons, lateral interactions, semi-supervised learning and the ability to represent high dimensional input space in lower dimensional maps. For each level of development the system is trained with the input data consisting of consecutive 3D body postures and tested with generalized input data that the system has never met before. The experimental results of different system level developments show that the system performs well with quite high accuracy for recognizing human actions.
Resource separation in dynamic logic of propositional assignments
Venue: arXiv cs.LO 2021
Year: 2021

We extend dynamic logic of propositional assignments by adding an operator of parallel composition that is inspired by separation logics. We provide an axiomatisation via reduction axioms, thereby establishing decidability. We also prove that the complexity of both the model checking and the satisfiability problem stay in PSPACE.
LightIoT: Lightweight and Secure Communication for Energy-Efficient IoT in Health Informatics
Venue: arXiv cs.CR 2021
Year: 2021

Internet of Things (IoT) is considered as a key enabler of health informatics. IoT-enabled devices are used for in-hospital and in-home patient monitoring to collect and transfer biomedical data pertaining to blood pressure, electrocardiography (ECG), blood sugar levels, body temperature, etc. Among these devices, wearables have found their presence in a wide range of healthcare applications. These devices generate data in real-time and transmit them to nearby gateways and remote servers for processing and visualization. The data transmitted by these devices are vulnerable to a range of adversarial threats, and as such, privacy and integrity need to be preserved. In this paper, we present LightIoT, a lightweight and secure communication approach for data exchanged among the devices of a healthcare infrastructure. LightIoT operates in three phases: initialization, pairing, and authentication. These phases ensure the reliable transmission of data by establishing secure sessions among the communicating entities (wearables, gateways and a remote server). Statistical results exhibit that our scheme is lightweight, robust, and resilient against a wide range of adversarial attacks and incurs much lower computational and communication overhead for the transmitted data in the presence of existing approaches.
Graph-Aware Evolutionary Algorithms for Influence Maximization
Venue: arXiv cs.NE 2021
Year: 2021

Social networks represent nowadays in many contexts the main source of information transmission and the way opinions and actions are influenced. For instance, generic advertisements are way less powerful than suggestions from our contacts. However, this process hugely depends on the influence of people who disseminate these suggestions. Therefore modern marketing often involves paying some targeted users, or influencers, for advertising products or ideas. Finding the set of nodes in a social network that lead to the highest information spread -- the so-called Influence Maximization (IM) problem -- is therefore a pressing question and as such it has recently attracted a great research interest. In particular, several approaches based on Evolutionary Algorithms (EAs) have been proposed, although they are known to scale poorly with the graph size. In this paper, we tackle this limitation in two ways. Firstly, we use approximate fitness functions to speed up the EA. Secondly, we include into the EA various graph-aware mechanisms, such as smart initialization, custom mutations and node filtering, to facilitate the EA convergence. Our experiments show that the proposed modifications allow to obtain a relevant runtime gain and also improve, in some cases, the spread results.
SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models
Venue: arXiv cs.CV 2021
Year: 2021

Single image super-resolution (SISR) aims to reconstruct high-resolution (HR) images from the given low-resolution (LR) ones, which is an ill-posed problem because one LR image corresponds to multiple HR images. Recently, learning-based SISR methods have greatly outperformed traditional ones, while suffering from over-smoothing, mode collapse or large model footprint issues for PSNR-oriented, GAN-driven and flow-based methods respectively. To solve these problems, we propose a novel single image super-resolution diffusion probabilistic model (SRDiff), which is the first diffusion-based model for SISR. SRDiff is optimized with a variant of the variational bound on the data likelihood and can provide diverse and realistic SR predictions by gradually transforming the Gaussian noise into a super-resolution (SR) image conditioned on an LR input through a Markov chain. In addition, we introduce residual prediction to the whole framework to speed up convergence. Our extensive experiments on facial and general benchmarks (CelebA and DIV2K datasets) show that 1) SRDiff can generate diverse SR results in rich details with state-of-the-art performance, given only one LR input; 2) SRDiff is easy to train with a small footprint; and 3) SRDiff can perform flexible image manipulation including latent space interpolation and content fusion.
Methodological Foundation of a Numerical Taxonomy of Urban Form
Venue: arXiv cs.CY 2021
Year: 2021

Cities are complex products of human culture, characterised by a startling diversity of visible traits. Their form is constantly evolving, reflecting changing human needs and local contingencies, manifested in space by many urban patterns. Urban Morphology laid the foundation for understanding many such patterns, largely relying on qualitative research methods to extract distinct spatial identities of urban areas. However, the manual, labour-intensive and subjective nature of such approaches represents an impediment to the development of a scalable, replicable and data-driven urban form characterisation. Recently, advances in Geographic Data Science and the availability of digital mapping products, open the opportunity to overcome such limitations. And yet, our current capacity to systematically capture the heterogeneity of spatial patterns remains limited in terms of spatial parameters included in the analysis and hardly scalable due to the highly labour-intensive nature of the task. In this paper, we present a method for numerical taxonomy of urban form derived from biological systematics, which allows the rigorous detection and classification of urban types. Initially, we produce a rich numerical characterisation of urban space from minimal data input, minimizing limitations due to inconsistent data quality and availability. These are street network, building footprint, and morphological tessellation, a spatial unit derivative of Voronoi tessellation, obtained from building footprints. Hence, we derive homogeneous urban tissue types and, by determining overall morphological similarity between them, generate a hierarchical classification of urban form. After framing and presenting the method, we test it on two cities - Prague and Amsterdam - and discuss potential applications and further developments.
Compactness of Hashing Modes and Efficiency beyond Merkle Tree
Venue: arXiv cs.CR 2021
Year: 2021

We revisit the classical problem of designing optimally efficient cryptographically secure hash functions. Hash functions are traditionally designed via applying modes of operation on primitives with smaller domains. The results of Shrimpton and Stam (ICALP 2008), Rogaway and Steinberger (CRYPTO 2008), and Mennink and Preneel (CRYPTO 2012) show how to achieve optimally efficient designs of $2n$-to-$n$-bit compression functions from non-compressing primitives with asymptotically optimal $2^{n/2-ε}$-query collision resistance. Designing optimally efficient and secure hash functions for larger domains ($> 2n$ bits) is still an open problem. In this work we propose the new \textit{compactness} efficiency notion. It allows us to focus on asymptotically optimally collision resistant hash function and normalize their parameters based on Stam's bound from CRYPTO 2008 to obtain maximal efficiency. We then present two tree-based modes of operation -Our first construction is an \underline{A}ugmented \underline{B}inary T\underline{r}ee (ABR) mode. The design is a $(2^{\ell}+2^{\ell-1} -1)n$-to-$n$-bit hash function making a total of $(2^{\ell}-1)$ calls to $2n$-to-$n$-bit compression functions for any $\ell\geq 2$. Our construction is optimally compact with asymptotically (optimal) $2^{n/2-ε}$-query collision resistance in the ideal model. For a tree of height $\ell$, in comparison with Merkle tree, the $ABR$ mode processes additional $(2^{\ell-1}-1)$ data blocks making the same number of internal compression function calls. -While the $ABR$ mode achieves collision resistance, it fails to achieve indifferentiability from a random oracle within $2^{n/3}$ queries. $ABR^{+}$ compresses only $1$ less data block than $ABR$ with the same number of compression calls and achieves in addition indifferentiability up to $2^{n/2-ε}$ queries.
Dandelion: multiplexing Byzantine agreements to unlock blockchain performance
Venue: arXiv cs.DC 2021
Year: 2021

Permissionless blockchain protocols are known to consume an outrageous amount of computing power and suffer from a trade-off between latency and confidence in transaction confirmation. The recently proposed Algorand blockchain protocol employs Byzantine agreements and has shown transaction confirmation latency on the order of seconds. Its strong resilience to Denial-of-Service and Sybil attacks and its low computing power footprint make it a strong candidate for the venue of decentralized economies and business ecosystems across industries. Nevertheless, Algorand's throughput is still far from the requirements of such applications. In this paper, we empower Algorand's protocol by multiplexing its byzantine agreements in order to improve performance. Experiments on wide area networks with up to ten thousand nodes show a 4-fold throughput increase compared to the original Algorand protocol.
DeFiRanger: Detecting Price Manipulation Attacks on DeFi Applications
Venue: arXiv cs.CR 2021
Year: 2021

The rapid growth of Decentralized Finance (DeFi) boosts the Ethereum ecosystem. At the same time, attacks towards DeFi applications (apps) are increasing. However, to the best of our knowledge, existing smart contract vulnerability detection tools cannot be directly used to detect DeFi attacks. That's because they lack the capability to recover and understand high-level DeFi semantics, e.g., a user trades a token pair X and Y in a Decentralized EXchange (DEX). In this work, we focus on the detection of two types of new attacks on DeFi apps, including direct and indirect price manipulation attacks. The former one means that an attacker directly manipulates the token price in DEX by performing an unwanted trade in the same DEX by attacking the vulnerable DeFi app. The latter one means that an attacker indirectly manipulates the token price of the vulnerable DeFi app (e.g., a lending app). To this end, we propose a platform-independent way to recover high-level DeFi semantics by first constructing the cash flow tree from raw Ethereum transactions and then lifting the low-level semantics to high-level ones, including token trade, liquidity mining, and liquidity cancel. Finally, we detect price manipulation attacks using the patterns expressed with the recovered DeFi semantics. We have implemented a prototype named \tool{} and applied it to more than 350 million transactions. It successfully detected 432 real-world attacks in the wild. We confirm that they belong to four known security incidents and five zero-day ones. We reported our findings. Two CVEs have been assigned. We further performed an attack analysis to reveal the root cause of the vulnerability, the attack footprint, and the impact of the attack. Our work urges the need to secure the DeFi ecosystem.
Achieving Causality with Physical Clocks
Venue: arXiv cs.DC 2021
Year: 2021

Physical clocks provide more precision than applications can use. For example, a 64 bit NTP clock allows a precision of 233 picoseconds. In this paper, we focus on whether the least significant bits that are not useful to the applications could be used to track (one way) causality among events. We present PWC (Physical clock With Causality) that uses the extraneous bits in the physical clock. We show that PWC is very robust to errors in clock skew and transient errors. We show that PWC can be used as both a physical and logical clock for a typical distributed application even if just 6-9 extraneous bits (corresponding to precision of 15-120 nanoseconds) are available. Another important characteristic of PWC is that the standard integer < operation can be used to compare timestamps to deduce (one-way) causality among events. Thus, PWC is significantly more versatile than previous approaches for using the physical clock to provide causality information.
Memory-Efficient Deep Learning Inference in Trusted Execution Environments
Venue: arXiv cs.CR 2021
Year: 2021

This study identifies and proposes techniques to alleviate two key bottlenecks to executing deep neural networks in trusted execution environments (TEEs): page thrashing during the execution of convolutional layers and the decryption of large weight matrices in fully-connected layers. For the former, we propose a novel partitioning scheme, y-plane partitioning, designed to (i) provide consistent execution time when the layer output is large compared to the TEE secure memory; and (ii) significantly reduce the memory footprint of convolutional layers. For the latter, we leverage quantization and compression. In our evaluation, the proposed optimizations incurred latency overheads ranging from 1.09X to 2X baseline for a wide range of TEE sizes; in contrast, an unmodified implementation incurred latencies of up to 26X when running inside of the TEE.