Reviewerly/exHarmony huggingface dataset:
Inductive and Transductive Few-Shot Video Classification via Appearance and Temporal Alignments

We present a novel method for few-shot video classification, which performs appearance and temporal alignments. In particular, given a pair of query and support videos, we conduct appearance alignment via frame-level feature matching to achieve the appearance similarity score between the videos, while utilizing temporal order-preserving priors for obtaining the temporal similarity score between the videos. Moreover, we introduce a few-shot video classification framework that leverages the above appearance and temporal similarity scores across multiple steps, namely prototype-based training and testing as well as inductive and transductive prototype refinement. To the best of our knowledge, our work is the first to explore transductive few-shot video classification. Extensive experiments on both Kinetics and Something-Something V2 datasets show that both appearance and temporal alignments are crucial for datasets with temporal order sensitivity such as Something-Something V2. Our approach achieves similar or better results than previous methods on both datasets. Our code is available at https://github.com/VinAIResearch/fsvc-ata .
Holographic communication in 5G networks

Holographic communication refers to real-time capturing, encoding, transporting and rendering of 3D representations, anchored in space, of remote persons shown as stereoscopic images or as 3D video in extended reality (XR) headsets that deliver a visual effect similar to a hologram.
PCA-Based Knowledge Distillation Towards Lightweight and Content-Style Balanced Photorealistic Style Transfer Models

Photorealistic style transfer entails transferring the style of a reference image to another image so the result seems like a plausible photo. Our work is inspired by the ob-servation that existing models are slow due to their large sizes. We introduce PCA-based knowledge distillation to distill lightweight models and show it is motivated by the-ory. To our knowledge, this is the first knowledge dis-tillation method for photorealistic style transfer. Our ex-periments demonstrate its versatility for use with differ-ent backbone architectures, VGG and MobileNet, across six image resolutions. Compared to existing models, our top-performing model runs at speeds 5-20x faster using at most 1% of the parameters. Additionally, our dis-tilled models achieve a better balance between stylization strength and content preservation than existing models. To support reproducing our method and models, we share the code at https://github.com/chiutaiyin/PCA-Knowledge-Distillation.
Learning Pixel-Level Distinctions for Video Highlight Detection

The goal of video highlight detection is to select the most attractive segments from a long video to depict the most interesting parts of the video. Existing methods typically focus on modeling relationship between different video segments in order to learning a model that can assign highlight scores to these segments; however, these approaches do not explicitly consider the contextual dependency within individual segments. To this end, we propose to learn pixel-level distinctions to improve the video highlight detection. This pixel-level distinction indicates whether or not each pixel in one video belongs to an interesting section. The advantages of modeling such fine-level distinctions are two-fold. First, it allows us to exploit the temporal and spatial relations of the content in one video, since the distinction of a pixel in one frame is highly dependent on both the content before this frame and the content around this pixel in this frame. Second, learning the pixel-level distinction also gives a good explanation to the video highlight task regarding what contents in a highlight segment will be attractive to people. We design an encoder-decoder network to estimate the pixel-level distinction, in which we leverage the 3D convolutional neural networks to exploit the temporal context information, and further take advantage of the visual saliency to model the spatial distinction. State-of-the-art performance on three public benchmarks clearly validates the effectiveness of our framework for video highlight detection.
Label2Label: A Language Modeling Framework for Multi-attribute Learning

Objects are usually associated with multiple attributes, and these attributes often exhibit high correlations. Modeling complex relationships between attributes poses a great challenge for multi-attribute learning. This paper proposes a simple yet generic framework named Label2Label to exploit the complex attribute correlations. Label2Label is the first attempt for multi-attribute prediction from the perspective of language modeling. Specifically, it treats each attribute label as a “word” describing the sample. As each sample is annotated with multiple attribute labels, these “words” will naturally form an unordered but meaningful “sentence”, which depicts the semantic information of the corresponding sample. Inspired by the remarkable success of pre-training language models in NLP, Label2Label introduces an image-conditioned masked language model, which randomly masks some of the “word” tokens from the label “sentence” and aims to recover them based on the masked “sentence” and the context conveyed by image features. Our intuition is that the instance-wise attribute relations are well grasped if the neural net can infer the missing attributes based on the context and the remaining attribute hints. Label2Label is conceptually simple and empirically powerful. Without incorporating task-specific prior knowledge and highly specialized network designs, our approach achieves state-of-the-art results on three different multi-attribute learning tasks, compared to highly customized domain-specific methods. Code is available at https://github.com/Li-Wanhua/Label2Label .
Rethinking Model Representation - A Taxonomy of Advanced Information Visualization in Conceptual Modeling

Conceptual modeling is an integral part of computer science research and is widely adopted in industrial practices, e.g., business process and enterprise architecture management. Providing adequate and usable modeling tools is necessary for the efficient adoption of modeling languages. Meta-modeling platforms provide a rich set of functionalities and are mature in realizing state-of-the-art modeling tools. However, despite their maturity and stability, most of these platforms did not yet leverage the full extent of functionalities and the ease of exploitation and integration enabled by web technologies. Current web technologies now enable much richer, advanced opportunities for visualizing and interacting with conceptual models. However, a structured and comprehensive overview of possible information visualization techniques linked to conceptual models and modeling tools is lacking. This paper aims to fill this gap by presenting a taxonomy of advanced information visualization, albeit its generic nature, applicable to conceptual modeling. We believe this taxonomy greatly benefits researchers by providing a standard frame to position their works and for method and tool engineers to spark innovation.
Set-Supervised Action Learning in Procedural Task Videos via Pairwise Order Consistency

We address the problem of set-supervised action learning, whose goal is to learn an action segmentation model using weak supervision in the form of sets of actions occurring in training videos. Our key observation is that videos within the same task have similar ordering of actions, which can be leveraged for effective learning. Therefore, we propose an attention-based method with a new Pairwise Ordering Consistency (POC) loss that encourages that for each common action pair in two videos of the same task, the attentions of actions follow a similar ordering. Unlike existing sequence alignment methods, which misalign actions in videos with different orderings or cannot reliably separate more from less consistent orderings, our POC loss efficiently aligns videos with different action orders and is differentiable, which enables end-to-end training. In addition, it avoids the time-consuming pseudo-label generation of prior works. Our method efficiently learns the actions and their temporal locations, therefore, extends the existing attention-based action localization methods from learning one action per video to multiple actions using our POC loss along with video-level and frame-level losses. By experiments on three datasets, we demonstrate that our method significantly improves the state of the art. We also show that our method, with a small modification, can effectively address the transcript-supervised action learning task, where actions and their ordering are available during training. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Code available at https://github.com/ZijiaLewisLu/CVPR22-POC.
Region Purity-Based Local Feature Selection: A Multiobjective Perspective

In contrast to the traditional feature selection (FS), local FS (LFS) partitions the whole sample space and obtains the feature subset for each local region. However, most existing LFS algorithms lack a problem-specific objective function and instead simply apply the distance-like objective function, which limits their classification performance. In addition, obtaining a good LFS model is essentially a multiobjective optimization problem. Therefore, in this article, we propose a region purity (RP)-based LFS (RP-LFS) where, besides the proportion of the selected features and region-based distance metric, we design a novel objective function, RP, from the perspective of combining local features with classifiers. To solve the RP-LFS, an improved nondominated sorting genetic algorithm III is proposed. Specifically, a network-inspired crossover operator and a quick bit mutation are applied, which can improve the ability to search for better solutions. A regional feature sharing strategy between different local models is developed, which can preserve more effective features. Experimental studies on 11 UCI datasets and nine high-dimensional datasets validate the effectiveness of our proposed RP. In comparison with various state-of-the-art FS and LFS algorithms, RP-LFS can achieve very competitive classification accuracy while obtaining a reduced feature subset size.
SmartAccess: Attribute-Based Access Control System for Medical Records Based on Smart Contracts

Cross-organisation data sharing is challenging because all the involved organisations must agree on 'how' and 'why' the data is processed.Due to a lack of transparency, the organisations need to trust that others comply with the agreements and regulations.We propose to exploit blockchain and smart contracts technologies to define an Attribute-Based Access Control System for cross-organisation medical records sharing, coined SmartAccess.SmartAccess offers joint agreement over access policies and dynamic access control besides blockchain transparency and auditability.We leverage the Attribute-Based Access Control model to implement smart contracts.We deploy and test them on a private and permissioned blockchain, transforming the access control process into a distributed smart contract execution.This paper proposes the SmartAccess system and its application in two healthcare use cases.We introduce the threat model and perform a security analysis of the system.To demonstrate the feasibility of our proposal, we implement a proof-of-concept of the smart contracts, written in Solidity language, with a size-efficient policy representation, and analyse the complexity and scalability of the contracts' functions.Furthermore, we present performance results, measuring the latency and throughput of the transactions to execute the access control functions with different blockchain network consensus setups.We also compare the performance of the SmartAccess system against two open-source Solidity implementations of smart contract-based access control, Role-based Access Control and Access Control List.Finally, we discuss the strengths and drawbacks of our proposal.SmartAccess requires the overhead of a decentralised system, but the trade-off is transparency, regulation compliance and auditability for complex cross-organisation data sharing.
Open Challenges in Deep Stereo: the Booster Dataset

We present a novel high-resolution and challenging stereo dataset framing indoor scenes annotated with dense and accurate ground-truth disparities. Peculiar to our dataset is the presence of several specular and transparent surfaces, i.e. the main causes of failures for state-of-the-art stereo networks. Our acquisition pipeline leverages a novel deep space-time stereo framework which allows for easy and accurate labeling with sub-pixel precision. We re-lease a total of 419 samples collected in 64 different scenes and annotated with dense ground-truth disparities. Each sample include a high-resolution pair (12 Mpx) as well as an unbalanced pair (Left: 12 Mpx, Right: 1.1 Mpx). Additionally, we provide manually annotated material segmentation masks and 15K unlabeled samples. We evaluate state-of-the-art deep networks based on our dataset, highlighting their limitations in addressing the open challenges in stereo and drawing hints for future research.
EnMAP Pre-Launch and Start Phase: Mission Update

The Environmental Mapping and Analysis Program (EnMAP) is a spaceborne German hyperspectral satellite mission that aims at monitoring and characterizing the Earth's environment on a global scale. The mission is now ready to start with the sensor being by end of 2021 in Flight Acceptance Review, ready to be shipped to the launch pad in early 2022. This paper presents first an update of the mission status with recent activities and developments from the space and the ground segment. Then, an update of selected highlights of the science segment activities at launch phase are presented including preparation and if possible early results for the validation of EnMAP products, updates on EnMAP science algorithms (EnMAP-Box) developed at GFZ, online education initiative (HYPERedu), and further mission support activities such as background mission.
Travel with Wander in the Metaverse: An AI chatbot to Visit the Future Earth

We developed Wander[00l] as an experiment to discuss several visions toward the metaverse: through crowd contribution, how an AI agent can become a highly accessible storyteller, and how to link the virtual and physical world through AI-generated content (AIGC). In this artwork, we implement a hybrid AIGC and user-generated content (UGC) system to facil-itate a narrative AI chatbot, Wander, that produces interactive fiction through knowledge graphs with text messages input by users on instant messaging social platforms. On Discord and WeChat, Wander can generate science-fiction-style travelogues about the future earth, including text, style-transferred images and global coordinates (GPS) based on real-world locations (e.g. Paris). The crowd interactions with Wander are visualised on an interactive globe map in real time, documenting the players' asynchronous contributions to exploring the speculative future earth. This paper presents Wander's concept, development and user study to demonstrate how people would interact with an AI agent in a narrative context for the future metaverse.
Choquet Integral-Based Aggregation for the Analysis of Anomalies Occurrence in Sustainable Transportation Systems

Anomaly detection is one of the most important problems of modern data science due to the threat to the security of information systems as well as their users. This applies in particular to logistic data, which is used to predict costs, times, and organization of travel routes. Data anomalies may endanger the welfare and safety of transport users, goods, handling companies, and consumers. Moreover, they contribute to the overexploitation of the natural environment. Therefore, it is extremely important to find methods that are responsible for their effective detection. The desired approach may be the Choquet integral and its extensions, which in various applications have proven that with their help it is possible to efficiently increase the quality of the classification measured, for example, with the help of the accuracy. Due to the fact that the Choquet integral is resistant to data fluctuations and takes into account the quality (significance) of the information source, it appears to be an effective proposition for the final determination of what data, or more precisely, which records can be considered anomalous. The innovative approach to analyze transport data has not been used before. This article considers four publicly available databases covering different fields of application of transport systems. In a series of comprehensive numerical experiments, the Choquet integral-based approach has proven high efficiency for each of them. Moreover, we made a comparative analysis of the solutions before applying the Choquet integral and the results after its application.
Error Detection Architectures for Hardware/Software Co-Design Approaches of Number-Theoretic Transform

Number-theoretic transform (NTT) is an efficient polynomial multiplication technique of lattice-based post-quantum cryptography including Kyber which is standardized as the NIST key encapsulation mechanism (KEM) in 2022. Prominent NTT architectures have recently been implemented on hardware/software coprocessors. In this article, we introduce new error detection schemes embedded efficiently in the NTT accelerator architecture, detecting both transient and permanent faults. By encoding the operands with two approaches, i.e., negating and swapping, we detect the faults in such constructions after recomputing and decoding. Through simulation, our schemes show high error coverage for the stuck-at fault model. Moreover, we implement the schemes on field-programmable gate array (FPGA) and assure that acceptable overhead is achieved for performance and implementation metrics. The low overhead and high efficiency of our schemes make them suitable for various constrained usage models. Additionally, our schemes are also applicable to similar classical and post-quantum sub-blocks to obtain more reliable respective hardware constructions.
Double-Layered Dual-Syndrome Trellis Codes Utilizing Channel Knowledge for Robust Steganography

Robust steganography aims to hide message in cover data with high security and guarantee the success of its message extraction although it is disturbed in transmission channel. In this paper we propose a framework of coding scheme extended from Dual-Syndrome Trellis Codes (Dual-STCs) for robust adaptive steganography. We use the conditional probability distribution of correct stego bits conditioned on disturbed stego data as channel knowledge, and formulate error-correcting as maximizing this probability. By extending Dual-STCs to double-layered embedding, we design an iteratively decoding scheme for error-correcting two layer stego bits from their joint conditional probabilities, and strictly prove its convergence. Besides, we design a method to estimate these probability distributions from stego data pairs uploaded/downloaded from the lossy transmission channel. The channel knowledge can also be used by steganographer, and we propose a universal method to revise steganographic distortion values for higher robustness under the guidance of the channel knowledge. Compared with existing coding methods for robust steganography, our method can make use of channel knowledge to improve error correcting ability and meanwhile maintain high security, which is demonstrated by experimental results.
Exploring Implicit Domain-Invariant Features for Domain Adaptive Object Detection

Recent researches have made a great progress in domain adaptive object detectors. These detectors aim to learn explicit domain-invariant features by adversarially mitigating domain divergence and simultaneously optimizing source risks. However, an inherent problem is that they ignore the informative knowledge implied in domain-specific features, which is recognized as implicit domain-invariant feature. This is mainly caused by the multimode structure underlying target distribution, characterized by various scales and categories of objects in target images. To solve that, we propose the Implicit Domain-invariant Faster R-CNN (IDF) by using non-adversarial domain discriminator, dual attention mechanism and selective feature perception. This idea is implemented on the Faster R-CNN backbone, but with an improved architecture of two branches, i.e. domain-invariant branch and domain-specific branch. The former can clearly learn explicit domain adaptive features w.r.t. easy samples, while the latter aims to learn implicit domain-invariant features w.r.t. hard samples. Experiments on numerous benchmark datasets, including the Cityscapes, Foggy Cityscapes, KITTI and SIM10K, show the superiority of our IDF over other state-of-the-art domain adaptive object detectors. The demo code is released in <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/sea123321/IDF</uri> .
Function Memory Optimization for Heterogeneous Serverless Platforms with CPU Time Accounting

Serverless Function-as-a-Service (FaaS) platforms often abstract the underlying infrastructure configuration into the single option of specifying a function's memory reservation size. This resource abstraction of coupling configurations options (e.g. vCPUs, memory, disk), combined with the lack of profiling, leaves developers to make ad hoc decisions on how to configure functions. Solutions are needed to mitigate exhaustive brute force searches of large parameter input spaces to find optimal configurations which can incur high costs. To address these challenges, we propose CPU Time Accounting Memory Selection (CPU-TAMS). CPU-TAMS is a workload agnostic memory selection method that utilizes CPU time accounting principles and regression modeling to recommend memory settings that reduce function runtime and subsequently, cost. Comparing CPU-TAMS to eight existing selection methods, we find that CPU-TAMS finds maximum value memory settings with only 8% runtime and 5% cost error compared to brute force testing while only requiring a single profiling run to evaluate function resource requirements. We adapt CPU-TAMS for use on four commercial FaaS platforms demonstrating efficacy to optimize function memory configurations where platforms feature heterogeneous infrastructure management policies.
Timestamp-Supervised Action Segmentation with Graph Convolutional Networks

We introduce a novel approach for temporal activity segmentation with timestamp supervision. Our main contribution is a graph convolutional network, which is learned in an end-to-end manner to exploit both frame features and connections between neighboring frames to generate dense framewise labels from sparse timestamp labels. The gener-ated dense framewise labels can then be used to train the segmentation model. In addition, we propose a framework for alternating learning of both the segmentation model and the graph convolutional model, which first initializes and then iteratively refines the learned models. Detailed experiments on four public datasets, including 50 Salads, GTEA, Breakfast, and Desktop Assembly, show that our method is superior to the multi-layer perceptron baseline, while performing on par with or better than the state of the art in temporal activity segmentation with timestamp supervision.