Reviewerly/exHarmony huggingface dataset:
Beyond Short Clips: End-to-End Video-Level Learning with Collaborative Memories

The standard way of training video models entails sampling at each iteration a single clip from a video and optimizing the clip prediction with respect to the video-level label. We argue that a single clip may not have enough temporal coverage to exhibit the label to recognize, since video datasets are often weakly labeled with categorical information but without dense temporal annotations. Furthermore, optimizing the model over brief clips impedes its ability to learn long-term temporal dependencies. To overcome these limitations, we introduce a collaborative memory mechanism that encodes information across multiple sampled clips of a video at each training iteration. This enables the learning of long-range dependencies beyond a single clip. We explore different design choices for the collaborative memory to ease the optimization difficulties. Our proposed framework is end-to-end trainable and significantly improves the accuracy of video classification at a negligible computational overhead. Through extensive experiments, we demonstrate that our framework generalizes to different video architectures and tasks, outperforming the state of the art on both action recognition (e.g., Kinetics-400 & 700, Charades, Something-Something-V1) and action detection (e.g., AVA v2.1 & v2.2).
Hey! Preparing Humans to do Tasks in Self-adaptive Systems

Many self-adaptive systems benefit from human involvement, where human operators can complement the capabilities of systems (e.g., by supervising decisions, or performing adaptations and tasks involving physical changes that cannot be automated). However, insufficient preparation (e.g., lack of task context comprehension) may hinder the effectiveness of human involvement, especially when operators are unexpectedly interrupted to perform a new task. Preparatory notification of a task provided in advance can sometimes help human operators focus their attention on the forthcoming task and understand its context before task execution, hence improving effectiveness. Nevertheless, deciding when to use preparatory notification as a tactic is not obvious and entails considering different factors that include uncertainties induced by human operator behavior (who might ignore the notice message), human attributes (e.g., operator training level), and other information that refers to the state of the system and its environment. In this paper, informed by work in cognitive science on human attention and context management, we introduce a formal framework to reason about the usage of preparatory notifications in self-adaptive systems involving human operators. Our framework characterizes the effects of managing attention via task notification in terms of task context comprehension. We also build on our framework to develop an automated probabilistic reasoning technique able to determine when and in what form a preparatory notification tactic should be used to optimize system goals. We illustrate our approach in a representative scenario of human-robot collaborative goods delivery.
Quantum-Tailored Machine-Learning Characterization of a Superconducting Qubit

Machine learning (ML) is a promising approach for performing challenging quantum-information tasks such as device characterization, calibration, and control. ML models can train directly on the data produced by a quantum device while remaining agnostic to the quantum nature of the learning task. However, these generic models lack physical interpretability and usually require large datasets in order to learn accurately. Here we incorporate features of quantum mechanics in the design of our ML approach to characterize the dynamics of a quantum device and learn device parameters. This physics-inspired approach outperforms physics-agnostic recurrent neural networks trained on numerically generated and experimental data obtained from continuous weak measurement of a driven superconducting transmon qubit. This demonstration shows how leveraging domain knowledge improves the accuracy and efficiency of this characterization task, thus laying the groundwork for more scalable characterization techniques.3 MoreReceived 10 August 2021Accepted 12 November 2021DOI:https://doi.org/10.1103/PRXQuantum.2.040355Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.Published by the American Physical SocietyPhysics Subject Headings (PhySH)Research AreasMachine learningQuantum benchmarkingQuantum information processingPhysical SystemsSuperconducting qubitsQuantum Information
ICT, Disability, and Motivation: Validation of a Measurement Scale and Consequence Model for Inclusive Digital Knowledge

The use of ICT (information communication technology) as an educational resource is becoming more evident in the education systems of most countries, even more so with the COVID-19 crisis. When it comes to disability and education, ICT becomes a tool for social and educational inclusion. This study presents the validation and evaluation of a measurement scale on ICT literacy for inclusive education. In addition, based on previous literature, a conceptual model is proposed and validated through PLS (partial least squares) using a sample of 142 teachers from all educational stages. The results show that teachers’ ICT knowledge to ensure inclusion consists of five dimensions on specific needs. ICT knowledge has a positive impact on teacher motivation and ICT use. Teachers at primary and early childhood education levels have a lower motivation and use of ICT, although they have a higher knowledge of disability. The results found allow progress to be made in measuring the educational inclusion of schools and the ICT knowledge needed to ensure care and support for all people. A notable implication is the need for training on ICT and disability within educational policies.
Full-field motion and deformation measurement of high speed rotation based on temporal phase-locking and 3D-DIC

The full-field and non-contact measurement of three-dimensional motion and deformation of high-speed rotating machinery is a very challenging task in the engineering field. Two issues, the cost of high-speed camera and the decorrelation between rotated images, are hard to avoid for investigators. In this paper, a systematic solution is presented, which combines high-speed rotation image acquisition using low-speed camera with 3D digital image correlation, both low-cost, flexible and accurate. Firstly, the frequency and initial phase of high-speed rotation are calculated by FFT, and temporal phase-locked signal is generated to trigger the dual-camera asynchronously to collect the circumferential images with equal phase interval. Then a layer-by-layer matching method is proposed to search uncoded targets between two rotated images, while a binocular rotation correlating strategy to correlate the speckle subsets. Finally, the corresponded uncoded targets and speckle subsets are reconstructed in three dimension via the camera calibration results, furtherly the 3D motion and deformation fields can be obtained. Five experiments, including two measurement repeatability tests, an accuracy test, a rigid motion measurement of rotating table and a dynamic deformation measurement of ventilation fan, prove the accuracy and effectiveness of the proposed method.
The Threat of Offensive AI to Organizations

AI has provided us with the ability to automate tasks, extract information from vast amounts of data, and synthesize media that is nearly indistinguishable from the real thing. However, positive tools can also be used for negative purposes. In particular, cyber adversaries can use AI to enhance their attacks and expand their campaigns. Although offensive AI has been discussed in the past, there is a need to analyze and understand the threat in the context of organizations. For example, how does an AI-capable adversary impact the cyber kill chain? Does AI benefit the attacker more than the defender? What are the most significant AI threats facing organizations today and what will be their impact on the future? In this study, we explore the threat of offensive AI on organizations. First, we present the background and discuss how AI changes the adversary’s methods, strategies, goals, and overall attack model. Then, through a literature review, we identify 32 offensive AI capabilities which adversaries can use to enhance their attacks. Finally, through a panel survey spanning industry, government and academia, we rank the AI threats and provide insights on the adversaries.
Explicit Cross-Modal Representation Learning for Visual Commonsense Reasoning

Given a question about an image, Visual Commonsense Reasoning (VCR) needs to provide not only a correct answer, but also a rationale to justify the answer. VCR is a challenging task due to the requirement of proper semantic alignment and reasoning between the image and linguistic expression. Recent approaches offer a great promise by exploring holistic attention mechanisms or graph-based networks, but most of them do implicit reasoning and ignore the semantic dependencies among the linguistic expression. In this paper, we propose a novel explicit cross-modal representation learning network for VCR by incorporating syntactic information into the visual reasoning and natural language understanding. The proposed method enjoys several merits. First, based on a two-branch neural module network, we can do explicit crossmodal reasoning guided by the high-level syntactic structure of linguistic expression. Second, the semantic structure of the linguistic expression is incorporated into a syntactic GCN to facilitate language understanding. Third, our explicit crossmodal representation learning network can provide a traceable reasoning-flow, which offers visible fine-grained evidence of the answer and rationale. Quantitative and qualitative evaluations on the public VCR dataset demonstrate that our approach performs favorably against state-of-the-art methods. The full code for our work is available in the supplementary material.
Informative and Consistent Correspondence Mining for Cross-Domain Weakly Supervised Object Detection

Cross-domain weakly supervised object detection aims to adapt object-level knowledge from a fully labeled source domain dataset (i.e., with object bounding boxes) to train object detectors for target domains that are weakly labeled (i.e., with image-level tags). Instead of domain-level distribution matching, as popularly adopted in the literature, we propose to learn pixel-wise cross-domain correspondences for more precise knowledge transfer. It is realized through a novel cross-domain co-attention scheme trained as region competition. In this scheme, the cross-domain correspondence module seeks for informative features on the target domain image, which if warped to the source domain image, could best explain its annotations. Meanwhile, a collaborative mask generator competes to mask out the relevant target image region to make the remaining features uninformative. Such competitive learning strives to correlate the full foreground in cross-domain image pairs, revealing the accurate object extent in target domain. To alleviate the ambiguity of inter-domain correspondence learning, a domain-cycle consistency regularizer is further proposed to leverage the more reliable intra-domain correspondence. The proposed approach achieves consistent improvements over existing approaches by a considerable margin, demonstrated by the experiments on various datasets.
Target Tracking With Equality/Inequality Constraints Based on Trajectory Function of Time

This letter addresses the constrained target tracking problem based on the approach of the trajectory function of time (T-FoT). Both state equality and inequality constraints such as the trajectory geometry and width are considered, respectively. The penalty function is used in the T-FoT framework to account for the constraint. Specifically, the logarithmic barrier function is used to solve the challenging inequality constraint. Different from existing works which are mostly based on constrained Bayesian filters, our approaches make full use of constraints with little approximation. Three representative scenarios have been considered in the simulation for demonstrating the performance of our proposed approaches in comparison with existing constrained approaches based on Bayes filters.
Land, Water and Sun

Recent HCI scholarship has critiqued anthropocentrism in design as contributing to the ecological crisis. The current paper contributes theoretically and empirically to this area, adopting the lens of Socio-Ecological Relations, inspired by Environmental History and STS. We argue that a socio-ecological lens can offer practical tools to HCI and Interaction Design to overcome the separation between humanity and nature-as-a-resource. For this reason, our unit of analysis is humanity-in-nature and nature-in-humanity, and we show how this double internality becomes visible using an historical and geographical approach in the design process. Methodologically, using Situational Analysis, we analyse an Action Research project that introduced a digital platform for community radio in rural contexts. We find that Socio-Ecological Relations, performed in specific places, shape the design process and are themselves transformed by becoming part of it. We provide a set of sensitivities for operationalizing this approach for design researchers.
Learning To Disambiguate Strongly Interacting Hands via Probabilistic Per-Pixel Part Segmentation

In natural conversation and interaction, our hands often overlap or are in contact with each other. Due to the homogeneous appearance of hands, this makes estimating the 3D pose of interacting hands from images difficult. In this paper we demonstrate that self-similarity, and the resulting ambiguities in assigning pixel observations to the respective hands and their parts, is a major cause of the final 3D pose error. Motivated by this insight, we propose DIGIT, a novel method for estimating the 3D poses of two interacting hands from a single monocular image. The method consists of two interwoven branches that process the input imagery into a per-pixel semantic part segmentation mask and a visual feature volume. In contrast to prior work, we do not decouple the segmentation from the pose estimation stage, but rather leverage the per-pixel probabilities directly in the downstream pose estimation task. To do so, the part probabilities are merged with the visual features and processed via fully-convolutional layers. We experimentally show that the proposed approach achieves new state-of-the-art performance on the InterHand2.6M [33] dataset. We provide detailed ablation studies to demonstrate the efficacy of our method and to provide insights into how the modelling of pixel ownership affects 3D hand pose estimation.
Augmented reality as a part of STEM lessons

Abstract Modern teachers and the managers of educational establishments have big challenge – to organize the lectures and the study process so that the students are provided with the necessary skills and to meet their educational needs, as well as their parents’ expectations. An integrated lesson that is developed with a synthesis of information on different educational subjects stimulates the students’ analytic thinking, which in exchange boosts the integral perception of the reality. That is why STEM lessons are becoming more and more popular among the educators, as far as the specifics of this approach can really solve a lot of tasks. Modern IT-developments are an appropriate T – component during STEM lessons and while developing STEM projects. One of these up-to-date trends is called Augmented Reality (AR), which allows visualizing the study material and improves its reception and memorizing. The purpose of the research is development of a mobile app (on Android) for STEM lessons, designed for visualization of the chemical structure of organic, biologically active substances, which can be used by the teacher and students in order to carry out integrated lessons of biochemical area; revealing key specifics of using STEM technologies while studying natural sciences.
Multi-Objective Optimization of Deadline and Budget-Aware Workflow Scheduling in Uncertain Clouds

Cloud technologies are being used nowadays to cope with the increased computing and storage requirements of services and applications.Nevertheless, decisions about resources to be provisioned and the corresponding scheduling plans are far from being easily made especially because of the variability and uncertainty affecting workload demands as well as technological infrastructure performance.In this paper we address these issues by formulating a multi-objective constrained optimization problem aimed at identifying the optimal scheduling plans for scientific workflows to be deployed in uncertain cloud environments.In particular, we focus on minimizing the expected workflow execution time and monetary cost under probabilistic constraints on deadline and budget.According to the proposed approach, this problem is solved offline, that is, prior to workflow execution, with the intention of allowing cloud users to choose the plan of the Pareto optimal set satisfying their requirements and preferences.The analysis of the combined effects of cloud uncertainty and probabilistic constraints has shown that the solutions of the optimization problem are strongly affected by uncertainty.Hence, to properly provision cloud resources, it is compelling to precisely quantify uncertainty and take explicitly into account its effects in the decision process.
The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architectures

In this paper, we tackle the problem of convolutional neural network design. Instead of focusing on the design of the overall architecture, we investigate a design space that is usually overlooked, i.e. adjusting the channel configurations of predefined networks. We find that this adjustment can be achieved by shrinking widened baseline networks and leads to superior performance. Based on that, we articulate the "heterogeneity hypothesis": with the same training protocol, there exists a layer-wise differentiated net-work architecture (LW-DNA) that can outperform the original network with regular channel configurations but with a lower level of model complexity.The LW-DNA models are identified without extra computational cost or training time compared with the original network. This constraint leads to controlled experiments which direct the focus to the importance of layer-wise specific channel configurations. LW-DNA models come with advantages related to overfitting, i.e. the relative relationship between model complexity and dataset size. Experiments are conducted on various networks and datasets for image classification, visual tracking and image restoration. The resultant LW-DNA models consistently outperform the baseline models. Code is available at https://github.com/ofsoundof/Heterogeneity_Hypothesis.git.
Feature-Based Style Randomization for Domain Generalization

As a recent noticeable topic, domain generalization (DG) aims to first learn a generic model on multiple source domains and then directly generalize to an arbitrary unseen target domain without any additional adaption. In previous DG models, by generating virtual data to supplement observed source domains, the data augmentation based methods have shown its effectiveness. To simulate the possible unseen domains, most of them enrich the diversity of original data via image-level style transformation. However, we argue that the potential styles are hard to be exhaustively illustrated and fully augmented due to the limited referred styles, leading the diversity could not be always guaranteed. Unlike image-level augmentation, we in this paper develop a simple yet effective feature-based style randomization module to achieve feature-level augmentation, which can produce random styles via integrating random noise into the original style. Compared with existing image-level augmentation, our feature-level augmentation favors a more goal-oriented and sample-diverse way. Furthermore, to sufficiently explore the efficacy of the proposed module, we design a novel progressive training strategy to enable all parameters of the network to be fully trained. Extensive experiments on three standard benchmark datasets, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , PACS, VLCS and Office-Home, highlight the superiority of our method compared to the state-of-the-art methods.
ACRE: Abstract Causal REasoning Beyond Covariation

Causal induction, i.e., identifying unobservable mechanisms that lead to the observable relations among variables, has played a pivotal role in modern scientific discovery, especially in scenarios with only sparse and limited data. Humans, even young toddlers, can induce causal relationships surprisingly well in various settings despite its notorious difficulty. However, in contrast to the commonplace trait of human cognition is the lack of a diagnostic benchmark to measure causal induction for modern Artificial Intelligence (AI) systems. Therefore, in this work, we introduce the Abstract Causal REasoning (ACRE) dataset for systematic evaluation of current vision systems in causal induction. Motivated by the stream of research on causal discovery in Blicket experiments, we query a visual reasoning system with the following four types of questions in either an independent scenario or an interventional scenario: direct, indirect, screening-off, and backward-blocking, intentionally going beyond the simple strategy of inducing causal relationships by covariation. By analyzing visual reasoning architectures on this testbed, we notice that pure neural models tend towards an associative strategy under their chance-level performance, whereas neuro-symbolic combinations struggle in backward-blocking reasoning. These deficiencies call for future research in models with a more comprehensive capability of causal induction.
Deeply Shape-guided Cascade for Instance Segmentation

The key to a successful cascade architecture for precise instance segmentation is to fully leverage the relationship between bounding box detection and mask segmentation across multiple stages. Although modern instance segmentation cascades achieve leading performance, they mainly make use of a unidirectional relationship, i.e., mask segmentation can benefit from iteratively refined bounding box detection. In this paper, we investigate an alternative direction, i.e., how to take the advantage of precise mask segmentation for bounding box detection in a cascade architecture. We propose a Deeply Shape-guided Cascade (DSC) for instance segmentation, which iteratively imposes the shape guidances extracted from mask prediction at previous stage on bounding box detection at current stage. It forms a bi-directional relationship between the two tasks by introducing three key components: (1) Initial shape guidance: A mask-supervised Region Proposal Network (mPRN) with the ability to generate class-agnostic masks; (2) Explicit shape guidance: A mask-guided regionof-interest (RoI) feature extractor, which employs mask segmentation at previous stage to focus feature extraction at current stage within a region aligned well with the shape of the instance-of-interest rather than a rectangular RoI; (3) Implicit shape guidance: A feature fusion operation which feeds intermediate mask features at previous stage to the bounding box head at current stage. Experimental results show that DSC outperforms the state-of-the-art instance segmentation cascade, Hybrid Task Cascade (HTC), by a large margin and achieves 51.8 box AP and 45.5 mask AP on COCO test-dev. The code is released at: https://github.com/hding2455/DSC.