Reviewerly/exHarmony huggingface dataset:
Mixed Abilities and Varied Experiences: a group autoethnography of a virtual summer internship

The COVID-19 pandemic forced many people to convert their daily work lives to a “virtual” format where everyone connected remotely from their home. In this new, virtual environment, accessibility barriers changed, in some respects for the better (e.g., more flexibility) and in other aspects, for the worse (e.g., problems including American Sign Language interpreters over video calls). Microsoft Research held its first cohort of all virtual interns in 2020. We the authors, full time and intern members and affiliates of the Ability Team, a research team focused on accessibility, reflect on our virtual work experiences as a team consisting of members with a variety of abilities, positions, and seniority during the summer intern season. Through our autoethnographic method, we provide a nuanced view into the experiences of a mixed-ability, virtual team, and how the virtual setting affected the team’s accessibility. We then reflect on these experiences, noting the successful strategies we used to promote access and the areas in which we could have further improved access. Finally, we present guidelines for future virtual mixed-ability teams looking to improve access.
Eyes Tell All: Irregular Pupil Shapes Reveal GAN-Generated Faces

Generative adversarial network (GAN) generated high-realistic human faces are visually challenging to discern from real ones. They have been used as profile images for fake social media accounts, which leads to high negative social impacts. In this work, we show that GAN-generated faces can be exposed via irregular pupil shapes. This phenomenon is caused by the lack of physiological constraints in the GAN models. We demonstrate that such artifacts exist widely in high-quality GAN-generated faces. We design an automatic method to segment the pupils from the eyes and analyze their shapes to distinguish GAN-generated faces from real ones. Qualitative and quantitative evaluations of our method on the Flickr-Faces-HQ dataset and a StyleGAN2 generated face dataset demonstrate the effectiveness and simplicity of our method.
Interpretable data science for decision making

This paper describes the foundations of interpretable data science for decision making and serves as an editorial to the corresponding special issue. Interpretable data science analyzes data that summarizes domain relationships to produce knowledge that is readily understandable by human decision makers. To this end, we contextualize the current role of interpretable data science for improved business decision making and introduce the notion of an interpretable decision support system (iDSS). We discuss five underlying characteristics of iDSS, i.e., performance, scalability, comprehensibility, justifiability and actionability. This paper further zooms in on pertinent data science decisions in the input, processing and output stage when designing iDSS. For each of the contributing papers in this special issue, we describe their major contributions to the field of interpretable data science for decision making.
Anomaly Detection in Medical Imaging - A Mini Review

The increasing digitization of medical imaging enables machine learning based improvements in detecting, visualizing and segmenting lesions, easing the workload for medical experts. However, supervised machine learning requires reliable labelled data, which is is often difficult or impossible to collect or at least time consuming and thereby costly. Therefore methods requiring only partly labeled data (semi-supervised) or no labeling at all (unsupervised methods) have been applied more regularly. Anomaly detection is one possible methodology that is able to leverage semi-supervised and unsupervised methods to handle medical imaging tasks like classification and segmentation. This paper uses a semi-exhaustive literature review of relevant anomaly detection papers in medical imaging to cluster into applications, highlight important results, establish lessons learned and give further advice on how to approach anomaly detection in medical imaging. The qualitative analysis is based on google scholar and 4 different search terms, resulting in 120 different analysed papers. The main results showed that the current research is mostly motivated by reducing the need for labelled data. Also, the successful and substantial amount of research in the brain MRI domain shows the potential for applications in further domains like OCT and chest X-ray.
Taxim: An Example-Based Simulation Model for GelSight Tactile Sensors

Simulation is widely used in robotics for system verification and large-scale data collection. However, simulating sensors, including tactile sensors, has been a long-standing challenge. In this letter, we propose Taxim, a realistic and high-speed simulation model for a vision-based tactile sensor, GelSight W. Yuan <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al</i> . A GelSight sensor uses a piece of soft elastomer as the medium of contact and embeds optical structures to capture the deformation of the elastomer, which infers the geometry and forces applied at the contact surface. We propose an example-based method for simulating GelSight: we simulate the optical response to the deformation with a polynomial look-up table. This table maps the contact geometries to pixel intensity sampled by the embedded camera. In order to simulate the surface markers’ motion that is caused by the surface stretch of the elastomer, we apply the linear displacement relationship and the superposition principle. The simulation model is calibrated with less than 100 data points from a real sensor. The example-based approach enables the model to easily migrate to other GelSight sensors or its variations. To the best of our knowledge, our simulation framework is the first to incorporate <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">marker motion field simulation</i> that derives from elastomer deformation together with the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">optical simulation</i> , creating a comprehensive and computationally efficient tactile simulation framework. Experiments reveal that our optical simulation has the lowest pixel-wise intensity errors compared to prior work and can run online with CPU computing. Our code and supplementary materials are open-sourced at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/CMURoboTouch/Taxim</uri> .
Robust and discrete matrix factorization hashing for cross-modal retrieval

Hashing based methods have gained great success for cross-modal similarity search, due to its fast query speed and low storage cost. However, there are some challenging problems that need to be further solved: 1) Many approaches are sensitive to noises and outliers, because ℓ2 norm is utilized in the objective function, the error may be amplified. 2) Most existing methods take relaxation or rounding scheme to generate binary codes, causing a large quantization loss. 3) Many supervised cross-media algorithms usually take a large n×n matrix to preserve the similarity relationship, leading to large calculation and making them unscalable. To mitigate these challenges, we develop a novel cross-media search algorithm, i.e., robust and discrete matrix factorization hashing, dubbed RDMH. The method takes a two-step strategy. In the first phase, the ℓ2,1 norm is utilized to improve the robustness, which makes our model not sensitive to noises and outliers. We can learn the hash codes directly by the proposed discrete optimization method instead of relaxation scheme, avoiding the large quantization loss. Moreover, RDMH correlates the hash codes and semantic labels directly instead of manipulating the large similarity matrix. In the second phase, we propose an autoencoder strategy to learn the hash functions, more valuable information can be preserved and making the hash functions more powerful. Comprehensive experiments on several databases demonstrate the superior performance and efficacy of the developed RDMH.
FedSim: Similarity guided model aggregation for Federated Learning

Federated Learning (FL) is a distributed machine learning approach in which clients contribute to learning a global model in a privacy preserved manner. Effective aggregation of client models is essential to create a generalised global model. To what extent a client is generalisable and contributing to this aggregation can be ascertained by analysing inter-client relationships. We use similarity between clients to model such relationships. We explore how similarity knowledge can be inferred from comparing client gradients, instead of inferring similarity on the basis of client data which violates the privacy-preserving constraint in FL. The similarity-guided FedSim algorithm, introduced in this paper, decomposes FL aggregation into local and global steps. Clients with similar gradients are clustered to provide local aggregations, which thereafter can be globally aggregated to ensure better coverage whilst reducing variance. Our comparative study also investigates the applicability of FedSim in both real-world datasets and on synthetic datasets where statistical heterogeneity can be controlled and studied systematically. A comparative study of FedSim with state-of-the-art FL baselines, FedAvg and FedProx, clearly shows significant performance gains. Our findings confirm that by exploiting latent inter-client similarities, FedSim’s performance is significantly better and more stable compared to both these baselines.
Relevance-guided Supervision for OpenQA with ColBERT

Abstract Systems for Open-Domain Question Answering (OpenQA) generally depend on a retriever for finding candidate passages in a large corpus and a reader for extracting answers from those passages. In much recent work, the retriever is a learned component that uses coarse-grained vector representations of questions and passages. We argue that this modeling choice is insufficiently expressive for dealing with the complexity of natural language questions. To address this, we define ColBERT-QA, which adapts the scalable neural retrieval model ColBERT to OpenQA. ColBERT creates fine-grained interactions between questions and passages. We propose an efficient weak supervision strategy that iteratively uses ColBERT to create its own training data. This greatly improves OpenQA retrieval on Natural Questions, SQuAD, and TriviaQA, and the resulting system attains state-of-the-art extractive OpenQA performance on all three datasets.
DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling

Since 2014 transfer learning has become the key driver for the improvement of spatial saliency prediction—however, with stagnant progress in the last 3-5 years. We conduct a large-scale transfer learning study which tests different ImageNet backbones, always using the same read out architecture and learning protocol adopted from DeepGaze II. By replacing the VGG19 backbone of DeepGaze II with ResNet50 features we improve the performance on saliency prediction from 78% to 85%. However, as we continue to test better ImageNet models as backbones—such as EfficientNetB5—we observe no additional improvement on saliency prediction. By analyzing the backbones further, we find that generalization to other datasets differs substantially, with models being consistently overconfident in their fixation predictions. We show that by combining multiple backbones in a principled manner a good confidence calibration on unseen datasets can be achieved. This new model "DeepGaze IIE" yields a significant leap in benchmark performance in and out-of-domain with a 15 percent point improvement over DeepGaze II to 93% on MIT1003, marking a new state of the art on the MIT/Tuebingen Saliency Benchmark in all available metrics (AUC: 88.3%, sAUC: 79.4%, CC: 82.4%).
Structural motion estimation via Hilbert transform enhanced phase-based video processing

The non-contact sensing techniques using image/video processing have flourished in recent years benefited from the rapid development of digital cameras. However, high-fidelity motion extraction from acquired image frames is still challenging. A novel motion estimation method based on phase-domain image processing, named Hilbert phase-based motion estimation, is proposed in this study to identify motions in a more accurate and efficient manner if compared to traditional phase-based motion estimation. The theoretical relationship between phase variation and physical motion is established based on the Hilbert transform; and then in order to reduce the computation cost, the forward and inverse fast Fourier transforms are integrated with the Hilbert transform. Under the proposed framework, phase variations are therefore obtained from the corresponding analytical signal. Furthermore, peak-picking procedure and the Butterworth ideal band-pass filter are employed to decompose the original video to its mono-component signal prior to identifying displacements. The proposed method is verified using a synthetic video containing a bell-shaped surface with randomly assigned motions. The proposed Hilbert phase-based motion estimation approach avoids the influence from manual parameter selection, and the correlation coefficient of the identification results can reach 99.55%. Experimental verification using a simply supported beam is also deployed, and comparison with the state of the art demonstrates the outperformance of the proposed algorithm.
Diagnosis of cadmium contamination in urban and suburban soils using visible-to-near-infrared spectroscopy

Previous studies have mostly focused on using visible-to-near-infrared spectral technique to quantitatively estimate soil cadmium (Cd) content, whereas little attention has been paid to identifying soil Cd contamination from a perspective of spectral classification. Here, we developed a framework to compare the potential of two spectral transformations (i.e., raw reflectance and continuum removal [CR]), three optimization strategies (i.e., full-spectrum, Boruta feature selection, and synthetic minority over-sampling technique [SMOTE]), and three classification algorithms (i.e., partial least squares discriminant analysis, random forest [RF], and support vector machine) for diagnosing soil Cd contamination. A total of 536 soil samples were collected from urban and suburban areas located in Wuhan City, China. Specifically, Boruta and SMOTE strategies were aimed at selecting the most informative predictors and obtaining balanced training datasets, respectively. Results indicated that soils contaminated by Cd induced decrease in spectral reflectance magnitude. Classification models developed after Boruta and SMOTE strategies out-performed to those from full-spectrum. A diagnose model combining CR preprocessing, SMOTE strategy, and RF algorithm achieved the highest validation accuracy for soil Cd (Kappa = 0.74). This study provides a theoretical reference for rapid identification of and monitoring of soil Cd contamination in urban and suburban areas.
Quantum Krylov subspace algorithms for ground- and excited-state energy estimation

Quantum Krylov subspace diagonalization (QKSD) algorithms provide a low-cost alternative to the conventional quantum phase estimation algorithm for estimating the ground- and excited-state energies of a quantum many-body system. While QKSD algorithms typically rely on using the Hadamard test for estimating Krylov subspace matrix elements of the form $\ensuremath{\langle}{\ensuremath{\phi}}_{i}|{e}^{\ensuremath{-}i\stackrel{\ifmmode \hat{}\else \^{}\fi{}}{H}\ensuremath{\tau}}|{\ensuremath{\phi}}_{j}\ensuremath{\rangle}$, the associated quantum circuits require an ancilla qubit with controlled multiqubit gates that can be quite costly for near-term quantum hardware. In this paper, we show that a wide class of Hamiltonians relevant to condensed-matter physics and quantum chemistry contain symmetries that can be exploited to avoid the use of the Hadamard test. We propose a multifidelity estimation protocol that can be used to compute such quantities, showing that our approach, when combined with efficient single-fidelity estimation protocols, provides a substantial reduction in circuit depth. In addition, we develop a unified theory of quantum Krylov subspace algorithms and present three quantum-classical algorithms for the ground- and excited-state energy estimation problems, where each algorithm provides various advantages and disadvantages in terms of total number of calls to the quantum computer, gate depth, classical complexity, and stability of the generalized eigenvalue problem within the Krylov subspace.
Psychological factors shaping public responses to COVID-19 digital contact tracing technologies in Germany

Abstract The COVID-19 pandemic has seen one of the first large-scale uses of digital contact tracing to track a chain of infection and contain the spread of a virus. The new technology has posed challenges both for governments aiming at high and effective uptake and for citizens weighing its benefits (e.g., protecting others’ health) against the potential risks (e.g., loss of data privacy). Our cross-sectional survey with repeated measures across four samples in Germany ( $$N = 4357$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mi>N</mml:mi><mml:mo>=</mml:mo><mml:mn>4357</mml:mn></mml:mrow></mml:math> ) focused on psychological factors contributing to the public adoption of digital contact tracing. We found that public acceptance of privacy-encroaching measures (e.g., granting the government emergency access to people’s medical records or location tracking data) decreased over the course of the pandemic. Intentions to use contact tracing apps—hypothetical ones or the Corona-Warn-App launched in Germany in June 2020—were high. Users and non-users of the Corona-Warn-App differed in their assessment of its risks and benefits, in their knowledge of the underlying technology, and in their reasons to download or not to download the app. Trust in the app’s perceived security and belief in its effectiveness emerged as psychological factors playing a key role in its adoption. We incorporate our findings into a behavioral framework for digital contact tracing and provide policy recommendations.
Smart Traffic Monitoring System Using Computer Vision and Edge Computing

Traffic management systems capture tremendous video data and leverage advances in video processing to detect and monitor traffic incidents. The collected data are traditionally forwarded to the traffic management center (TMC) for in-depth analysis and may thus exacerbate the network paths to the TMC. To alleviate such bottlenecks, we propose to utilize edge computing by equipping edge nodes that are close to cameras with computing resources (e.g., cloudlets). A cloudlet, with limited computing resources as compared to TMC, provides limited video processing capabilities. In this paper, we focus on two common traffic monitoring tasks, congestion detection, and speed detection, and propose a two-tier edge computing based model that takes into account of both the limited computing capability in cloudlets and the unstable network condition to the TMC. Our solution utilizes two algorithms for each task, one implemented at the edge and the other one at the TMC, which are designed with the consideration of different computing resources. While the TMC provides strong computation power, the video quality it receives depends on the underlying network conditions. On the other hand, the edge processes very high-quality video but with limited computing resources. Our model captures this trade-off. We evaluate the performance of the proposed two-tier model as well as the traffic monitoring algorithms via test-bed experiments under different weather as well as network conditions and show that our proposed hybrid edge-cloud solution outperforms both the cloud-only and edge-only solutions.
Exploring augmented reality for worker assistance versus training

This paper aims at advancing the fundamental understanding of the affordances of Augmented Reality (AR) as a workplace-based learning and training technology in supporting manual or semi-automated manufacturing tasks that involve both complex manipulation and reasoning. Between-subject laboratory experiments involving 20 participants are conducted on a real-life electro-mechanical assembly task to investigate the impacts of various modes of information delivery through AR compared to traditional training methods on task efficiency, number of errors, learning, independence, and cognitive load. The AR application is developed in Unity and deployed on HoloLens 2 headsets. Interviews with experts from industry and academia are also conducted to create new insights into the affordances of AR as a training versus assistive tool for manufacturing workers, as well as the need for intelligent mechanisms that enable adaptive and personalized interactions between workers and AR. The findings indicate that despite comparable performance between the AR and control groups in terms of task completion time, learning curve, and independence from instructions, AR dramatically decreases the number of errors compared to traditional instruction, which is sustained after the AR support is removed. Several insights drawn from the experiments and expert interviews are discussed to inform the design of future AR technologies for both training and assisting incumbent and future manufacturing workers on complex manipulation and reasoning tasks.
Continual Learning for Fake News Detection from Social Media

The prevalence of fake news over social media has a profound impact on justice, public trust and society as a whole. Although significant effort has been applied to mitigate its negative impact, our study shows that existing fake news detection algorithms may perform poorly on new data. In other words, the performance of a model trained on one dataset degrades on another and potentially vastly different dataset. Considering that in practice a deployed fake news detection system is likely to observe unseen data, it is crucial to solve this problem without re-training the model on the entire data from scratch, which would become prohibitively expensive as the data volumes grow. An intuitive solution is to further train the model on the new dataset, but our results show that this direct incremental training approach does not work, as the model only performs well on the latest dataset it is trained on, which is similar to the problem of catastrophic forgetting in the field of continual learning. Instead, in this work, (1) we first demonstrate that with only minor computational overhead, balanced performance can be restored on both existing and new datasets, by utilising Gradient Episodic Memory (GEM) and Elastic Weight Consolidation (EWC)—two techniques from continual learning. (2) We improve the algorithm of GEM so that the drop in model performance on the previous task can be further minimised. Specifically, we investigate different techniques to optimise the sampling process for GEM, as an improvement over random selection as originally designed. (3) We conduct extensive experiments on two datasets with thousands of labelled news items to verify our results.