Reviewerly/exHarmony huggingface dataset:
Egocentric video co-summarization using transfer learning and refined random walk on a constrained graph

• Co-summarization for short and long duration egocentric videos • Transfer Learning with a new loss function • Constrained Video Representation Graph providing computational benefits • Random Walks with label refinements for accurate summary In this paper, we address the problem of egocentric video co-summarization. We show how a shot level accurate summary can be obtained in a time-efficient manner using random walk on a constrained graph in transfer learned feature space with label refinement. While applying transfer learning, we propose a new loss function capturing egocentric characteristics in a pre-trained ResNet on the set of auxiliary egocentric videos. Transfer learning is used to generate i) an improved feature space and ii) a set of labels to be used as seeds for the test egocentric video. A complete weighted graph is created for a test video in the new transfer learned feature space with shots as the vertices. We derive two types of cluster label constraints in form of Must-Link (ML) and Cannot-link (CL) based on the similarity of the shots. ML constraints are used to prune the complete graph which is shown to result in substantial computational advantage, especially, for the long duration videos. We derive expressions for the number of vertices and edges for the ML-constrained graph and show that this graph remains connected. Random walk is applied to obtain labels of the unmarked shots in this new graph. CL constraints are applied to refine the cluster labels. Finally, shots closest to individual cluster centres are used to build the summary. Experiments on the short duration videos as in CoSum and TVSum datasets and long duration videos as in ADL and EPIC-Kitchens datasets clearly demonstrate the advantage of our solution over several state-of-the-art methods.
Efficient Meta label correction based on Meta Learning and bi-level optimization

The design of highly accurate deep learning architectures is related to different parameters through different optimizations at different levels of the design. While the architectural design and the training procedure of the neural networks play significant role, the quantity and the quality of the available data remain also very important issues. This research aims to address the quality of the data issue: the data contain imprecise or noisy labels. Numerous approaches have been proposed in the literature to leverage noisy data in case of different noisy labels as well as the strategies and frameworks developed to use the data in practice. Recently, meta-learning based methods have shown good performance to optimize models that take into account the presence of noisy labels within the training dataset. In this work, we propose a new method called Meta label correction via Meta Learning and bi-level optimization (CO-META). It is based on extracting and providing only relevant features to enhance the performance of the label corrector network to tackle the raised problem. The obtained results demonstrate that the proposed method outperforms the state-of-the-art accuracy on the well-known image classification datasets.
Multi-modal fusion transformer for visual question answering in remote sensing

With the new generation of satellite technologies, the archives of remote sensing (RS) images are growing very fast. To make the intrinsic information of each RS image easily accessible, visual question answering (VQA) has been introduced in RS. VQA allows a user to formulate a free-form question concerning the content of RS images to extract generic information. It has been shown that the fusion of the input modalities (i.e., image and text) is crucial for the performance of VQA systems. Most of the current fusion approaches use modalityspecific representations in their fusion modules instead of joint representation learning. However, to discover the underlying relation between both the image and question modality, the model is required to learn the joint representation instead of simply combining (e.g., concatenating, adding, or multiplying) the modality-specific representations. We propose a multi-modal transformer-based architecture to overcome this issue. Our proposed architecture consists of three main modules: i) the feature extraction module for extracting the modality-specific features; ii) the fusion module, which leverages a user-defined number of multi-modal transformer layers of the VisualBERT model (VB); and iii) the classification module to obtain the answer. In contrast to recently proposed transformer-based models in RS VQA, the presented architecture (called VBFusion) is not limited to specific questions, e.g., questions concerning pre-defined objects. Experimental results obtained on the RSVQAxBEN and RSVQA-LR datasets (which are made up of RGB bands of Sentinel-2 images) demonstrate the effectiveness of VBFusion for VQA tasks in RS. To analyze the importance of using other spectral bands for the description of the complex content of RS images in the framework of VQA, we extend the RSVQAxBEN dataset to include all the spectral bands of Sentinel-2 images with 10m and 20m spatial resolution. Experimental results show the importance of utilizing these bands to characterize the land-use land-cover classes present in the images in the framework of VQA. The code of the proposed method is publicly available at <a href="https://git.tu-berlin.de/rsim/multimodal- fusion-transformer-for-vqa-in-rs">https://git.tu-berlin.de/rsim/multimodal- fusion-transformer-for-vqa-in-rs</a>.
Register identification from the unrestricted open Web using the Corpus of Online Registers of English

Abstract This article examines the automatic identification of Web registers, that is, text varieties such as news articles and reviews. Most studies have focused on corpora restricted to include only preselected classes with well-defined characteristics. These corpora feature only a subset of documents found on the unrestricted open Web, for which register identification has been particularly difficult because the range of linguistic variation on the Web is known to be substantial. As part of this study, we present the first open release of the Corpus of Online Registers of English (CORE), which is drawn from the unrestricted open Web and, currently, is the largest collection of manually annotated Web registers. Furthermore, we demonstrate that the CORE registers can be automatically identified with competitive results, with the best performance being an F1-score of 68% with the deep learning model BERT. The best performance was achieved using two modeling strategies. The first one involved modeling the registers using propagated register labels, that is, repeating the main register label along with its corresponding subregister label in a multilabel model. In the second one, we explored how the length of the document affects model performance, discovering that the beginning provided superior classification accuracy. Overall, the current study presents a systematic approach for the automatic identification of a large number of Web registers from the unrestricted Web, hence providing new pathways for future studies.
Novel View Synthesis Of Transparent Object From a Single Image

Abstract We propose a method for converting a single image of a transparent object into multi‐view photo that enables users observing the object from multiple new angles, without inputting any 3D shape. The complex light paths formed by refraction and reflection makes it challenging to compute the lighting effects of transparent objects from a new angle. We construct an encoder–decoder network for normal reconstruction and texture extraction, which enables synthesizing novel views of transparent object from a set of new views and new environment maps using only one RGB image. By simultaneously considering the optical transmission and perspective variation, our network learns the characteristics of optical transmission and the change of perspective as guidance to the conversion from RGB colours to surface normals. A texture extraction subnetwork is proposed to alleviate the contour loss phenomenon during normal map generation. We test our method using 3D objects within and without our training data, including real 3D objects that exists in our lab, and completely new environment maps that we take using our phones. The results show that our method performs better on view synthesis of transparent objects in complex scenes using only a single‐view image.
GIFAIR-FL: A Framework for Group and Individual Fairness in Federated Learning

In this paper, we propose GIFAIR-FL, a framework that imposes group and individual fairness (GIFAIR) to federated learning (FL) settings. By adding a regularization term, our algorithm penalizes the spread in the loss of client groups to drive the optimizer to fair solutions. Our framework GIFAIR-FL can accommodate both global and personalized settings. Theoretically, we show convergence in nonconvex and strongly convex settings. Our convergence guarantees hold for both independent and identically distributed (i.i.d.) and non-i.i.d. data. To demonstrate the empirical performance of our algorithm, we apply our method to image classification and text prediction tasks. Compared with existing algorithms, our method shows improved fairness results while retaining superior or similar prediction accuracy. History: Kwok-Leung Tsui served as the senior editor for this article. Funding: This work was supported by NSF CAREER [Grant 2144147]. Data Ethics &amp; Reproducibility Note: The code capsule is available on Code Ocean at https://codeocean.com/capsule/2590027/tree/v1 and in the e-Companion to this article (available at https://doi.org/10.1287/ijds.2022.0022 ).
Enhancing IoT Project Success through Agile Best Practices

Worldwide spending on Internet of Things (IoT) applications is forecasted to surpass $1 trillion by 2022. To stay competitive in this growing technological industry segment, lowering costs while increasing productivity and shortening time-to-market will become increasingly important. Adopting Agile Software Development practices for IoT projects may provide this competitive advantage, as it enables organizations to respond to change, while being dynamic and innovative. Applying a mixed-methods approach, agile IoT practitioners around the world and from diverse industries were surveyed and interviewed. Our study recommends that Agile Software Development team makeup, practices, and methods should be tailored to the specific industry, culture, people, and IT application of an organization. People play an important role in the success of agile projects; therefore, our research focuses on identifying the critical attributes of agile teams to maximize success. Our study identified the five critical agile practices: Collective Code Ownership, Continuous Integration, Single Team, Dedicated Customer, and Sprint Planning and found that both technical and soft skills are essential for successful IoT development.
Continuous Quantum Gate Sets and Pulse-Class Meta-Optimization

Reducing the circuit depth of quantum circuits is a crucial bottleneck to enabling quantum technology. This depth is inversely proportional to the number of available quantum gates that have been synthesised. Moreover, quantum gate synthesis and control problems exhibit a vast range of external parameter dependencies, both physical and application-specific. In this article we address the possibility of learning families of optimal control pulses which depend adaptively on various parameters, in order to obtain a global optimal mapping from the space of potential parameter values to the control space, and hence continuous classes of gates. Our proposed method is tested on different experimentally relevant quantum gates and proves capable of producing high-fidelity pulses even in presence of multiple variable or uncertain parameters with wide ranges.
Development and implementation of centralised, cloud-based, employee health contact tracing database and predictive modelling framework in the COVID-19 pandemic

At the onset of the COVID-19 pandemic in early March, 2020, it became apparent that non-electronic exposure investigations would not meet the demands required by the mounting number of employees with COVID-19 in the Mount Sinai Health System in New York City (NY, USA), a multicentre, academic medical institution and hospital system. New York City was an epicentre early in the pandemic, resulting in a peak of more than 6000 cases daily and more than 1000 deaths per day.1 Agile exposure investigation and contact tracing were crucial to containing the spread of COVID-19 among hospital staff.
User Experience during an Immersive Virtual Reality-Based Cognitive Task: A Comparison between Estonian and Italian Older Adults with MCI

Mild cognitive impairment (MCI) is an early stage of cognitive abilities loss and puts older adults at higher risk of developing dementia. Virtual reality (VR) could represent a tool for the early assessment of this pathological condition and for administering cognitive training. This work presents a study evaluating the acceptance and the user experience of an immersive VR application representing a supermarket. As the same application had already been assessed in Italy, we aimed to perform the same study in Estonia in order to compare the outcomes in the two populations. Fifteen older adults with MCI were enrolled in one Rehabilitation Center of Estonia and tried the supermarket once. Afterwards, they were administered questionnaires aimed at evaluating their technology acceptance, sense of presence, and cybersickness. Estonian participants reported low side effects and discrete enjoyment, and a sense of presence. Nonetheless, their intention to use the technology decreased after the experience. The comparison between Italian and Estonian older adults showed that cybersickness was comparable, but technology acceptance and sense of presence were significantly lower in the Estonian group. Thus, we argue that: (i) cultural and social backgrounds influence technology acceptance; (ii) technology acceptance was rather mediated by the absence of positive feelings rather than cybersickness.
Rapid determination of the roasting degree of cocoa beans by extreme learning machine (ELM)-based imaging analysis

The determination of the levels of roasting of cocoa relies on expensive analytical equipment, sensory panel, and, in the cases of small processors and growers, empiricism. In this study, cocoa beans were roasted for 10–40 min to obtain different levels of roasting, and the images of the beans were captured by a smartphone camera. An extreme learning machine (ELM)-based algorithm was developed to predict the roasting degree of cocoa beans using the images of the cocoa bean cross-sections. A 22-dimension feature vector, including color and texture features, is extracted from each sample, and a total of 350 samples are used to train an ELM network. A majority rule-based voting method was used to make the decision. Experimental results showed that the proposed method achieved a classification accuracy of 93.75%. GC-MS analysis was conducted to determine the chemical compounds in the raw and roasted cocoa beans, and enrichment analysis, principal components analysis, partial least-squares–discriminant analysis, and Pearson correlation analysis were conducted to identify major chemicals respond to roasting time and classify the cocoa beans samples. Caffeine and theobromine were identified as primary chemical compounds that responded to roasting time, and cocoa beans with different levels of roasting were successfully classified.
MagneShape: A Non-electrical Pin-Based Shape-Changing Display

Pin-based shape-changing displays can present dynamic shape changes by actuating a number of pins. However, the use of many linear actuators to achieve this makes the electrical structure and mechanical construction of the display complicated. We propose a simple pin-based shape-changing display that outputs shape and motions without any electronic elements. Our display consists of magnetic pins in a pin housing, with a magnetic sheet underneath it. The magnetic sheet has a specific magnetic pattern on its surface, and each magnetic pin has a magnet at its lower end. The repulsive force generated between the magnetic sheet and the magnetic pin levitates the pin vertically, and the height of the pin-top varies depending on the magnetic pattern. This paper introduces the basic structure of the display and compares several fabrication methods for the magnetic pins, to highlight the applicability of this method. We have also demonstrated some applications and discussed future possibilities.
Optimizing the Timing of Intelligent Suggestion in Virtual Reality

Intelligent suggestion techniques can enable low-friction selection-based input within virtual or augmented reality (VR/AR) systems. Such techniques leverage probability estimates from a target prediction model to provide users with an easy-to-use method to select the most probable target in an environment. For example, a system could highlight the predicted target and enable a user to select it with a simple click. However, as the probability estimates can be made at any time, it is unclear when an intelligent suggestion should be presented. Earlier suggestions could save a user time and effort but be less accurate. Later suggestions, on the other hand, could be more accurate but save less time and effort. This paper thus proposes a computational framework that can be used to determine the optimal timing of intelligent suggestions based on user-centric costs and benefits. A series of studies demonstrated the value of the framework for minimizing task completion time and maximizing suggestion usage and showed that it was both theoretically and empirically effective at determining the optimal timing for intelligent suggestions.
Concept-Annotated Examples for Library Comparison

Programmers often rely on online resources—such as code examples, documentation, blogs, and Q&A forums—to compare similar libraries and select the one most suitable for their own tasks and contexts. However, this comparison task is often done in an ad-hoc manner, which may result in suboptimal choices. Inspired by Analogical Learning and Variation Theory, we hypothesize that rendering many concept-annotated code examples from different libraries side-by-side can help programmers (1) develop a more comprehensive understanding of the libraries' similarities and distinctions and (2) make more robust, appropriate library selections. We designed a novel interactive interface, ParaLib, and used it as a technical probe to explore to what extent many side-by-side concepted-annotated examples can facilitate the library comparison and selection process. A within-subjects user study with 20 programmers shows that, when using ParaLib, participants made more consistent, suitable library selections and provided more comprehensive summaries of libraries' similarities and differences.
AirLogic: Embedding Pneumatic Computation and I/O in 3D Models to Fabricate Electronics-Free Interactive Objects

Researchers have developed various tools and techniques towards the vision of on-demand fabrication of custom, interactive devices. Recent work has 3D-printed artefacts like speakers, electromagnetic actuators, and hydraulic robots. However, these are non-trivial to instantiate as they require post-fabrication mechanical– or electronic assembly. We introduce AirLogic: a technique to create electronics-free, interactive objects by embedding pneumatic input, logic processing, and output widgets in 3D-printable models. AirLogic devices can perform basic computation on user inputs and create visible, audible, or haptic feedback; yet they do not require electronic circuits, physical assembly, or resetting between uses. Our library of 13 exemplar widgets can embed AirLogic-style computational capabilities in existing 3D models. We evaluate our widgets' performance—quantifying the loss of airflow (1) in each widget type, (2) based on printing orientation, and (3) from internal object geometry. Finally, we present five applications that illustrate AirLogic's potential.
HapTag: A Compact Actuator for Rendering Push-Button Tactility on Soft Surfaces

As touch interactions become ubiquitous in the field of human computer interactions, it is critical to enrich haptic feedback to improve efficiency, accuracy, and immersive experiences. This paper presents HapTag, a thin and flexible actuator to support the integration of push button tactile renderings to daily soft surfaces. Specifically, HapTag works under the principle of hydraulically amplified electroactive actuator (HASEL) while being optimized by embedding a pressure sensing layer, and being activated with a dedicated voltage appliance in response to users’ input actions, resulting in fast response time, controllable and expressive push-button tactile rendering capabilities. HapTag is in a compact formfactor and can be attached, integrated, or embedded on various soft surfaces like cloth, leather, and rubber. Three common push button tactile patterns were adopted and implemented with HapTag. We validated the feasibility and expressiveness of HapTag by demonstrating a series of innovative applications under different circumstances.
Look over there! Investigating Saliency Modulation for Visual Guidance with Augmented Reality Glasses

Augmented Reality has traditionally been used to display digital overlays in real environments. Many AR applications such as remote collaboration, picking tasks, or navigation require highlighting physical objects for selection or guidance. These highlights use graphical cues such as outlines and arrows. Whilst effective, they greatly contribute to visual clutter, possibly occlude scene elements, and can be problematic for long-term use. Substituting those overlays, we explore saliency modulation to accentuate objects in the real environment to guide the user's gaze. Instead of manipulating video streams, like done in perception and cognition research, we investigate saliency modulation of the real world using optical-see-through head-mounted displays. This is a new challenge, since we do not have full control over the view of the real environment. In this work we provide our specific solution to this challenge, including built prototypes and their evaluation.
InstantXR: Instant XR Environment on the Web Using Hybrid Rendering of Cloud-based NeRF with 3D Assets

For an XR environment to be used on a real-life task, it is crucial all the contents are created and delivered when we want, where we want, and most importantly, on time. To deliver an XR environment faster and correctly, the time spent on modeling should be considerably reduced or eliminated. In this paper, we propose a hybrid method that fuses the conventional method of rendering 3D assets with the Neural Radiance Fields (NeRF) technology, which uses photographs to create and display an instantly generated XR environment in real-time, without a modeling process. While NeRF can generate a relatively realistic space without human supervision, it has disadvantages owing to its high computational complexity. We propose a cloud-based distributed acceleration architecture to reduce computational latency. Furthermore, we implemented an XR streaming structure that can process the input from an XR device in real-time. Consequently, our proposed hybrid method for real-time XR generation using NeRF and 3D graphics is available for lightweight mobile XR clients, such as untethered HMDs. The proposed technology makes it possible to quickly virtualize one location and deliver it to another remote location, thus making virtual sightseeing and remote collaboration more accessible to the public. The implementation of our proposed architecture along with the demo video is available at https://moonsikpark.github.io/instantxr/.