Reviewerly/exHarmony huggingface dataset:
Graphhopper: Multi-hop Scene Graph Reasoning for Visual Question Answering

Abstract Visual Question Answering (VQA) is concerned with answering free-form questions about an image. Since it requires a deep semantic and linguistic understanding of the question and the ability to associate it with various objects that are present in the image, it is an ambitious task and requires multi-modal reasoning from both computer vision and natural language processing. We propose Graphhopper, a novel method that approaches the task by integrating knowledge graph reasoning, computer vision, and natural language processing techniques. Concretely, our method is based on performing context-driven, sequential reasoning based on the scene entities and their semantic and spatial relationships. As a first step, we derive a scene graph that describes the objects in the image, as well as their attributes and their mutual relationships. Subsequently, a reinforcement learning agent is trained to autonomously navigate in a multi-hop manner over the extracted scene graph to generate reasoning paths, which are the basis for deriving answers. We conduct an experimental study on the challenging dataset GQA, based on both manually curated and automatically generated scene graphs. Our results show that we keep up with human performance on manually curated scene graphs. Moreover, we find that Graphhopper outperforms another state-of-the-art scene graph reasoning model on both manually curated and automatically generated scene graphs by a significant margin.
Improving Inductive Link Prediction Using Hyper-relational Facts

For many years, link prediction on knowledge graphs (KGs) has been a purely transductive task, not allowing for reasoning on unseen entities. Recently, increasing efforts are put into exploring semi- and fully inductive scenarios, enabling inference over unseen and emerging entities. Still, all these approaches only consider triple-based KGs, whereas their richer counterparts, hyper-relational KGs (e.g., Wikidata), have not yet been properly studied. In this work, we classify different inductive settings and study the benefits of employing hyper-relational KGs on a wide range of semi- and fully inductive link prediction tasks powered by recent advancements in graph neural networks. Our experiments on a novel set of benchmarks show that qualifiers over typed edges can lead to performance improvements of 6% of absolute gains (for the Hits@10 metric) compared to triple-only baselines. Our code is available at https://github.com/mali-git/hyper_relational_ilp .
Region Coverage Path Planning of Multiple Disconnected Convex Polygons Based on Simulated Annealing Algorithm

Over the past decades, Unmanned Aerial Vehicles (UAVs) have been widely used in both industrial and military operations around the world. As part of mission planning, path planning will play a key role in improving the viability and mission capabilities of UAVs. However, the problem of multi-region coverage path planning is limited by many factors such as UAV's performance, endurance and target region. In this paper, an improved simulated annealing algorithm (ISA) has been proposed to solve the problem. The ISA method uses simulated annealing algorithm to determine the access order of regions, takes minimum-distance-first strategy to select the entry of region, and uses back-and-forth path (BFP) to cover the convex polygon region. The effectiveness of the proposed method is verified through simulation experiments with randomly generated targets.
ArmSym: A Virtual Human–Robot Interaction Laboratory for Assistive Robotics

Research in human–robot interaction for assistive robotics usually presents many technical challenges for experimenters, forcing researchers to split their time between solving technical problems and conducting experiments. In addition, previous work in virtual reality setups tends to focus on a single assistive robotics application. In order to alleviate these problems, we present ArmSym, a virtual reality laboratory with a fully simulated and developer-friendly robot arm. The system is intended as a testbed to run many sorts of experiments on human control of a robotic arm in a realistic environment, ranging from an upper limb prosthesis to a wheelchair-mounted robotic manipulator. To highlight the possibilities of this system, we perform a study comparing different sorts of prosthetic control types. Looking at nonimpaired subjects, we study different psychological metrics that evaluate the interaction of the user with the robot under different control conditions. Subjects report a perception of embodiment in the absence of realistic cutaneous touch, supporting previous studies in the topic. We also find interesting correlations between control and perceived ease of use. Overall our results confirm that ArmSym can be used to gather data from immersive experiences prosthetics, opening the door to closer collaboration between device engineers and experience designers in the future.
Advanced Hough-based method for on-device document localization

The demand for on-device document recognition systems increases in conjunction with the emergence of more strict privacy and security requirements. In such systems, there is no data transfer from the end device to a third-party information processing servers. The response time is vital to the user experience of on-device document recognition. Combined with the unavailability of discrete GPUs, powerful CPUs, or a large RAM capacity on consumer-grade end devices such as smartphones, the time limitations put significant constraints on the computational complexity of the applied algorithms for on-device execution. In this work, we consider document location in an image without prior knowledge of the document content or its internal structure. In accordance with the published works, at least 5 systems offer solutions for on-device document location. All these systems use a location method which can be considered Hough-based. The precision of such systems seems to be lower than that of the state-of-the-art solutions which were not designed to account for the limited computational resources. We propose an advanced Hough-based method. In contrast with other approaches, it accounts for the geometric invariants of the central projection model and combines both edge and color features for document boundary detection. The proposed method allowed for the second best result for SmartDoc dataset in terms of precision, surpassed by U-net like neural network. When evaluated on a more challenging MIDV-500 dataset, the proposed algorithm guaranteed the best precision compared to published methods. Our method retained the applicability to on-device computations.
3D Semantic Mapping from Arthroscopy Using Out-of-Distribution Pose and Depth and In-Distribution Segmentation Training

Minimally invasive surgery (MIS) has many documented advantages, but the surgeon’s limited visual contact with the scene can be problematic. Hence, systems that can help surgeons navigate, such as a method that can produce a 3D semantic map, can compensate for the limitation above. In theory, we can borrow 3D semantic mapping techniques developed for robotics, but this requires finding solutions to the following challenges in MIS: 1) semantic segmentation, 2) depth estimation, and 3) pose estimation. In this paper, we propose the first 3D semantic mapping system from knee arthroscopy that solves the three challenges above. Using out-of-distribution non-human datasets, where pose could be labeled, we jointly train depth+pose estimators using self-supervised and supervised losses. Using an in-distribution human knee dataset, we train a fully-supervised semantic segmentation system to label arthroscopic image pixels into femur, ACL, and meniscus. Taking testing images from human knees, we combine the results from these two systems to automatically create 3D semantic maps of the human knee. The result of this work opens the pathway to the generation of intra-operative 3D semantic mapping, registration with pre-operative data, and robotic-assisted arthroscopy. Source code: https://github.com/YJonmo/EndoMapNet.
Exploring a Federated Learning Approach to Enhance Authorship Attribution of Misleading Information from Heterogeneous Sources

Authorship Attribution (AA) is currently applied in several applications, among which fraud detection and anti-plagiarism checks: this task can leverage stylometry and Natural Language Processing techniques. In this work, we explored some strategies to enhance the performance of an AA task for the automatic detection of false and misleading information (e.g., fake news). We set up a text classification model for AA based on stylometry exploiting recurrent deep neural networks and implemented two learning tasks trained on the same collection of fake and real news, comparing their performances: one is based on Federated Learning architecture, the other on a centralized architecture. The goal was to discriminate potential fake information from true ones when the fake news comes from heterogeneous sources, with different styles. Preliminary experiments show that a distributed approach significantly improves recall with respect to the centralized model. As expected, precision was lower in the distributed model. This aspect, coupled with the statistical heterogeneity of data, represents some open issues that will be further investigated in future work.
Annealing Processing Architecture of 28-nm CMOS Chip for Ising Model With 512 Fully Connected Spins

With the development of the Internet of things (IoT), sensors are being mounted on various objects. This trend has prompted demand for low-power, high-performance information processing on the edge side. Here, an Ising model architecture that can efficiently solve optimization problems would be an efficient processing solution for edges. In this study, we implemented a 512- spin fully connected Ising model on an LSI chip fabricated in a 28-nm CMOS process. The fully connected Ising model was implemented in the chip by using pseudo-annealing (PA), which is easier to implement than simulated annealing (SA). In addition, we devised a multi-spin-thread structure, concurrent update structure, and a folded interaction placement for accuracy, speed, and compactness. Because eight spin threads are implemented, the calculation throughput could be increased by a factor of eight in comparison with a single spin-thread implementation. Moreover, as a measure of solution accuracy, the average route length of a 22-city traveling salesman problem was reduced by 19% and the standard deviation (SD) was reduced by 46%. Likewise, the average cut value of a 512- node max-cut problem was increased by 1.6% and SD was decreased by 60%. The concurrent update almost doubled the calculation speed in comparison with the case of no concurrent update. In addition, the circuit area was reduced by about 38% as a result of the folded interaction placement. The time required to obtain a solution was 128 ms. The chip at annealing processing (main processing) had a power consumption of 12 mW at 1 MHz.
Continuous Copy-Paste for One-stage Multi-object Tracking and Segmentation

Current one-step multi-object tracking and segmentation (MOTS) methods lag behind recent two-step methods. By separating the instance segmentation stage from the tracking stage, two-step methods can exploit non-video datasets as extra data for training instance segmentation. Moreover, instances belonging to different IDs on different frames, rather than limited numbers of instances in raw consecutive frames, can be gathered to allow more effective hard example mining in the training of trackers. In this paper, we bridge this gap by presenting a novel data augmentation strategy named continuous copy-paste (CCP). Our intuition behind CCP is to fully exploit the pixel-wise annotations provided by MOTS to actively increase the number of instances as well as unique instance IDs in training. Without any modifications to frameworks, current MOTS methods achieve significant performance gains when trained with CCP. Based on CCP, we propose the first effective one-stage online MOTS method named CCPNet, which generates instance masks as well as the tracking results in one shot. Our CCPNet surpasses all state-of-the-art methods by large margins (3.8% higher sMOTSA and 4.1% higher MOTSA for pedestrians on the KITTI MOTS Validation) and ranks 1st on the KITTI MOTS leaderboard. Evaluations across three datasets also demonstrate the effectiveness of both CCP and CCPNet. Our codes are publicly available at: https://github.com/detectRecog/CCP.
A CNN based Handwritten Numeral Recognition Model for Four Arithmetic Operations

The pandemic of Covid-19 has caused a shift of paradigm of education, from face-to-face to e-learning. E-learning leads to an escalation in digitalization of handwritten documents because it requires submission of homework and assignments through online. To help teachers in checking digitalized handwritten homework, this paper proposes an automatic checking system based on a convolutional neural network (CNN) for handwritten numeral recognition. The CNN is used to recognize four arithmetic operations in mathematical questions consisting of addition, deduction, multiplication and division. The performance CNN in handwritten numeral recognition have been optimized in terms of activation function and gradient descent algorithm. The proposed CNN is also trained and tested with the MNIST handwritten data set. The experimental results show that the recognition accuracy the improved CNN improves to a certain extent as compared to before optimization.
Correlated Randomness Teleportation via Semi-trusted Hardware—Enabling Silent Multi-party Computation

With the advancement of the trusted execution environment (TEE) technologies, hardware-supported secure computing becomes increasingly popular due to its efficiency. During the protocol execution, typically, the players need to contact a third-party server for remote attestation, ensuring the validity of the involved trusted hardware component, such as Intel SGX, as well as the integrity of the computation result. When the hardware manufacturer is not fully trusted, sensitive information may be leaked to the third-party server through backdoors, steganography, and kleptography, etc. In this work, we introduce a new security notion called semi-trusted hardware model, where the adversary is allowed to passively or maliciously corrupt the hardware. Therefore, she can learn the input of the hardware component and might also tamper its output. We then show how to utilize such semi-trusted hardwares for correlated randomness teleportation. When the semi-trusted hardware is instantiated by Intel SGX, to generate 10k random OT’s, our protocol is 24X and 450X faster than the EMP-IKNP-ROT in the LAN and WAN setting, respectively. When SGX is used to teleport Garbled circuits, the resulting two-party computation protocol is 5.3-5.7X and 43-47X faster than the EMP-SH2PC in the LAN and WAN setting, respectively, for the AES-128, SHA-256, and SHA-512 evaluation. We also show how to achieve malicious security with little overhead.
New Attacks on LowMC Instances with a Single Plaintext/Ciphertext Pair

Cryptanalysis of the LowMC block cipher when the attacker has access to a single known plaintext/ciphertext pair is a mathematically challenging problem. This is because the attacker is unable to employ most of the standard techniques in symmetric cryptography like linear and differential cryptanalysis. This scenario is particularly relevant while arguing the security of the PICNIC digital signature scheme in which the plaintext/ciphertext pair generated by the LowMC block cipher serves as the public (verification) key and the corresponding LowMC encryption key also serves as the secret (signing) key of the signature scheme. In the paper by Banik et al. (IACR ToSC 2020:4), the authors used a linearization technique of the LowMC S-box to mount attacks on some instances of the block cipher. In this paper, we first make a more precise complexity analysis of the linearization attack. Then, we show how to perform a 2-stage MITM attack on LowMC. The first stage reduces the key candidates corresponding to a fraction of key bits of the master key. The second MITM stage between this reduced candidate set and the remaining fraction of key bits successfully recovers the master key. We show that the combined computational complexity of both these stages is significantly lower than those reported in the ToSC paper by Banik et al.
An Overview of Sensors, Design and Healthcare Challenges in Smart Homes: Future Design Questions

The ageing population increases the demand for customized home care. As a result, sensing technologies are finding their way into the home environment. However, challenges associated with how users interact with sensors and data are not well-researched, particularly from a design perspective. This review explores the literature on important research projects around sensors, design and smart healthcare in smart homes, and highlights challenges for design research. A PRISMA protocol-based screening procedure is adopted to identify relevant articles (n = 180) on the subject of sensors, design and smart healthcare. The exploration and analysis of papers are performed using hierarchical charts, force-directed layouts and 'bedraggled daisy' Venn diagrams. The results show that much work has been carried out in developing sensors for smart home care. Less attention is focused on addressing challenges posed by sensors in homes, such as data accessibility, privacy, comfort, security and accuracy, and how design research might solve these challenges. This review raises key design research questions, particularly in working with sensors in smart home environments.
Chasing the high-resolution mapping of rotational and strain FRFs as receptance processing from different full-field optical measuring technologies

Thanks to the recent advances in digital vision systems, a question might arise about where the full-field optical and contactless methods can bring modern design procedures. An answer lies on the methodologically detailed comparison of the results, processed from different full-field optical techniques, in exploring consistent and high-resolution maps of rotational and strain FRFs that is done through this paper. Great exertions were first put in thoroughly testing a thin aluminium plate, in its real dynamics as a lightweight structure with broad frequency band dynamics and high modal density, in a unique comparative set-up, to obtain Receptance FRF maps of displacements-over-force by means of 3 different full-field optical techniques (SLDV, DIC, ESPI). There resulted superior quality Receptance maps in a broad and dense frequency domain, with high-resolution and continuity-wise consistent mapping at each frequency line. This paper exploits the here detailed robust numerical differentiation and signal processing in order to calculate the Impedance-based models of rotational and strain FRFs. Especially for DIC and for ESPI, both rarely used on many frequency lines, the fully populated FRFs for rotations and strains are a clear novelty, with rotational Coherence functions as added quality assessment features. The systematic comparison of the results obtained in the same location of the sample, by means of spatial and frequency domain metrics, is possible because of the proposed pointwise procedure, permitting the first numerical matching assessment of the 3 optical technologies on these awaited experiment-based quantities, as the full-field rotational and strain FRFs.
DepthInSpace: Exploitation and Fusion of Multiple Video Frames for Structured-Light Depth Estimation

We present DepthInSpace, a self-supervised deep-learning method for depth estimation using a structured-light camera. The design of this method is motivated by the commercial use case of embedded depth sensors in nowadays smartphones. We first propose to use estimated optical flow from ambient information of multiple video frames as a complementary guide for training a single-frame depth estimation network, helping to preserve edges and reduce over-smoothing issues. Utilizing optical flow, we also propose to fuse the data of multiple video frames to get a more accurate depth map. In particular, fused depth maps are more robust in occluded areas and incur less in flying pixels artifacts. We finally demonstrate that these more precise fused depth maps can be used as self-supervision for fine-tuning a single-frame depth estimation network to improve its performance. Our models’ effectiveness is evaluated and compared with state-of-the-art models on both synthetic and our newly introduced real datasets. The implementation code, training procedure, and both synthetic and captured real datasets are available at https://www.idiap.ch/paper/depthinspace.
Statistical model building: Background “knowledge” based on inappropriate preselection causes misspecification

Statistical model building requires selection of variables for a model depending on the model's aim. In descriptive and explanatory models, a common recommendation often met in the literature is to include all variables in the model which are assumed or known to be associated with the outcome independent of their identification with data driven selection procedures. An open question is, how reliable this assumed "background knowledge" truly is. In fact, "known" predictors might be findings from preceding studies which may also have employed inappropriate model building strategies.We conducted a simulation study assessing the influence of treating variables as "known predictors" in model building when in fact this knowledge resulting from preceding studies might be insufficient. Within randomly generated preceding study data sets, model building with variable selection was conducted. A variable was subsequently considered as a "known" predictor if a predefined number of preceding studies identified it as relevant.Even if several preceding studies identified a variable as a "true" predictor, this classification is often false positive. Moreover, variables not identified might still be truly predictive. This especially holds true if the preceding studies employed inappropriate selection methods such as univariable selection.The source of "background knowledge" should be evaluated with care. Knowledge generated on preceding studies can cause misspecification.
Personality Prediction Through Handwriting Analysis Using Convolutional Neural Networks

The prediction of personality traits through handwriting analysis is done by the Graphologist. The famous methodology of doing this analysis is commonly known as graphology. The prediction of personality traits using handwriting depends upon how much the graphologist is efficient in doing so. The development of computer-based handwriting analysis algorithm can make a great impact in predicting personality traits. In this paper, convolutional neural network (CNN) is used to predict the five big personality traits (Agreeableness, Conscientiousness, Extraversion, Neuroticism, Openness) of the writer at the time of writing.