Reviewerly/exHarmony huggingface dataset:
Online recommendation based on incremental-input self-organizing map

• We change the basic structure of the input layer with a fixed number of units, which is usually adopted by most neural networks, and develop an incremental-input SOM whose input layer can be “expanded” by adding new input units. • We propose an online recommendation algorithm based on the incremental-input SOM and provide two different updating mechanisms, which can handle the arrivals of new consumers and new products simultaneously. • The algorithm is superior to several existing online recommendation algorithms, and performs similarly to the static recommendation algorithm which need to be retrained by all of the data. E-commerce platforms usually use recommendation algorithms to help consumers find products that they may prefer. Most traditional recommendation models, which are generally built on the basis of static data, may fail in managing newly arrived ratings of new consumers or on new products. These static recommendation models need to be retrained offline using the newly arrived ratings, which is often inefficient and time consuming. In this paper, we propose a novel incremental-input self-organizing map (SOM) and apply it to generate an online recommendation model. We add new consumers into the recommendation model by augmenting the input units and updating the corresponding weights between the input layer and the output layer. We treat new products as new samples arriving at the SOM and specially design an incremental learning strategy for SOM. Thus, the proposed model does not have to be retrained and can be updated online with the very few ratings of new consumers or new products. Experimental results on real-world datasets indicate that the proposed algorithm is superior to several existing online recommendation algorithms in terms of recommendation accuracy and performs similarly to the static recommendation algorithm which needs to be retrained by all of the data.
Hashing for Localization (HfL): A Baseline for Fast Localizing Objects in a Large-Scale Scene

Advanced remote-sensing instruments produce massively large scenes from the surface of the earth, with very high spatial resolution and dimensionality. Developing methods for efficiently localizing specific objects in a large-scale scene presents a significant challenge, mainly because of the high computational requirements involved. To tackle this issue, we propose a new hashing for localization (HfL) framework that efficiently searches for specific objects in the large-scale scene. It begins by dividing the scene into a large number of overlapping local patches. A lightweight deep hash model, referred to as a tiny hashing network (THNet), encodes the local patches into hash codes. The Hamming distances between the hash code of an object image, i.e., an image containing the specific class of objects to be localized in the scene, and those of all local patches are computed. Small values of the Hamming distance indicate local patches that are similar to the object image. The positions of these local patches in the large-scale scene reflect the regional locations of the specific objects. The hash codes are binary and do not take up much space, and the Hamming distance carries very low-computational overheads. Further, we exploit a class center loss as the THNet training objective, which can comprehensively manage multiple object classes. These features mean that the HfL framework can localize specific objects very quickly, regardless of the size of the scene. Extensive experiments validate the effectiveness and efficiency of the framework. For instance, HfL can find objects in a remote-sensing image of 19584 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$ </tex-math></inline-formula> 19584 pixels in only 4.388 s (on a single RTX2080ti), with remarkable localization results. The source codes and datasets are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/lrhan/HfL</uri> , together providing a baseline for fast localizing objects in a large-scale scene.
Medical Matting: A New Perspective on Medical Segmentation with Uncertainty

In medical image segmentation, it is difficult to mark ambiguous areas accurately with binary masks, especially when dealing with small lesions. Therefore, it is a challenge for radiologists to reach a consensus by using binary masks under the condition of multiple annotations. However, these uncertain areas may contain anatomical structures that are conducive to diagnosis. Uncertainty is introduced to study these situations. Nevertheless, the uncertainty is usually measured by the variances between predictions in a multiple trial way. It is not intuitive, and there is no exact correspondence in the image. Inspired by image matting, we introduce matting as a soft segmentation method and a new perspective to deal with and represent uncertain regions into medical scenes, namely medical matting. More specifically, because there is no available medical matting dataset, we first labeled two medical datasets with alpha matte. Secondly, the matting methods applied to the natural image are not suitable for the medical scene, so we propose a new architecture to generate binary masks and alpha matte in a row. Thirdly, the uncertainty map is introduced to highlight the ambiguous regions from the binary results and improve the matting performance. Evaluated on these datasets, the proposed model outperformed state-of-the-art matting algorithms by a large margin, and alpha matte is proved to be a more efficient labeling form than a binary mask.
Pancreatic cancer segmentation in unregistered multi-parametric MRI with adversarial learning and multi-scale supervision

Automated pancreatic cancer segmentation is crucial for successful clinical aid diagnosis and surgical planning. However, the tiny size and inconspicuous boundaries of pancreatic cancer lesions lead to poor segmentation performance with single-modality imaging. The commonly used registration-based multi-modal fusion strategies may not only introduce uncertainties arising from registration, but also underutilize the complementary information between different modalities. Thus, to achieve different modality-based tumor segmentation, we propose for the first time a registration-free multi-modal and multi-scale adversarial segmentation network (MMSA-Net), which consist of a shared encoder and a dual decoder. Specifically, MMSA-Net combine two complementary modules, inter-modality adversarial learning and intra-modality multi-scale adversarial supervision, to obtain mode-specific segmentation results while facilitating multi-modal fusion. The inter-modality adversarial learning module facilitates the fusion of modality-shared features among different modalities by strengthening the similarity of features extracted by the shared encoder. The intra-modality multi-scale adversarial supervision emphasizes modality-specific features in different decoding paths, inherits and fuses modality-shared features while preserving the feature specificity of each modality, thus outputting competitive modality-specific segmentation results for each modality. Quantitative and qualitative experimental results on multi-parametric MRI pancreatic cancer data show that our method can effectively improve the performance of multi-modal segmentation. The method proposed in this work is expected to be another potential paradigm for addressing multi-modal segmentation tasks in addition to registration. Our source codes will be released at https://github.com/SJTUBME-QianLab/PancreaticCancer_Seg_AdvLearning, once the manuscript is accepted for publication.
Document Collection Visual Question Answering

Current tasks and methods in Document Understanding aims to process documents as single elements. However, documents are usually organized in collections (historical records, purchase invoices), that provide context useful for their interpretation. To address this problem, we introduce Document Collection Visual Question Answering (DocCVQA) a new dataset and related task, where questions are posed over a whole collection of document images and the goal is not only to provide the answer to the given question, but also to retrieve the set of documents that contain the information needed to infer the answer. Along with the dataset we propose a new evaluation metric and baselines which provide further insights to the new dataset and task.
Evaluating FFT-based algorithms for strided convolutions on ARMv8 architectures

Convolutional Neural Networks (CNNs) have been widely adopted in many kinds of artificial intelligence applications. Most of the computational overhead of CNNs is spent on convolutions. An effective approach to reducing the overhead is transforming convolutions in the time domain into multiplications in the frequency domain by means of Fast Fourier Transform (FFT) algorithms, known as FFT-based fast algorithms for convolutions. However, current FFT-based fast implementations only work for unit-strided convolutions with stride as 1, and cannot be directly applied to strided convolutions with stride size greater than 1, which are usually used as the first layer of CNNs and as an effective alternative to the pooling layers for downsampling. In this paper, we first introduce rearrangement- and sampling-based methods for applying FFT-based fast algorithms to strided convolutions, and the arithmetic complexities of these two methods and the direct method are compared in detail. Then, the highly optimized parallel implementations of the two methods on ARMv8-based many-core CPU are presented. Lastly, we benchmark these implementations against two GEMM-based implementations on this ARMv8 CPU. Our experimental results with convolutions of different kernels, feature maps, and batch sizes show that the rearrangement-based method generally exceeds the sampling-based one under the same optimizations in most cases, and these two methods can get much better performance than GEMM-based ones when the kernels, feature maps and batch sizes are large. The experimental results on the convolutional layers in popular CNNs further demonstrate the conclusions above.
How Prior Knowledge Affects Visual Attention of Japanese Mimicry and Onomatopoeia and Learning Outcomes: Evidence from Virtual Reality Eye Tracking

According to the United Nations Sustainable Development Goal (SDG) 4, “achieving inclusive and quality education for all”, foreign language learning has come to be seen as a process of integrating sustainable development into the socio-cultural aspects of education and learning. The aim of this study was to employ virtual reality (VR) eye tracker to examine how students with different levels of prior knowledge process visual behaviors for Japanese Mimicry and Onomatopoeia (MIO) while learning Japanese as a second foreign language. A total of 20 students studying at the Department of Applied Japanese at the university of Southern Taiwan were recruited. Based on the Japanese language proficiency test (JLPT) level, 20 participants were divided into high prior knowledge group (levels N1–N3) with 7 participants, and low prior knowledge group (level N4 or below) with 13 participants. The learning stimuli materials were created by Unreal Engine 4 (UE4) development tool to design a 3D virtual MIO paradise, including 5 theme amusement parks. Through a VR eye tracker, participants’ visual behaviors were tracked and recorded based on 24 different regions of interest (ROIs) (i.e., ROI1–ROI24). This was done to discuss the distribution of visual attention in terms of different ROIs of each theme amusement park based on four eye movement indicators, including latency of first fixation (LFF), duration of first fixation (DFF), total fixation durations (TFD), and fixation counts (FC). Each ROI of the two groups were then compared. In addition, a heat zone map was also generated to show the overall visual distribution of each group. After the experiment, based on the eye movement indicators and test scores in the pre-test and post-test phases, statistical analysis was used to examine and evaluate the differences in visual attention and learning outcomes. The results revealed that the gaze sequences of the two prior knowledge groups gazing at the ROIs in theme parks were different, except for the gaze sequence in the circus theme park. Different prior knowledge groups exhibited differences in visual attention in the ROIs fixated on in each amusement park. Additionally, in terms of TFD and FC of different groups in each amusement park, there was no significant difference except in ROI10, ROI16, and ROI18. Moreover, after receiving cognitive comprehension processes introduced in the VR-simulated MIO scenes, students from both groups achieved higher post-test scores compared with pre-test scores, and such differences had statistical significance. In conclusion, the implications of VR eye movement analysis on developing students’ competence related to learning Japanese and cross-cultural aspects, compatible with sustainable development, were presented.
Deep Structural Point Process for Learning Temporal Interaction Networks

This work investigates the problem of learning temporal interaction networks. A temporal interaction network consists of a series of chronological interactions between users and items. Previous methods tackle this problem by using different variants of recurrent neural networks to model interaction sequences, which fail to consider the structural information of temporal interaction networks and inevitably lead to sub-optimal results. To this end, we propose a novel Deep Structural Point Process termed as DSPP for learning temporal interaction networks. DSPP simultaneously incorporates the topological structure and long-range dependency structure into the intensity function to enhance model expressiveness. To be specific, by using the topological structure as a strong prior, we first design a topological fusion encoder to obtain node embeddings. An attentive shift encoder is then developed to learn the long-range dependency structure between users and items in continuous time. The proposed two modules enable our model to capture the user-item correlation and dynamic influence in temporal interaction networks. DSPP is evaluated on three real-world datasets for both tasks of item prediction and time prediction. Extensive experiments demonstrate that our model achieves consistent and significant improvements over state-of-the-art baselines.
Assessing the Risk of Software Development in Agile Methodologies Using Simulation

Agile methodologies aim to reduce software development risk using short iterations, feature-driven development, continuous integration, testing automation, and other practices. However, the risk of project failure or time and budget overruns is still a relevant problem. The aim of this paper is to present and discuss a new approach to model some key risk factors in agile development, using software process simulation modeling (SPSM), which can complement other approaches, and whose usage is particularly suited for agile development. We introduce a new approach to modeling some key risk factors – namely project duration, number of implemented issues, and key statistics of issue completion time – using a simulator of agile development, which we developed to this purpose. The approach includes modeling the agile process, gathering data from the tool used for project management, and performing Monte Carlo simulations of the process, to get insights about the expected time and effort to complete the project, and to their distributions. The model’s parameters that can cause risk are errors in effort estimation of the features to develop, variations in developers’ assignment to these features, impediments related to developers’ availability and work completion. To validate the simulator, and to demonstrate how the method can be used, we analyzed three open source projects, gathering their data from JIRA repositories. We ran Monte Carlo simulations of these projects, showing that the simulator can well approximate the progress of the real project, then varying the identified risk factors and statistically evaluating their effects on the risk parameters. The proposed approach is relevant for project managers, being able to quantitatively evaluate the risks, provided that the process and the project’s data are properly modeled and gathered. As for the future, we are working to improve our risk assessment method, evaluating it on more case studies, scaling the model from a single team to multiple teams involved in one or more projects.
Fused Twins: A Cognitive Approach to Augmented Reality Media Architecture

Recent advances in Augmented Reality (AR), the Internet of Things (IoT), cloud computing, and Digital Twins transform the types, rates, and volume of information generated in buildings as well as the mediums through which they can be perceived by users. These advances push the standard approach of media architecture to embed screens in the built environment to its limits because screens lack the immersive capacity that newer media afford. To bridge this gap, we propose a novel AR approach to media architecture that uses a Digital Twin as a platform for structuring and accessing data from various sources, including IoT and simulations. Our technical contribution to media architecture is threefold. First, we extend the possibilities of media architecture beyond embedded screens to three dimensions by presenting a Digital Twin using AR with a head-mounted display. This approach results in a shared and consistent augmented experience across large architectural spaces. Second, we use the Digital Twin to integrate and visualize real physical sensor information. Third, we make artificial occupancy simulations accessible to everyday users by presenting them within their natural context in the Digital Twin. Observing the Digital Twin in situ of the Physical Twin also has applications beyond media architecture. Fusing the two twins using AR can reduce the cognitive load of users from consuming big and complex information sources and enhance their experience. We present two use cases of the proposed Fused Twins in a university building at ETH Zürich. In the first use case, we visualize a dense indoor sensor network (DISN) with 390 IoT sensors that collected data from March 2020 to May 2021. In the second use case, we immerse visitors in agent-based simulations to enable insights into the real and projected uses of space. This work brings forward an ambitious vision for media architecture beyond traditional flat screens, and showcases its potential through fusing state of the art simulations, sensor data integration and augmented reality, finally making the jump from fiction to reality.
Smooth Variational Graph Embeddings for Efficient Neural Architecture Search

Neural architecture search (NAS) has recently been addressed from various directions, including discrete, sampling-based methods and efficient differentiable approaches. While the former are notoriously expensive, the latter suffer from imposing strong constraints on the search space. Architecture optimization from a learned embedding space for example through graph neural network based variational autoencoders builds a middle ground and leverages advantages from both sides. Such approaches have recently shown good performance on several benchmarks. Yet, their stability and predictive power heavily depends on their capacity to reconstruct networks from the embedding space. In this paper, we propose a two-sided variational graph autoencoder, which allows to smoothly encode and accurately reconstruct neural architectures from various search spaces. We evaluate the proposed approach on neural architectures defined by the ENAS approach, the NAS-Bench-101 and the NAS-Bench-201 search space and show that our smooth embedding space allows to directly extrapolate the performance prediction to architectures outside the seen domain (e.g. with more operations). Thus, it facilitates to predict good network architectures even without expensive Bayesian optimization or reinforcement learning.
Key generation analysis for satellite quantum key distribution

Developing global quantum communication networks is integral to the realisation of the quantum internet, which is expected to impart a similar revolutionary impact on the technological landscape as the classical internet. Satellite-based quantum communications provides a practical route to global quantum networking. In this work, we model finite statistics to determine the finite secret key length generation in SatQKD systems that implement trusted-node downlink operation with weak coherent pulse sources. We optimise the finite key rate for different practical operations and determine the key generation footprints. Our work provides an essential guide for future satellite missions to establish performance benchmarks for both sources and detectors.
To automatically map source code entities to architectural modules with Naive Bayes

The process of mapping a source code entity onto an architectural module is to a large degree a manual task. Automating this process could increase the use of static architecture conformance checking methods, such as reflexion modeling, in industry. Current techniques rely on user parameterization and a highly cohesive design. A machine learning approach would potentially require less parameters and better use of the available information to aid in automatic mapping. We investigate how a classifier can be trained to map from source code to architecture modules automatically. This classifier is trained with semantic and syntactic dependency information extracted from the source code and from architecture descriptions. The classifier is implemented using multinomial naive Bayes and evaluated. We perform experiments and compare the classifier with three state-of-the-art mapping functions in eight open-source Java systems with known ground-truth-mappings. We find that the classifier outperforms the state-of-the-art in all cases and that it provides a useful baseline for further research in the area of semi-automatic incremental clustering. We conclude that machine learning is a useful approach that performs better and with less need for parameterization compared to other approaches. Future work includes investigating problematic mappings and a more diverse set of subject systems.