Reviewerly/exHarmony huggingface dataset:
DinosaurVR: Using Virtual Reality to Enhance a Museum Exhibition

Museums featuring dinosaur fossils have always attracted the attention of the crowd. However, sustaining public interest in science becomes more challenging year after year, even for popular attractions, calling for changes in how exhibits appear in the face of new technologies. To enrich the user’s experience and enhance the exhibition’s attractiveness, we developed and evaluated an immersive and interactive Virtual Reality (VR) experience to integrate the Paleontology exhibition at the Câmara Cascudo Museum in Natal, Brazil. Experienced VR users were interviewed and reported, through software tests, that the system was stable and performed as intended without any noticeable issues that could affect the user experience. They also found the content adequate, except for the suggestion to include more information about Paleoichnology and the finding of fossil tracksites. The graphical quality received mixed reviews, with some participants suggesting improvements to the terrain, such as adding vegetation and enhancing lighting. They also noted that the experience would appeal to younger audiences, thanks to the novelty of VR technology and the accessibility offered by the museum. Improvements were made to the environment based on their feedback, including changes to the terrain and lighting. Additionally, we developed two alternative versions of the experience, one for multi-projection in a room and another without interactive elements. Our results indicate that the VR experience can be successfully integrated with the exhibition and has the potential to enhance museum visits. It can connect the audience with the actual fossils on display by using a dinosaur in the exhibition. It allows one to visualize how their region changed in the past million years. Finally, this experiment helped bridge the population from an unprivileged region with the science that is produced upon the fossils found in their context. Such knowledge broadens the public’s imagination and triggers a whole chain of development for the community around the museum. Our VR exhibition prototype showed great potential to amplify this mission.
Systemization of Knowledge: Robust Deep Learning using Hardware-software co-design in Centralized and Federated Settings

Deep learning (DL) models are enabling a significant paradigm shift in a diverse range of fields, including natural language processing and computer vision, as well as the design and automation of complex integrated circuits. While the deep models – and optimizations based on them, e.g., Deep Reinforcement Learning (RL) – demonstrate a superior performance and a great capability for automated representation learning, earlier works have revealed the vulnerability of DL to various attacks. The vulnerabilities include adversarial samples, model poisoning, and fault injection attacks. On the one hand, these security threats could divert the behavior of the DL model and lead to incorrect decisions in critical tasks. On the other hand, the susceptibility of DL to potential attacks might thwart trustworthy technology transfer as well as reliable DL deployment. In this work, we investigate the existing defense techniques to protect DL against the above-mentioned security threats. Particularly, we review end-to-end defense schemes for robust deep learning in both centralized and federated learning settings. Our comprehensive taxonomy and horizontal comparisons reveal an important fact that defense strategies developed using DL/software/hardware co-design outperform the DL/software-only counterparts and show how they can achieve very efficient and latency-optimized defenses for real-world applications. We believe our systemization of knowledge sheds light on the promising performance of hardware-software co-design of DL security methodologies and can guide the development of future defenses.
Deep Neural Network model for Underwater Image Enhancement

In recent years, there has been a growing interest in the field of underwater image enhancement, driven by its significance in underwater robotics and ocean engineering. Initially, research efforts focused on physics-based approaches, but with advancements in technology, the utilization of deep convolutional neural networks (CNNs) and generative adversarial networks (GANs) has become prevalent. These state-of-the-art algorithms have shown impressive results; however, their computational complexity and memory requirements pose challenges to their practical implementation on portable devices used for underwater exploration tasks. Furthermore, these models are often trained on either synthetic or limited real-world datasets, limiting their applicability in real-world scenarios. In this paper, we propose a novel deep neural network architecture that maintains high performance while reducing the number of parameters compared to existing state-of-the-art models. Our approach aims to address the computational and memory limitations associated with underwater image enhancement algorithms. By leveraging the strengths of our architecture, we demonstrate its generalization capability by evaluating its performance on a combination of synthetic and real-world datasets. This approach enhances the practicality and applicability of our model in real-world underwater scenarios.
Communication about sensors and communication through sensors: localizing the Internet of Things in rural communities

Abstract Internet of Things (IoT) sensor networks are an emerging technology at the center of the datafication and optimization of far-reaching environmental infrastructures—from “smart cities” to workplace efficiencies. However, this low-power, low-cost technology is also well suited to local deployments in rural communities, which are often overlooked by digital development initiatives. Therefore, we used a social construction of technology approach to study how various U.S.-based IoT stakeholders—including designers and advocates as well as citizen stakeholders—understand and value sensor network technologies. Through observational methods, in-depth interviews, and participatory design research in a rural Upstate New York municipality, we worked to design sensor networks with rural community members to generate data about and for community members to further local knowledge. We found that designing rural sensor networks requires stakeholders to navigate obstacles of communication about sensors and communication through sensors to facilitate secure, ethical, and localized sensing in rural communities.
Digital governance with smart sensors: exploring grid administration in Zhejiang’s “Future Community”

Abstract This study explored sensor-based e-government practices in eight pilot “Future Communities” in Zhejiang Province, China. Adopting the approach of “e-government in everyday practice,” it examines how grid members at the grassroots level make sense of smart sensors and their mediated governance through the parameters of passive response, collaborative human and nonhuman networks, and calculability. In an emerging sensing environment, grid members are exposed to contradictory governance regimes, experimenting with and performing competing sets of discourses and actions that are either efficient or inefficient, manual or automatic, and progressive or regressive. Neither facilitated nor emptied by sensor-enabled Weberian functions, they actively improvise along with the dynamism of specific local interactions and situations. However, currently sensors and attendant technologies merely leave space for them to engage in ambiguities and negotiations concerning the extent and boundaries of technical solutionism rather than opening up possibilities for empowerment to refine the forms of governance.
Sensors as media and sensor-mediated communication: an introduction to the special issue

Abstract This special issue examines mediated communication through the rise of sensors. Sensors are increasingly in the phones we carry, in the cars we drive, and throughout the homes and communities in which we live. In this introduction to the special issue, we define sensor-mediated communication (SMC) and argue the embedded, automatic, and datafied nature of sensors belie the glitches and biases in sensor mechanisms, networks, and infrastructure. The collection of articles in this issue explores SMC across a variety of contexts and cases, including municipal infrastructure, community, health, industry, and the domestic. They represent studies of voice assistants, self-tracking apps, self-driving cars, fitness games, home health care, as well as municipal sensor networks in urban, indigenous, and rural communities. Across them all we see the different ways through which mediated communication is initiated, transformed, and maintained by sensing technologies. Together they represent an important evolution in the study of computer-mediated communication.
A lightweight multi-feature fusion structure for automatic modulation classification

Automatic modulation classification (AMC) is a useful technology for Internet of Things devices to automatically distinguish the modulation scheme of the received signal. Recently, deep learning methods have been a popular solution for AMC, but these methods generate unsatisfactorily high model complexity while achieving acceptable classification accuracy. Therefore, developing an AMC method that can balance classification accuracy and model complexity is a challenging task, especially when targeting resource-constrained devices. In this paper, we propose a lightweight multi-feature fusion structure (lightMFFS) for AMC. This structure consists of three parts, namely data processing, feature extraction, and classification. Specifically, in the data processing part, we adopt three types of signal data, including IQ data, AP data, and IQ-transformed data. In the feature extraction part, we design three lightweight extractors to extract different spatial features of the signal from different data. In the classification part, an attention fuser is used to integrate all the features and enhance the important channel features to provide comprehensive and effective signal knowledge for classification. Extensive comparative experiments with current popular models show that our method achieves higher classification accuracy with fewer training parameters and has a more stable and superior learning capability.
Multi-Normal-Inverse Wishart mixture distribution based nonlinear filter with applications

In Kalman filter-based spacecraft autonomous celestial navigation, Inertial/Celestial integrated navigation system and asynchronous multi-sensor data fusion, complex environments and sensor alignment errors are likely to lead to inaccurate statistical priors for the noise covariance matrices and the non-zero measurement noise mean vector (MNMV). To address this issue, this paper firstly proposes a Multi-Normal-Inverse Wishart (MNIW) mixture distribution modelling the joint probability density function (PDF) for one-step prediction and measure likelihood, then the MNIW mixture distribution is decomposed into a Gaussian hierarchy, and finally the posterior estimates of the state and variables are obtained using variational Bayesian technique. In this paper, a Multi-Normal-Inverse Wishart mixture distribution-based variational Bayesian extended Kalman filter (VB-EKF) is proposed, in which a first-order Taylor expansion is used to address the non-linear problem. The proposed filter can be used to address non-linear filtering problem with inaccurate noise covariance matrices and measurement bias. Simulations of spacecraft autonomous celestial navigation validate the effectiveness and superiority of the proposed filter.
MAFH: Multilabel aware framework for bit-scalable cross-modal hashing

Deep hashing techniques have received considerable attention and are widely used in the field of cross-modal retrieval. The training mode of deep cross-modal hashing models has also gradually converged. However, the commonly used training mode ignores noisy labels from different modality original datasets and can only generate fixed-length hash codes, resulting in poor robustness and flexibility of the training generated hash models. Therefore, to effectively optimize the training mode, we propose a multilabel aware framework for bit-scalable deep cross-modal hashing (MAFH) with the following contributions. First, we introduce a label network module to indirectly used the hash representation of multilabels to more accurately supervise the model training process across all modalities. Second, we design a scalable coding strategy to freely manipulate the length of the hash code, which makes the trained model flexible to adapt to different scenarios. Third, we propose a layered multilabel similarity algorithm to preserve more complete feature information, which avoids the loss of discriminative feature information during model training. The training mode of MAFH can effectively enhance model robustness, flexibility and accuracy. We selected thirteen representative hash methods to compare with the proposed method, and experimental results on four public datasets show that the proposed method achieves good performance. https://github.com/x-28/MAFH.git.
Mutually improved response generation and dialogue summarization for multi-domain task-oriented dialogue systems

With the development of pre-trained language models (PrLM), the research of PrLM-based multi-domain task-oriented dialogue systems (TOD) has attracted growing attention and has achieved great progress. However, most current studies suffer from two problems. First, they model dialogue state tracking as an independent subtask supervised by slot-value pairs, resulting in poor adaptability when transferring to new task domains. Second, these studies ignore the fact that not all dialogue histories are valuable for the ensuing turns as they increase in length. To tackle these two issues, we propose a simple and novel framework to explore multi-domain TOD by jointly training response generation and dialogue summarization with PrLM as the backbone. Specifically, first, we use fluent text generated by the dialogue summarization model to replace formatted dialogue states, treating the dialogue state identification task as a natural language generation task, which allows dialogue state tracking to be easily extended to new task domains. Second, dialogue summarization removes redundant and useless information from the current dialogue process and is fed into the response decoder enabling the system to focus on crucial details in long dialogues. Furthermore, we employ a dialogue chunk detector to assist the dialogue summarization model and design a fusion mechanism to dynamically integrate helpful dialogue summarization into the response generation process. Experimental results show that the proposed model achieves state-of-the-art performance in both automatic and human evaluations on two public datasets and demonstrate that the joint framework can benefit both tasks from each other.
Increasing visitors attention with introductory portal technology to complex cultural sites

We investigate the effectiveness of visually impacting interfaces which were located next to the entrance of the San Martino Charterhouse in Naples (Italy), by using high quality 3D reconstructions annotated with semantic information. Semantic data were used to develop a gesture based orienting interface, for the generation of camera movements and pointing gestures for a 3D avatar. Using the Visitor Employed Photography protocol, we collected data about visitors noticing details and discovered that those exposed to the informative systems were able to detect more details than visitors who did not experience it.
Reinforcement learning for few-shot text generation adaptation

This paper proposes a novel method based on reinforcement learning (RL) to control the generation model in adapting to new domains with limited samples. To avoid the problem of overfitting, the method combines maximum likelihood estimation (MLE) with RL process to improve the sample utilization rate and reduce the sample requirement of RL. The training process is divided into two parts: pre-training and fine-tuning, to effectively express the semantic of the target domain. In order to ensure the robustness of the reward function, adversarial training is introduced. A new measurement called “Net Accuracy” is proposed to better evaluate the domain relevance of the generated text and eliminate the problem of inaccurate domain relevance measurement caused by overfitting and generating a large amount of duplicate text. Finally, experimental results show the effectiveness and superiority of the proposed method in five target domains.
RAPID: Enabling fast online policy learning in dynamic public cloud environments

Resource sharing between multiple workloads has become a prominent practice among cloud service providers, motivated by demand for improved resource utilization and reduced cost of ownership. Effective resource sharing, however, remains an open challenge due to the adverse effects that resource contention can have on high-priority, user-facing workloads with strict Quality of Service (QoS) requirements. Although recent approaches have demonstrated promising results, those works remain largely impractical in public cloud environments since workloads are not known in advance and may only run for a brief period, thus prohibiting offline learning and significantly hindering online learning. In this paper, we propose RAPID, a novel framework for fast, fully-online resource allocation policy learning in highly dynamic operating environments. RAPID leverages lightweight QoS predictions, enabled by domain-knowledge-inspired techniques for sample efficiency and bias reduction, to decouple control from conventional feedback sources and guide policy learning at a rate orders of magnitude faster than prior work. Evaluation on a real-world server platform with representative cloud workloads confirms that RAPID can learn stable resource allocation policies in minutes, as compared with hours in prior state-of-the-art, while improving QoS by 9.0x and increasing best-effort workload performance by 19%–43%.
Handwriting word spotting in the space of difference between representations using vision transformers

Word spotting in handwritten documents is challenging due to the high intra-class and inter-class variability of handwritten forms. This paper addresses the word spotting problem in the segmentation and the training scenarios. Overall, this paper makes the following three contributions: (1) a new word text representation, called the Pyramid of Bidirectional Character Sequences (PBCS), which can solve both the word spotting problem and the word recognition problem. The use of the PBCS representation allows trained models to identify the character subsequences shared by words. Thus, words that are not seen during training can be represented and spotted. In addition, the PBCS representation encodes word texts redundantly, allowing for word discrimination. (2) A binary classification modeling of the word spotting problem in the difference space between representations, where spotting of non-vocabulary words is more efficient. Finally, (3) a new deep neural network architecture that combines the strengths of convolutional layers and transformers. We evaluated our solution on IAM and RIEMS datasets and showed that it outperforms recent state-of-the-art methods in the query-by-example scenario.
Multi-relational dynamic graph representation learning

In recent years, many dynamic graph representation learning methods have emerged due to the ubiquity of dynamic graph networks, such as social networks, medical networks, citation networks and traffic networks. Many researchers only consider the unitary relational topological information in a dynamic graph. However, real dynamic networks contain a large amount of multi-relational topological information. For example, there are different interactive relations such as sending a message, adding a friend, making a phone call, and sending an email in a social network, and they have different effects on node representation and should be distinguished. In addition, the non-topological information of nodes plays an important role in the node representation. Although these two types of information have been shown to improve the performance of many dynamic graph tasks, existing dynamic graph representation learning models could not integrate them well. Therefore, in this paper, we propose MRDGNN, a Multi-Relational Dynamic Graph Neural Network model, which can capture the dynamic evolution under each relational topology in the graph through a temporal multi-relational topology updater, including the participation of multi-relational topological and non-topological information of nodes. These two kinds of information will be adaptively fused into the representation of nodes by a merger. MRDGNN is continuously updated with the evolution of dynamic graphs and is a real-time learnable representation learning framework. Finally, we validate the effectiveness of MRDGNN for link prediction and relation prediction on four real datasets.
Noise sensitivity and stability of deep neural networks for binary classification

A first step is taken towards understanding often observed non-robustness phenomena of deep neural net (DNN) classifiers. This is done from the perspective of Boolean functions by asking if certain sequences of Boolean functions represented by common DNN models are noise sensitive or noise stable, concepts defined in the Boolean function literature. Due to the natural randomness in DNN models, these concepts are extended to annealed and quenched versions. Here we sort out the relation between these definitions and investigate the properties of two standard DNN architectures, the fully connected and convolutional models, when initiated with Gaussian weights.
A redesigned secure and efficient data transaction protocol for mobile payment system

Generally, the secure protocol was proposed from the certificateless signature (CLS) scheme to provide the corresponding security for the actual application. However, for the previous application protocols, most of the underlying CLS schemes cannot have their claimed security, because some proposals cannot resist forgery attacks or the master secret key can be captured by the probabilistic polynomial time adversary. To further address the above problems, we take several previous CLS schemes as instances to make a detailed analysis of the causes of the above security flaws and point out that, in a secure CLS scheme, we should appropriately publish the public parameters. Based on the above analysis, we design an efficient and secure CLS scheme in this paper, and the unforgeability is proved based on the hardness of discrete logarithm complexity assumption through the Forking lemma. Compared with the existing constructions, our new proposal has advantages in communication and computation efficiency but also has higher security. Furthermore, a concrete construction of a certificateless aggregate signature (CLAS) scheme is designed with the above basic CLS scheme to realize efficient verification of multiple signatures. Finally, we further improve the key conditions that need to be paid attention to design a secure CLS scheme.