OpenRaiser/Intern-Atlas huggingface dataset:
How to Do Things with Deep Learning Code
Venue: arXiv cs.CL 2023
Year: 2023

The premise of this article is that a basic understanding of the composition and functioning of large language models is critically urgent. To that end, we extract a representational map of OpenAI's GPT-2 with what we articulate as two classes of deep learning code, that which pertains to the model and that which underwrites applications built around the model. We then verify this map through case studies of two popular GPT-2 applications: the text adventure game, AI Dungeon, and the language art project, This Word Does Not Exist. Such an exercise allows us to test the potential of Critical Code Studies when the object of study is deep learning code and to demonstrate the validity of code as an analytical focus for researchers in the subfields of Critical Artificial Intelligence and Critical Machine Learning Studies. More broadly, however, our work draws attention to the means by which ordinary users might interact with, and even direct, the behavior of deep learning systems, and by extension works toward demystifying some of the auratic mystery of "AI." What is at stake is the possibility of achieving an informed sociotechnical consensus about the responsible applications of large language models, as well as a more expansive sense of their creative capabilities-indeed, understanding how and where engagement occurs allows all of us to become more active participants in the development of machine learning systems.
hDesigner: Real-Time Haptic Feedback Pattern Designer
Venue: arXiv cs.HC 2023
Year: 2023

Haptic sensing can provide a new dimension to enhance people's musical and cinematic experiences. However, designing a haptic pattern is neither intuitive nor trivial. Imagined haptic patterns tend to be different from experienced ones. As a result, researchers use simple step-curve patterns to create haptic stimuli. To this end, we designed and developed an intuitive haptic pattern designer that lets you rapidly prototype creative patterns. Our simple architecture, wireless connectivity, and easy-to-program communication protocol make it modular and easy to scale. In this demo, workshop participants can select from a library of haptic patterns and design new ones. They can feel the pattern as they make changes in the user interface. With this new workflow, researchers and artists can design and rapidly test haptic patterns for downstream tasks such as research experiments or create new musical and cinematic experiences. More details about the project are available at https://www.snehesh.com/hDesigner
TieFake: Title-Text Similarity and Emotion-Aware Fake News Detection
Venue: arXiv cs.CL 2023
Year: 2023

Fake news detection aims to detect fake news widely spreading on social media platforms, which can negatively influence the public and the government. Many approaches have been developed to exploit relevant information from news images, text, or videos. However, these methods may suffer from the following limitations: (1) ignore the inherent emotional information of the news, which could be beneficial since it contains the subjective intentions of the authors; (2) pay little attention to the relation (similarity) between the title and textual information in news articles, which often use irrelevant title to attract reader' attention. To this end, we propose a novel Title-Text similarity and emotion-aware Fake news detection (TieFake) method by jointly modeling the multi-modal context information and the author sentiment in a unified framework. Specifically, we respectively employ BERT and ResNeSt to learn the representations for text and images, and utilize publisher emotion extractor to capture the author's subjective emotion in the news content. We also propose a scale-dot product attention mechanism to capture the similarity between title features and textual features. Experiments are conducted on two publicly available multi-modal datasets, and the results demonstrate that our proposed method can significantly improve the performance of fake news detection. Our code is available at https://github.com/UESTC-GQJ/TieFake.
HyperStyle3D: Text-Guided 3D Portrait Stylization via Hypernetworks
Venue: arXiv cs.CV 2023
Year: 2023

Portrait stylization is a long-standing task enabling extensive applications. Although 2D-based methods have made great progress in recent years, real-world applications such as metaverse and games often demand 3D content. On the other hand, the requirement of 3D data, which is costly to acquire, significantly impedes the development of 3D portrait stylization methods. In this paper, inspired by the success of 3D-aware GANs that bridge 2D and 3D domains with 3D fields as the intermediate representation for rendering 2D images, we propose a novel method, dubbed HyperStyle3D, based on 3D-aware GANs for 3D portrait stylization. At the core of our method is a hyper-network learned to manipulate the parameters of the generator in a single forward pass. It not only offers a strong capacity to handle multiple styles with a single model, but also enables flexible fine-grained stylization that affects only texture, shape, or local part of the portrait. While the use of 3D-aware GANs bypasses the requirement of 3D data, we further alleviate the necessity of style images with the CLIP model being the stylization guidance. We conduct an extensive set of experiments across the style, attribute, and shape, and meanwhile, measure the 3D consistency. These experiments demonstrate the superior capability of our HyperStyle3D model in rendering 3D-consistent images in diverse styles, deforming the face shape, and editing various attributes.
Secure Mobile Payment Architecture Enabling Multi-factor Authentication
Venue: arXiv cs.CR 2023
Year: 2023

The rise of smartphones has led to a significant increase in the usage of mobile payments. Mobile payments allow individuals to access financial resources and make transactions through their mobile devices while on the go. However, the current mobile payment systems were designed to align with traditional payment structures, which limits the full potential of smartphones, including their security features. This has become a major concern in the rapidly growing mobile payment market. To address these security concerns,in this paper we propose new mobile payment architecture. This architecture leverages the advanced capabilities of modern smartphones to verify various aspects of a payment, such as funds, biometrics, location, and others. The proposed system aims to guarantee the legitimacy of transactions and protect against identity theft by verifying multiple elements of a payment. The security of mobile payment systems is crucial, given the rapid growth of the market. Evaluating mobile payment systems based on their authentication, encryption, and fraud detection capabilities is of utmost importance. The proposed architecture provides a secure mobile payment solution that enhances the overall payment experience by taking advantage of the advanced capabilities of modern smartphones. This will not only improve the security of mobile payments but also offer a more user-friendly payment experience for consumers.
Learning Robust Visual-Semantic Embedding for Generalizable Person Re-identification
Venue: arXiv cs.CV 2023
Year: 2023

Generalizable person re-identification (Re-ID) is a very hot research topic in machine learning and computer vision, which plays a significant role in realistic scenarios due to its various applications in public security and video surveillance. However, previous methods mainly focus on the visual representation learning, while neglect to explore the potential of semantic features during training, which easily leads to poor generalization capability when adapted to the new domain. In this paper, we propose a Multi-Modal Equivalent Transformer called MMET for more robust visual-semantic embedding learning on visual, textual and visual-textual tasks respectively. To further enhance the robust feature learning in the context of transformer, a dynamic masking mechanism called Masked Multimodal Modeling strategy (MMM) is introduced to mask both the image patches and the text tokens, which can jointly works on multimodal or unimodal data and significantly boost the performance of generalizable person Re-ID. Extensive experiments on benchmark datasets demonstrate the competitive performance of our method over previous approaches. We hope this method could advance the research towards visual-semantic representation learning. Our source code is also publicly available at https://github.com/JeremyXSC/MMET.
Secure Split Learning against Property Inference, Data Reconstruction, and Feature Space Hijacking Attacks
Venue: arXiv cs.LG 2023
Year: 2023

Split learning of deep neural networks (SplitNN) has provided a promising solution to learning jointly for the mutual interest of a guest and a host, which may come from different backgrounds, holding features partitioned vertically. However, SplitNN creates a new attack surface for the adversarial participant, holding back its practical use in the real world. By investigating the adversarial effects of highly threatening attacks, including property inference, data reconstruction, and feature hijacking attacks, we identify the underlying vulnerability of SplitNN and propose a countermeasure. To prevent potential threats and ensure the learning guarantees of SplitNN, we design a privacy-preserving tunnel for information exchange between the guest and the host. The intuition is to perturb the propagation of knowledge in each direction with a controllable unified solution. To this end, we propose a new activation function named R3eLU, transferring private smashed data and partial loss into randomized responses in forward and backward propagations, respectively. We give the first attempt to secure split learning against three threatening attacks and present a fine-grained privacy budget allocation scheme. The analysis proves that our privacy-preserving SplitNN solution provides a tight privacy budget, while the experimental results show that our solution performs better than existing solutions in most cases and achieves a good tradeoff between defense and model usability.
Decadal Temperature Prediction via Chaotic Behavior Tracking
Venue: arXiv cs.LG 2023
Year: 2023

Decadal temperature prediction provides crucial information for quantifying the expected effects of future climate changes and thus informs strategic planning and decision-making in various domains. However, such long-term predictions are extremely challenging, due to the chaotic nature of temperature variations. Moreover, the usefulness of existing simulation-based and machine learning-based methods for this task is limited because initial simulation or prediction errors increase exponentially over time. To address this challenging task, we devise a novel prediction method involving an information tracking mechanism that aims to track and adapt to changes in temperature dynamics during the prediction phase by providing probabilistic feedback on the prediction error of the next step based on the current prediction. We integrate this information tracking mechanism, which can be considered as a model calibrator, into the objective function of our method to obtain the corrections needed to avoid error accumulation. Our results show the ability of our method to accurately predict global land-surface temperatures over a decadal range. Furthermore, we demonstrate that our results are meaningful in a real-world context: the temperatures predicted using our method are consistent with and can be used to explain the well-known teleconnections within and between different continents.
DynamicRead: Exploring Robust Gaze Interaction Methods for Reading on Handheld Mobile Devices under Dynamic Conditions
Venue: arXiv cs.HC 2023
Year: 2023

Enabling gaze interaction in real-time on handheld mobile devices has attracted significant attention in recent years. An increasing number of research projects have focused on sophisticated appearance-based deep learning models to enhance the precision of gaze estimation on smartphones. This inspires important research questions, including how the gaze can be used in a real-time application, and what type of gaze interaction methods are preferable under dynamic conditions in terms of both user acceptance and delivering reliable performance. To address these questions, we design four types of gaze scrolling techniques: three explicit technique based on Gaze Gesture, Dwell time, and Pursuit; and one implicit technique based on reading speed to support touch-free, page-scrolling on a reading application. We conduct a 20-participant user study under both sitting and walking settings and our results reveal that Gaze Gesture and Dwell time-based interfaces are more robust while walking and Gaze Gesture has achieved consistently good scores on usability while not causing high cognitive workload.
A Survey of Developable Surfaces: From Shape Modeling to Manufacturing
Venue: arXiv cs.GR 2023
Year: 2023

Developable surfaces are commonly observed in various applications such as architecture, product design, manufacturing, mechanical materials, and data physicalization as well as in the development of tangible interaction and deformable robots, with the characteristics of easy-to-product, low-cost, transport-friendly, and deformable. Transforming shapes into developable surfaces is a complex and comprehensive task, which forms a variety of methods of segmentation, unfolding, and manufacturing for shapes with different geometry and topology, resulting in the complexity of developable surfaces. In this paper, we reviewed relevant methods and techniques for the study of developable surfaces, characterize them with our proposed pipeline, and categorize them based on digital modeling, physical modeling, interaction, and application. Through the analysis to the relevant literature, we also discussed some of the research challenges and future research opportunities.
CASOG: Conservative Actor-critic with SmOoth Gradient for Skill Learning in Robot-Assisted Intervention
Venue: arXiv cs.RO 2023
Year: 2023

Robot-assisted intervention has shown reduced radiation exposure to physicians and improved precision in clinical trials. However, existing vascular robotic systems follow master-slave control mode and entirely rely on manual commands. This paper proposes a novel offline reinforcement learning algorithm, Conservative Actor-critic with SmOoth Gradient (CASOG), to learn manipulation skills from human demonstrations on vascular robotic systems. The proposed algorithm conservatively estimates Q-function and smooths gradients of convolution layers to deal with distribution shift and overfitting issues. Furthermore, to focus on complex manipulations, transitions with larger temporal-difference error are sampled with higher probability. Comparative experiments in a pre-clinical environment demonstrate that CASOG can deliver guidewire to the target at a success rate of 94.00\% and mean backward steps of 14.07, performing closer to humans and better than prior offline reinforcement learning methods. These results indicate that the proposed algorithm is promising to improve the autonomy of vascular robotic systems.
Resource Allocation in the RIS Assisted SCMA Cellular Network Coexisting with D2D Communications
Venue: arXiv cs.IT 2023
Year: 2023

The cellular network coexisting with device-to-device (D2D) communications has been studied extensively. Reconfigurable intelligent surface (RIS) and non-orthogonal multiple access (NOMA) are promising technologies for the evolution of 5G, 6G and beyond. Besides, sparse code multiple access (SCMA) is considered suitable for next-generation wireless network in code-domain NOMA. In this paper, we consider the RIS-aided uplink SCMA cellular network simultaneously with D2D users. We formulate the optimization problem which aims to maximize the cellular sum-rate by jointly designing D2D users resource block (RB) association, the transmitted power for both cellular users and D2D users, and the phase shifts at the RIS. The power limitation and users communication requirements are considered. The problem is non-convex, and it is challenging to solve it directly. To handle this optimization problem, we propose an efficient iterative algorithm based on block coordinate descent (BCD) method. The original problem is decoupled into three subproblems to solve separately. Simulation results demonstrate that the proposed scheme can significantly improve the sum-rate performance over various schemes.
From Artifacts to Outcomes: Comparison of HMD VR, Desktop, and Slides Lectures for Food Microbiology Laboratory Instruction
Venue: arXiv cs.HC 2023
Year: 2023

Despite the value of VR (Virtual Reality) for educational purposes, the instructional power of VR in Biology Laboratory education remains under-explored. Laboratory lectures can be challenging due to students' low motivation to learn abstract scientific concepts and low retention rate. Therefore, we designed a VR-based lecture on fermentation and compared its effectiveness with lectures using PowerPoint slides and a desktop application. Grounded in the theory of distributed cognition and motivational theories, our study examined how learning happens in each condition from students' learning outcomes, behaviors, and perceptions. Our result indicates that VR facilitates students' long-term retention to learn by cultivating their longer visual attention and fostering a higher sense of immersion, though students' short-term retention remains the same across all conditions. This study extends current research on VR studies by identifying the characteristics of each teaching artifact and providing design implications for integrating VR technology into higher education.
GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information
Venue: arXiv cs.CL 2023
Year: 2023

While large language models (LLMs) have been successfully applied to various tasks, they still face challenges with hallucinations. Augmenting LLMs with domain-specific tools such as database utilities can facilitate easier and more precise access to specialized knowledge. In this paper, we present GeneGPT, a novel method for teaching LLMs to use the Web APIs of the National Center for Biotechnology Information (NCBI) for answering genomics questions. Specifically, we prompt Codex to solve the GeneTuring tests with NCBI Web APIs by in-context learning and an augmented decoding algorithm that can detect and execute API calls. Experimental results show that GeneGPT achieves state-of-the-art performance on eight tasks in the GeneTuring benchmark with an average score of 0.83, largely surpassing retrieval-augmented LLMs such as the new Bing (0.44), biomedical LLMs such as BioMedLM (0.08) and BioGPT (0.04), as well as GPT-3 (0.16) and ChatGPT (0.12). Our further analyses suggest that: (1) API demonstrations have good cross-task generalizability and are more useful than documentations for in-context learning; (2) GeneGPT can generalize to longer chains of API calls and answer multi-hop questions in GeneHop, a novel dataset introduced in this work; (3) Different types of errors are enriched in different tasks, providing valuable insights for future improvements.
Grooming Connectivity Intents in IP-Optical Networks Using Directed Acyclic Graphs
Venue: arXiv cs.NI 2023
Year: 2023

During the last few years, there have been concentrated efforts toward intent-driven networking. While relying upon Software-Defined Networking (SDN), Intent-Based Networking (IBN) pushes the frontiers of efficient networking by decoupling the intentions of a network operator (i.e., what is desired to be done) from the implementation (i.e., how is it achieved). The advantages of such a paradigm have long been argued and include, but are not limited to, the reduction of human errors, reduced expertise requirements among operator personnel, and faster business plan adaptation. In previous work, we have shown how incorporating IBN in multi-domain networks can have a significantly positive impact as it can enable decentralized operation, accountability, and confidentiality. The pillar of our previous contribution is the compilation of intents using system-generated intent trees. In this work, we extend the architecture to enable grooming among the user intents. Therefore, separate intents can now end up using the same network resources. While this makes the intent system reasonably more complex, it indisputably improves resource allocation. To represent the intent relationships of the newly enhanced architecture, we use Directed Acyclic Graphs (DAGs). Furthermore, we appropriately adapt an advanced established technique from the literature to solve the Routing, Modulation, and Spectrum Assignment (RMSA) problem for the intent compilation. We demonstrate a realistic scenario in which we evaluate our architecture and the intent compilation strategy. Our current approach successfully consolidates the advantages of having an intent-driven architecture and, at the same time, flexibly choosing among advanced resource allocation techniques.
A Multi-robot Coverage Path Planning Algorithm Based on Improved DARP Algorithm
Venue: arXiv cs.RO 2023
Year: 2023

The research on multi-robot coverage path planning (CPP) has been attracting more and more attention. In order to achieve efficient coverage, this paper proposes an improved DARP coverage algorithm. The improved DARP algorithm based on A* algorithm is used to assign tasks to robots and then combined with STC algorithm based on Up-First algorithm to achieve full coverage of the task area. Compared with the initial DARP algorithm, this algorithm has higher efficiency and higher coverage rate.
Advances on Concept Drift Detection in Regression Tasks using Social Networks Theory
Venue: arXiv cs.LG 2023
Year: 2023

Mining data streams is one of the main studies in machine learning area due to its application in many knowledge areas. One of the major challenges on mining data streams is concept drift, which requires the learner to discard the current concept and adapt to a new one. Ensemble-based drift detection algorithms have been used successfully to the classification task but usually maintain a fixed size ensemble of learners running the risk of needlessly spending processing time and memory. In this paper we present improvements to the Scale-free Network Regressor (SFNR), a dynamic ensemble-based method for regression that employs social networks theory. In order to detect concept drifts SFNR uses the Adaptive Window (ADWIN) algorithm. Results show improvements in accuracy, especially in concept drift situations and better performance compared to other state-of-the-art algorithms in both real and synthetic data.
Automatic Interaction and Activity Recognition from Videos of Human Manual Demonstrations with Application to Anomaly Detection
Venue: arXiv cs.CV 2023
Year: 2023

This paper presents a new method to describe spatio-temporal relations between objects and hands, to recognize both interactions and activities within video demonstrations of manual tasks. The approach exploits Scene Graphs to extract key interaction features from image sequences while simultaneously encoding motion patterns and context. Additionally, the method introduces event-based automatic video segmentation and clustering, which allow for the grouping of similar events and detect if a monitored activity is executed correctly. The effectiveness of the approach was demonstrated in two multi-subject experiments, showing the ability to recognize and cluster hand-object and object-object interactions without prior knowledge of the activity, as well as matching the same activity performed by different subjects.