OpenRaiser/Intern-Atlas huggingface dataset:
Koopman Invertible Autoencoder: Leveraging Forward and Backward Dynamics for Temporal Modeling
Venue: arXiv cs.LG 2023
Year: 2023

Accurate long-term predictions are the foundations for many machine learning applications and decision-making processes. However, building accurate long-term prediction models remains challenging due to the limitations of existing temporal models like recurrent neural networks (RNNs), as they capture only the statistical connections in the training data and may fail to learn the underlying dynamics of the target system. To tackle this challenge, we propose a novel machine learning model based on Koopman operator theory, which we call Koopman Invertible Autoencoders (KIA), that captures the inherent characteristic of the system by modeling both forward and backward dynamics in the infinite-dimensional Hilbert space. This enables us to efficiently learn low-dimensional representations, resulting in more accurate predictions of long-term system behavior. Moreover, our method's invertibility design guarantees reversibility and consistency in both forward and inverse operations. We illustrate the utility of KIA on pendulum and climate datasets, demonstrating 300% improvements in long-term prediction capability for pendulum while maintaining robustness against noise. Additionally, our method excels in long-term climate prediction, further validating our method's effectiveness.
Julia as a unifying end-to-end workflow language on the Frontier exascale system
Venue: arXiv cs.DC 2023
Year: 2023

We evaluate Julia as a single language and ecosystem paradigm powered by LLVM to develop workflow components for high-performance computing. We run a Gray-Scott, 2-variable diffusion-reaction application using a memory-bound, 7-point stencil kernel on Frontier, the US Department of Energy's first exascale supercomputer. We evaluate the performance, scaling, and trade-offs of (i) the computational kernel on AMD's MI250x GPUs, (ii) weak scaling up to 4,096 MPI processes/GPUs or 512 nodes, (iii) parallel I/O writes using the ADIOS2 library bindings, and (iv) Jupyter Notebooks for interactive analysis. Results suggest that although Julia generates a reasonable LLVM-IR, a nearly 50% performance difference exists vs. native AMD HIP stencil codes when running on the GPUs. As expected, we observed near-zero overhead when using MPI and parallel I/O bindings for system-wide installed implementations. Consequently, Julia emerges as a compelling high-performance and high-productivity workflow composition language, as measured on the fastest supercomputer in the world.
Rigorously Assessing Natural Language Explanations of Neurons
Venue: arXiv cs.CL 2023
Year: 2023

Natural language is an appealing medium for explaining how large language models process and store information, but evaluating the faithfulness of such explanations is challenging. To help address this, we develop two modes of evaluation for natural language explanations that claim individual neurons represent a concept in a text input. In the observational mode, we evaluate claims that a neuron $a$ activates on all and only input strings that refer to a concept picked out by the proposed explanation $E$. In the intervention mode, we construe $E$ as a claim that the neuron $a$ is a causal mediator of the concept denoted by $E$. We apply our framework to the GPT-4-generated explanations of GPT-2 XL neurons of Bills et al. (2023) and show that even the most confident explanations have high error rates and little to no causal efficacy. We close the paper by critically assessing whether natural language is a good choice for explanations and whether neurons are the best level of analysis.
Who to Trust, How and Why: Untangling AI Ethics Principles, Trustworthiness and Trust
Venue: arXiv cs.AI 2023
Year: 2023

We present an overview of the literature on trust in AI and AI trustworthiness and argue for the need to distinguish these concepts more clearly and to gather more empirically evidence on what contributes to people s trusting behaviours. We discuss that trust in AI involves not only reliance on the system itself, but also trust in the developers of the AI system. AI ethics principles such as explainability and transparency are often assumed to promote user trust, but empirical evidence of how such features actually affect how users perceive the system s trustworthiness is not as abundance or not that clear. AI systems should be recognised as socio-technical systems, where the people involved in designing, developing, deploying, and using the system are as important as the system for determining whether it is trustworthy. Without recognising these nuances, trust in AI and trustworthy AI risk becoming nebulous terms for any desirable feature for AI systems.
A First Look at SVCB and HTTPS DNS Resource Records in the Wild
Venue: arXiv cs.NI 2023
Year: 2023

The Internet Engineering Task Force is standardizing new DNS resource records, namely SVCB and HTTPS. Both records inform clients about endpoint and service properties such as supported application layer protocols, IP address hints or Encrypted Client Hello (ECH) information. Therefore, they allow clients to reduce required DNS queries and potential retries during connection establishment and thus help to improve the quality of experience and privacy of the client. The latter is achieved by reducing visible meta-data, which is further improved with encrypted DNS and ECH. The standardization is in its final stages and companies announced support, e.g., Cloudflare and Apple. Therefore, we provide the first large-scale overview of actual record deployment by analyzing more than 400 M domains. We find 3.96 k SVCB and 10.5 M HTTPS records. As of March 2023, Cloudflare hosts and serves most domains, and most records only contain Application-Layer Protocol Negotiation (ALPN) and IP address hints. Besides Cloudflare, we see adoption by a variety of authoritative name servers and hosting providers indicating increased adoption in the near future. Lastly, we can verify the correctness of records for more than 93 % of domains based on three application layer scans.
Language Guided Adversarial Purification
Venue: arXiv cs.LG 2023
Year: 2023

Adversarial purification using generative models demonstrates strong adversarial defense performance. These methods are classifier and attack-agnostic, making them versatile but often computationally intensive. Recent strides in diffusion and score networks have improved image generation and, by extension, adversarial purification. Another highly efficient class of adversarial defense methods known as adversarial training requires specific knowledge of attack vectors, forcing them to be trained extensively on adversarial examples. To overcome these limitations, we introduce a new framework, namely Language Guided Adversarial Purification (LGAP), utilizing pre-trained diffusion models and caption generators to defend against adversarial attacks. Given an input image, our method first generates a caption, which is then used to guide the adversarial purification process through a diffusion network. Our approach has been evaluated against strong adversarial attacks, proving its effectiveness in enhancing adversarial robustness. Our results indicate that LGAP outperforms most existing adversarial defense techniques without requiring specialized network training. This underscores the generalizability of models trained on large datasets, highlighting a promising direction for further research.
Pointing out Human Answer Mistakes in a Goal-Oriented Visual Dialogue
Venue: arXiv cs.CV 2023
Year: 2023

Effective communication between humans and intelligent agents has promising applications for solving complex problems. One such approach is visual dialogue, which leverages multimodal context to assist humans. However, real-world scenarios occasionally involve human mistakes, which can cause intelligent agents to fail. While most prior research assumes perfect answers from human interlocutors, we focus on a setting where the agent points out unintentional mistakes for the interlocutor to review, better reflecting real-world situations. In this paper, we show that human answer mistakes depend on question type and QA turn in the visual dialogue by analyzing a previously unused data collection of human mistakes. We demonstrate the effectiveness of those factors for the model's accuracy in a pointing-human-mistake task through experiments using a simple MLP model and a Visual Language Model.
SideGAN: 3D-Aware Generative Model for Improved Side-View Image Synthesis
Venue: arXiv cs.CV 2023
Year: 2023

While recent 3D-aware generative models have shown photo-realistic image synthesis with multi-view consistency, the synthesized image quality degrades depending on the camera pose (e.g., a face with a blurry and noisy boundary at a side viewpoint). Such degradation is mainly caused by the difficulty of learning both pose consistency and photo-realism simultaneously from a dataset with heavily imbalanced poses. In this paper, we propose SideGAN, a novel 3D GAN training method to generate photo-realistic images irrespective of the camera pose, especially for faces of side-view angles. To ease the challenging problem of learning photo-realistic and pose-consistent image synthesis, we split the problem into two subproblems, each of which can be solved more easily. Specifically, we formulate the problem as a combination of two simple discrimination problems, one of which learns to discriminate whether a synthesized image looks real or not, and the other learns to discriminate whether a synthesized image agrees with the camera pose. Based on this, we propose a dual-branched discriminator with two discrimination branches. We also propose a pose-matching loss to learn the pose consistency of 3D GANs. In addition, we present a pose sampling strategy to increase learning opportunities for steep angles in a pose-imbalanced dataset. With extensive validation, we demonstrate that our approach enables 3D GANs to generate high-quality geometries and photo-realistic images irrespective of the camera pose.
Hierarchical Modeling of Spatial Cues via Spherical Harmonics for Multi-Channel Speech Enhancement
Venue: arXiv cs.SD 2023
Year: 2023

Multi-channel speech enhancement utilizes spatial information from multiple microphones to extract the target speech. However, most existing methods do not explicitly model spatial cues, instead relying on implicit learning from multi-channel spectra. To better leverage spatial information, we propose explicitly incorporating spatial modeling by applying spherical harmonic transforms (SHT) to the multi-channel input. In detail, a hierarchical framework is introduced whereby lower order harmonics capturing broader spatial patterns are estimated first, then combined with higher orders to recursively predict finer spatial details. Experiments on TIMIT demonstrate the proposed method can effectively recover target spatial patterns and achieve improved performance over baseline models, using fewer parameters and computations. Explicitly modeling spatial information hierarchically enables more effective multi-channel speech enhancement.
Lifting star-autonomous structures
Venue: arXiv cs.LO 2023
Year: 2023

For a functor $Q$ from a category $C$ to the category $Pos$ of ordered sets and order-preserving functions, we study liftings of various kind of structures from the base category $C$ to the total(or Grothendieck) category $\int Q$. That lifting a monoidal structure corresponds to giving some lax natural transformation making $Q$ almost monoidal, might be part of folklore in category theory.We rely on and generalize the tools supporting this correspondence so to provide exact conditions for lifting symmetric monoidal closed and star-autonomous structures.A corollary of these characterizations is that, if $Q$ factors as a monoidal functor through $SLatt$, the category of complete lattices and sup-preserving functions, then $\int Q$ is always symmetric monoidalclosed. In this case, we also provide a method, based on the double negation nucleus from quantale theory, to turn $\int Q$ into a star-autonomous category.The theory developed, originally motivated from the categories $P-Set$ of Schalk and de Paiva, yields a wide generalization of Hyland and Schalk construction of star-autonomous categories by means of orthogonality structures.
Enhanced C-V2X Mode 4 to Optimize Age of Information and Reliability for IoV
Venue: arXiv cs.NI 2023
Year: 2023

Internet of vehicles (IoV) has emerged as a key technology to realize real-time vehicular application. For IoV, vehicles adopt cellular vehicle-to-everything (C-V2X) standard to support direct communication among them. C-V2X mode 4 controls resource allocation without the assistance of cellular network, hence it is widely used for IoV. However, C-V2X mode 4 has two drawbacks. First is that vehicles cannot communicate with each other for a period in some case which will cause an increase in age of information (AoI); second is that vehicles may select resource already occupied by others which will deteriorate the reliability. To address the two drawbacks, we propose an enhanced C-V2X mode 4 to optimize AoI and reliability. In addition, we consider the fact that for most vehicular applications, each vehicle periodically requires fresh information of vehicles within a certain distance and propose a new performance metric to evaluate the system AoI for IoV. Furthermore, we construct a platform through integrating SUMO and NS3. We demonstrate the superiority of the enhanced C-V2X mode 4 base on this simulation platform.
Love or Hate? Share or Split? Privacy-Preserving Training Using Split Learning and Homomorphic Encryption
Venue: arXiv cs.CR 2023
Year: 2023

Split learning (SL) is a new collaborative learning technique that allows participants, e.g. a client and a server, to train machine learning models without the client sharing raw data. In this setting, the client initially applies its part of the machine learning model on the raw data to generate activation maps and then sends them to the server to continue the training process. Previous works in the field demonstrated that reconstructing activation maps could result in privacy leakage of client data. In addition to that, existing mitigation techniques that overcome the privacy leakage of SL prove to be significantly worse in terms of accuracy. In this paper, we improve upon previous works by constructing a protocol based on U-shaped SL that can operate on homomorphically encrypted data. More precisely, in our approach, the client applies homomorphic encryption on the activation maps before sending them to the server, thus protecting user privacy. This is an important improvement that reduces privacy leakage in comparison to other SL-based works. Finally, our results show that, with the optimum set of parameters, training with HE data in the U-shaped SL setting only reduces accuracy by 2.65% compared to training on plaintext. In addition, raw training data privacy is preserved.
Retinex-guided Channel-grouping based Patch Swap for Arbitrary Style Transfer
Venue: arXiv cs.CV 2023
Year: 2023

The basic principle of the patch-matching based style transfer is to substitute the patches of the content image feature maps by the closest patches from the style image feature maps. Since the finite features harvested from one single aesthetic style image are inadequate to represent the rich textures of the content natural image, existing techniques treat the full-channel style feature patches as simple signal tensors and create new style feature patches via signal-level fusion, which ignore the implicit diversities existed in style features and thus fail for generating better stylised results. In this paper, we propose a Retinex theory guided, channel-grouping based patch swap technique to solve the above challenges. Channel-grouping strategy groups the style feature maps into surface and texture channels, which prevents the winner-takes-all problem. Retinex theory based decomposition controls a more stable channel code rate generation. In addition, we provide complementary fusion and multi-scale generation strategy to prevent unexpected black area and over-stylised results respectively. Experimental results demonstrate that the proposed method outperforms the existing techniques in providing more style-consistent textures while keeping the content fidelity.
A Cognitively-Inspired Neural Architecture for Visual Abstract Reasoning Using Contrastive Perceptual and Conceptual Processing
Venue: arXiv cs.AI 2023
Year: 2023

We introduce a new neural architecture for solving visual abstract reasoning tasks inspired by human cognition, specifically by observations that human abstract reasoning often interleaves perceptual and conceptual processing as part of a flexible, iterative, and dynamic cognitive process. Inspired by this principle, our architecture models visual abstract reasoning as an iterative, self-contrasting learning process that pursues consistency between perceptual and conceptual processing of visual stimuli. We explain how this new Contrastive Perceptual-Conceptual Network (CPCNet) works using matrix reasoning problems in the style of the well-known Raven's Progressive Matrices intelligence test. Experiments on the machine learning dataset RAVEN show that CPCNet achieves higher accuracy than all previously published models while also using the weakest inductive bias. We also point out a substantial and previously unremarked class imbalance in the original RAVEN dataset, and we propose a new variant of RAVEN -- AB-RAVEN -- that is more balanced in terms of abstract concepts.
Instrument for the assessment of road user automated vehicle acceptance: A pyramid of user needs of automated vehicles
Venue: arXiv cs.HC 2023
Year: 2023

This study proposed a new methodological approach for the assessment of automated vehicle acceptance (AVA) from the perspective of road users inside and outside of AVs pre- and post- AV experience. Users can be drivers and passengers, but also external road users, such as pedestrians, (motor-)cyclists, and other car drivers, interacting with AVs. A pyramid was developed, which provides a hierarchical representation of user needs. Fundamental user needs are organized at the bottom of the pyramid, while higher-level user needs are at the top of the pyramid. The pyramid distinguishes between six levels of needs, which are safety trust, efficiency, comfort and pleasure, social influence, and well-being. Some user needs universally exist across users, while some are user-specific needs. These needs are translated into operationalizable indicators representing items of a questionnaire for the assessment of AVA of users inside and outside AVs. The formulation of the questionnaire items was derived from established technology acceptance models. As the instrument was based on the same model for all road users, the comparison of AVA between different road users is now possible. We recommend future research to validate this questionnaire, administering it in studies to contribute to the development of a short, efficient, and standardized metric for the assessment of AVA.
A Novel Hybrid Algorithm for Optimized Solutions in Ocean Renewable Energy Industry: Enhancing Power Take-Off Parameters and Site Selection Procedure of Wave Energy Converters
Venue: arXiv cs.NE 2023
Year: 2023

Ocean renewable energy, particularly wave energy, has emerged as a pivotal component for diversifying the global energy portfolio, reducing dependence on fossil fuels, and mitigating climate change impacts. This study delves into the optimization of power take-off (PTO) parameters and the site selection process for an offshore oscillating surge wave energy converter (OSWEC). However, the intrinsic dynamics of these interactions, coupled with the multi-modal nature of the optimization landscape, make this a daunting challenge. Addressing this, we introduce the novel Hill Climb - Explorative Gray Wolf Optimizer (HC-EGWO). This new methodology blends a local search method with a global optimizer, incorporating dynamic control over exploration and exploitation rates. This balance paves the way for an enhanced exploration of the solution space, ensuring the identification of superior-quality solutions. Further anchoring our approach, a feasibility landscape analysis based on linear water wave theory assumptions and the flap's maximum angular motion is conducted. This ensures the optimized OSWEC consistently operates within safety and efficiency parameters. Our findings hold significant promise for the development of more streamlined OSWEC power take-off systems. They provide insights for selecting the prime offshore site, optimizing power output, and bolstering the overall adoption of ocean renewable energy sources. Impressively, by employing the HC-EGWO method, we achieved an upswing of up to 3.31% in power output compared to other methods. This substantial increment underscores the efficacy of our proposed optimization approach. Conclusively, the outcomes offer invaluable knowledge for deploying OSWECs in the South Caspian Sea, where unique environmental conditions intersect with considerable energy potential.
Robin: A Novel Method to Produce Robust Interpreters for Deep Learning-Based Code Classifiers
Venue: arXiv cs.SE 2023
Year: 2023

Deep learning has been widely used in source code classification tasks, such as code classification according to their functionalities, code authorship attribution, and vulnerability detection. Unfortunately, the black-box nature of deep learning makes it hard to interpret and understand why a classifier (i.e., classification model) makes a particular prediction on a given example. This lack of interpretability (or explainability) might have hindered their adoption by practitioners because it is not clear when they should or should not trust a classifier's prediction. The lack of interpretability has motivated a number of studies in recent years. However, existing methods are neither robust nor able to cope with out-of-distribution examples. In this paper, we propose a novel method to produce \underline{Rob}ust \underline{in}terpreters for a given deep learning-based code classifier; the method is dubbed Robin. The key idea behind Robin is a novel hybrid structure combining an interpreter and two approximators, while leveraging the ideas of adversarial training and data augmentation. Experimental results show that on average the interpreter produced by Robin achieves a 6.11\% higher fidelity (evaluated on the classifier), 67.22\% higher fidelity (evaluated on the approximator), and 15.87x higher robustness than that of the three existing interpreters we evaluated. Moreover, the interpreter is 47.31\% less affected by out-of-distribution examples than that of LEMNA.
Cross-modal and Cross-domain Knowledge Transfer for Label-free 3D Segmentation
Venue: arXiv cs.CV 2023
Year: 2023

Current state-of-the-art point cloud-based perception methods usually rely on large-scale labeled data, which requires expensive manual annotations. A natural option is to explore the unsupervised methodology for 3D perception tasks. However, such methods often face substantial performance-drop difficulties. Fortunately, we found that there exist amounts of image-based datasets and an alternative can be proposed, i.e., transferring the knowledge in the 2D images to 3D point clouds. Specifically, we propose a novel approach for the challenging cross-modal and cross-domain adaptation task by fully exploring the relationship between images and point clouds and designing effective feature alignment strategies. Without any 3D labels, our method achieves state-of-the-art performance for 3D point cloud semantic segmentation on SemanticKITTI by using the knowledge of KITTI360 and GTA5, compared to existing unsupervised and weakly-supervised baselines.