OpenRaiser/Intern-Atlas huggingface dataset:
SoK: Evaluating Privacy and Security Concerns of Using Web Services for the Disabled Population
Venue: arXiv cs.CR 2023
Year: 2023

The online privacy and security of the disabled community is a complex field that has implications for every user who navigates web services. While many disciplines have separately researched the disabled population and their online privacy and security concerns, the overlap between the two is very high but under-researched. Moreover, a complex relationship exists between the disabled population and web services where the interaction depends on several web service developmental factors, including usability and accessibility. To this aid, we explored this intersection of privacy and security of web services as perceived by the disabled community through previous studies by conducting a detailed systematic literature review and analysis of 63 articles. Our findings encompassed several topics, including how the disabled population navigates around authentication interfaces, online privacy concerns, universal design practices, and how security methods such as CAPTCHAs can be improved to become more accessible and usable for people of all needs and abilities. We further discuss the gap in the current research, including solutions such as the universal implementation of inclusive privacy and security tools and protocols.
PaRK-Detect: Towards Efficient Multi-Task Satellite Imagery Road Extraction via Patch-Wise Keypoints Detection
Venue: arXiv cs.CV 2023
Year: 2023

Automatically extracting roads from satellite imagery is a fundamental yet challenging computer vision task in the field of remote sensing. Pixel-wise semantic segmentation-based approaches and graph-based approaches are two prevailing schemes. However, prior works show the imperfections that semantic segmentation-based approaches yield road graphs with low connectivity, while graph-based methods with iterative exploring paradigms and smaller receptive fields focus more on local information and are also time-consuming. In this paper, we propose a new scheme for multi-task satellite imagery road extraction, Patch-wise Road Keypoints Detection (PaRK-Detect). Building on top of D-LinkNet architecture and adopting the structure of keypoint detection, our framework predicts the position of patch-wise road keypoints and the adjacent relationships between them to construct road graphs in a single pass. Meanwhile, the multi-task framework also performs pixel-wise semantic segmentation and generates road segmentation masks. We evaluate our approach against the existing state-of-the-art methods on DeepGlobe, Massachusetts Roads, and RoadTracer datasets and achieve competitive or better results. We also demonstrate a considerable outperformance in terms of inference speed.
Reclaiming Privacy and Performance over Centralized DNS
Venue: arXiv cs.CR 2023
Year: 2023

The Domain Name System (DNS) is both a key determinant of users' quality of experience (QoE) and privy to their tastes, preferences, and even the devices they own. Growing concern about user privacy and QoE has brought a number of alternative DNS services, from public DNS to encrypted and Oblivious DNS. While offering valuable features, these DNS variants are operated by a handful of providers, reinforcing a trend towards centralization that has raised concerns about privacy, competition, resilience and Web QoE. The goal of this work is to let users take advantage of third-party DNS services, without sacrificing privacy or performance. We follow Wheeler's advice, adding another level of indirection with an end-system DNS resolver, Onoma, that improves privacy, avoiding DNS-based user-reidentification by inserting and sharding requests across resolvers, and improves performance by running resolution races among resolvers and reinstating the client-resolver proximity assumption content delivery networks rely on. As our evaluation shows, while there may not be an ideal service for all clients in all places, Onoma dynamically finds the best service for any given location.
Learning cross space mapping via DNN using large scale click-through logs
Venue: arXiv cs.CV 2023
Year: 2023

The gap between low-level visual signals and high-level semantics has been progressively bridged by continuous development of deep neural network (DNN). With recent progress of DNN, almost all image classification tasks have achieved new records of accuracy. To extend the ability of DNN to image retrieval tasks, we proposed a unified DNN model for image-query similarity calculation by simultaneously modeling image and query in one network. The unified DNN is named the cross space mapping (CSM) model, which contains two parts, a convolutional part and a query-embedding part. The image and query are mapped to a common vector space via these two parts respectively, and image-query similarity is naturally defined as an inner product of their mappings in the space. To ensure good generalization ability of the DNN, we learn weights of the DNN from a large number of click-through logs which consists of 23 million clicked image-query pairs between 1 million images and 11.7 million queries. Both the qualitative results and quantitative results on an image retrieval evaluation task with 1000 queries demonstrate the superiority of the proposed method.
Understanding Social Media Cross-Modality Discourse in Linguistic Space
Venue: arXiv cs.MM 2023
Year: 2023

The multimedia communications with texts and images are popular on social media. However, limited studies concern how images are structured with texts to form coherent meanings in human cognition. To fill in the gap, we present a novel concept of cross-modality discourse, reflecting how human readers couple image and text understandings. Text descriptions are first derived from images (named as subtitles) in the multimedia contexts. Five labels -- entity-level insertion, projection and concretization and scene-level restatement and extension -- are further employed to shape the structure of subtitles and texts and present their joint meanings. As a pilot study, we also build the very first dataset containing 16K multimedia tweets with manually annotated discourse labels. The experimental results show that the multimedia encoder based on multi-head attention with captions is able to obtain the-state-of-the-art results.
NSANet: Noise Seeking Attention Network
Venue: arXiv cs.CV 2023
Year: 2023

LiDAR (Light Detection and Ranging) technology has remained popular in capturing natural and built environments for numerous applications. The recent technological advancements in electro-optical engineering have aided in obtaining laser returns at a higher pulse repetition frequency (PRF), which considerably increased the density of the 3D point cloud. Conventional techniques with lower PRF had a single pulse-in-air (SPIA) zone, large enough to avoid a mismatch among pulse pairs at the receiver. New multiple pulses-in-air (MPIA) technology guarantees various windows of operational ranges for a single flight line and no blind zones. The disadvantage of the technology is the projection of atmospheric returns closer to the same pulse-in-air zone of adjacent terrain points likely to intersect with objects of interest. These noise properties compromise the perceived quality of the scene and encourage the development of new noise-filtering neural networks, as existing filters are significantly ineffective. We propose a novel dual-attention noise-filtering neural network called Noise Seeking Attention Network (NSANet) that uses physical priors and local spatial attention to filter noise. Our research is motivated by two psychology theories of feature integration and attention engagement to prove the role of attention in computer vision at the encoding and decoding phase. The presented results of NSANet show the inclination towards attention engagement theory and a performance boost compared to the state-of-the-art noise-filtering deep convolutional neural networks.
Efficient physics-informed neural networks using hash encoding
Venue: arXiv cs.LG 2023
Year: 2023

Physics-informed neural networks (PINNs) have attracted a lot of attention in scientific computing as their functional representation of partial differential equation (PDE) solutions offers flexibility and accuracy features. However, their training cost has limited their practical use as a real alternative to classic numerical methods. Thus, we propose to incorporate multi-resolution hash encoding into PINNs to improve the training efficiency, as such encoding offers a locally-aware (at multi resolution) coordinate inputs to the neural network. Borrowed from the neural representation field community (NeRF), we investigate the robustness of calculating the derivatives of such hash encoded neural networks with respect to the input coordinates, which is often needed by the PINN loss terms. We propose to replace the automatic differentiation with finite-difference calculations of the derivatives to address the discontinuous nature of such derivatives. We also share the appropriate ranges for the hash encoding hyperparameters to obtain robust derivatives. We test the proposed method on three problems, including Burgers equation, Helmholtz equation, and Navier-Stokes equation. The proposed method admits about a 10-fold improvement in efficiency over the vanilla PINN implementation.
Tweets Under the Rubble: Detection of Messages Calling for Help in Earthquake Disaster
Venue: arXiv cs.SI 2023
Year: 2023

The importance of social media is again exposed in the recent tragedy of the 2023 Turkey and Syria earthquake. Many victims who were trapped under the rubble called for help by posting messages in Twitter. We present an interactive tool to provide situational awareness for missing and trapped people, and disaster relief for rescue and donation efforts. The system (i) collects tweets, (ii) classifies the ones calling for help, (iii) extracts important entity tags, and (iv) visualizes them in an interactive map screen. Our initial experiments show that the performance in terms of the F1 score is up to 98.30 for tweet classification, and 84.32 for entity extraction. The demonstration, dataset, and other related files can be accessed at https://github.com/avaapm/deprem
P4L: Privacy Preserving Peer-to-Peer Learning for Infrastructureless Setups
Venue: arXiv cs.LG 2023
Year: 2023

Distributed (or Federated) learning enables users to train machine learning models on their very own devices, while they share only the gradients of their models usually in a differentially private way (utility loss). Although such a strategy provides better privacy guarantees than the traditional centralized approach, it requires users to blindly trust a centralized infrastructure that may also become a bottleneck with the increasing number of users. In this paper, we design and implement P4L: a privacy preserving peer-to-peer learning system for users to participate in an asynchronous, collaborative learning scheme without requiring any sort of infrastructure or relying on differential privacy. Our design uses strong cryptographic primitives to preserve both the confidentiality and utility of the shared gradients, a set of peer-to-peer mechanisms for fault tolerance and user churn, proximity and cross device communications. Extensive simulations under different network settings and ML scenarios for three real-life datasets show that P4L provides competitive performance to baselines, while it is resilient to different poisoning attacks. We implement P4L and experimental results show that the performance overhead and power consumption is minimal (less than 3mAh of discharge).
Parameters of several families of binary duadic codes and their related codes
Venue: arXiv cs.IT 2023
Year: 2023

Binary duadic codes are an interesting subclass of cyclic codes since they have large dimensions and their minimum distances may have a square-root bound. In this paper, we present several families of binary duadic codes of length $2^m-1$ and develop some lower bounds on their minimum distances by using the BCH bound on cyclic codes, which partially solves one case of the open problem proposed in \cite{LLD}. It is shown that the lower bounds on their minimum distances are close to the square root bound. Moreover, the parameters of the dual and extended codes of these binary duadic codes are investigated.
FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning
Venue: arXiv cs.LG 2023
Year: 2023

Federated learning (FL) has emerged as a new paradigm for privacy-preserving computation in recent years. Unfortunately, FL faces two critical challenges that hinder its actual performance: data distribution heterogeneity and high resource costs brought by large foundation models. Specifically, the non-IID data in different clients make existing FL algorithms hard to converge while the high resource costs, including computational and communication costs that increase the deployment difficulty in real-world scenarios. In this paper, we propose an effective yet simple method, named FedCLIP, to achieve fast generalization and personalization for CLIP in federated learning. Concretely, we design an attention-based adapter for the large model, CLIP, and the rest operations merely depend on adapters. Lightweight adapters can make the most use of pretrained model information and ensure models be adaptive for clients in specific tasks. Simultaneously, small-scale operations can mitigate the computational burden and communication burden caused by large models. Extensive experiments are conducted on three datasets with distribution shifts. Qualitative and quantitative results demonstrate that FedCLIP significantly outperforms other baselines (9% overall improvements on PACS) and effectively reduces computational and communication costs (283x faster than FedAVG). Our code will be available at: https://github.com/microsoft/PersonalizedFL.
Strategize Before Teaching: A Conversational Tutoring System with Pedagogy Self-Distillation
Venue: arXiv cs.CL 2023
Year: 2023

Conversational tutoring systems (CTSs) aim to help students master educational material with natural language interaction in the form of a dialog. CTSs have become a key pillar in educational data mining research. A key challenge in CTSs is to engage the student in the conversation while exposing them to a diverse set of teaching strategies, akin to a human teacher, thereby, helping them learn in the process. Different from previous work that generates responses given the strategies as input, we propose to jointly predict teaching strategies and generate tutor responses accordingly, which fits a more realistic application scenario. We benchmark several competitive models on three dialog tutoring datasets and propose a unified framework that combines teaching response generation and pedagogical strategy prediction, where a self-distillation mechanism is adopted to guide the teaching strategy learning and facilitate tutor response generation. Our experiments and analyses shed light on how teaching strategies affect dialog tutoring.
CBA: Contextual Background Attack against Optical Aerial Detection in the Physical World
Venue: arXiv cs.CV 2023
Year: 2023

Patch-based physical attacks have increasingly aroused concerns. However, most existing methods focus on obscuring targets captured on the ground, and some of these methods are simply extended to deceive aerial detectors. They smear the targeted objects in the physical world with the elaborated adversarial patches, which can only slightly sway the aerial detectors' prediction and with weak attack transferability. To address the above issues, we propose to perform Contextual Background Attack (CBA), a novel physical attack framework against aerial detection, which can achieve strong attack efficacy and transferability in the physical world even without smudging the interested objects at all. Specifically, the targets of interest, i.e. the aircraft in aerial images, are adopted to mask adversarial patches. The pixels outside the mask area are optimized to make the generated adversarial patches closely cover the critical contextual background area for detection, which contributes to gifting adversarial patches with more robust and transferable attack potency in the real world. To further strengthen the attack performance, the adversarial patches are forced to be outside targets during training, by which the detected objects of interest, both on and outside patches, benefit the accumulation of attack efficacy. Consequently, the sophisticatedly designed patches are gifted with solid fooling efficacy against objects both on and outside the adversarial patches simultaneously. Extensive proportionally scaled experiments are performed in physical scenarios, demonstrating the superiority and potential of the proposed framework for physical attacks. We expect that the proposed physical attack method will serve as a benchmark for assessing the adversarial robustness of diverse aerial detectors and defense methods.
Aegis: Mitigating Targeted Bit-flip Attacks against Deep Neural Networks
Venue: arXiv cs.CR 2023
Year: 2023

Bit-flip attacks (BFAs) have attracted substantial attention recently, in which an adversary could tamper with a small number of model parameter bits to break the integrity of DNNs. To mitigate such threats, a batch of defense methods are proposed, focusing on the untargeted scenarios. Unfortunately, they either require extra trustworthy applications or make models more vulnerable to targeted BFAs. Countermeasures against targeted BFAs, stealthier and more purposeful by nature, are far from well established. In this work, we propose Aegis, a novel defense method to mitigate targeted BFAs. The core observation is that existing targeted attacks focus on flipping critical bits in certain important layers. Thus, we design a dynamic-exit mechanism to attach extra internal classifiers (ICs) to hidden layers. This mechanism enables input samples to early-exit from different layers, which effectively upsets the adversary's attack plans. Moreover, the dynamic-exit mechanism randomly selects ICs for predictions during each inference to significantly increase the attack cost for the adaptive attacks where all defense mechanisms are transparent to the adversary. We further propose a robustness training strategy to adapt ICs to the attack scenarios by simulating BFAs during the IC training phase, to increase model robustness. Extensive evaluations over four well-known datasets and two popular DNN structures reveal that Aegis could effectively mitigate different state-of-the-art targeted attacks, reducing attack success rate by 5-10$\times$, significantly outperforming existing defense methods.
BLiRF: Bandlimited Radiance Fields for Dynamic Scene Modeling
Venue: arXiv cs.CV 2023
Year: 2023

Reasoning the 3D structure of a non-rigid dynamic scene from a single moving camera is an under-constrained problem. Inspired by the remarkable progress of neural radiance fields (NeRFs) in photo-realistic novel view synthesis of static scenes, extensions have been proposed for dynamic settings. These methods heavily rely on neural priors in order to regularize the problem. In this work, we take a step back and reinvestigate how current implementations may entail deleterious effects, including limited expressiveness, entanglement of light and density fields, and sub-optimal motion localization. As a remedy, we advocate for a bridge between classic non-rigid-structure-from-motion (\nrsfm) and NeRF, enabling the well-studied priors of the former to constrain the latter. To this end, we propose a framework that factorizes time and space by formulating a scene as a composition of bandlimited, high-dimensional signals. We demonstrate compelling results across complex dynamic scenes that involve changes in lighting, texture and long-range dynamics.
Invariant Representations of Embedded Simplicial Complexes
Venue: arXiv cs.LG 2023
Year: 2023

Analyzing embedded simplicial complexes, such as triangular meshes and graphs, is an important problem in many fields. We propose a new approach for analyzing embedded simplicial complexes in a subdivision-invariant and isometry-invariant way using only topological and geometric information. Our approach is based on creating and analyzing sufficient statistics and uses a graph neural network. We demonstrate the effectiveness of our approach using a synthetic mesh data set.
Towards Audit Requirements for AI-based Systems in Mobility Applications
Venue: arXiv cs.LG 2023
Year: 2023

Various mobility applications like advanced driver assistance systems increasingly utilize artificial intelligence (AI) based functionalities. Typically, deep neural networks (DNNs) are used as these provide the best performance on the challenging perception, prediction or planning tasks that occur in real driving environments. However, current regulations like UNECE R 155 or ISO 26262 do not consider AI-related aspects and are only applied to traditional algorithm-based systems. The non-existence of AI-specific standards or norms prevents the practical application and can harm the trust level of users. Hence, it is important to extend existing standardization for security and safety to consider AI-specific challenges and requirements. To take a step towards a suitable regulation we propose 50 technical requirements or best practices that extend existing regulations and address the concrete needs for DNN-based systems. We show the applicability, usefulness and meaningfulness of the proposed requirements by performing an exemplary audit of a DNN-based traffic sign recognition system using three of the proposed requirements.
kNN-BOX: A Unified Framework for Nearest Neighbor Generation
Venue: arXiv cs.CL 2023
Year: 2023

Augmenting the base neural model with a token-level symbolic datastore is a novel generation paradigm and has achieved promising results in machine translation (MT). In this paper, we introduce a unified framework kNN-BOX, which enables quick development and interactive analysis for this novel paradigm. kNN-BOX decomposes the datastore-augmentation approach into three modules: datastore, retriever and combiner, thus putting diverse kNN generation methods into a unified way. Currently, kNN-BOX has provided implementation of seven popular kNN-MT variants, covering research from performance enhancement to efficiency optimization. It is easy for users to reproduce these existing works or customize their own models. Besides, users can interact with their kNN generation systems with kNN-BOX to better understand the underlying inference process in a visualized way. In the experiment section, we apply kNN-BOX for machine translation and three other seq2seq generation tasks, namely, text simplification, paraphrase generation and question generation. Experiment results show that augmenting the base neural model with kNN-BOX leads to a large performance improvement in all these tasks. The code and document of kNN-BOX is available at https://github.com/NJUNLP/knn-box.