OpenRaiser/Intern-Atlas huggingface dataset:
LOKI: Large-scale Data Reconstruction Attack against Federated Learning through Model Manipulation
Venue: arXiv cs.LG 2023
Year: 2023

Federated learning was introduced to enable machine learning over large decentralized datasets while promising privacy by eliminating the need for data sharing. Despite this, prior work has shown that shared gradients often contain private information and attackers can gain knowledge either through malicious modification of the architecture and parameters or by using optimization to approximate user data from the shared gradients. However, prior data reconstruction attacks have been limited in setting and scale, as most works target FedSGD and limit the attack to single-client gradients. Many of these attacks fail in the more practical setting of FedAVG or if updates are aggregated together using secure aggregation. Data reconstruction becomes significantly more difficult, resulting in limited attack scale and/or decreased reconstruction quality. When both FedAVG and secure aggregation are used, there is no current method that is able to attack multiple clients concurrently in a federated learning setting. In this work we introduce LOKI, an attack that overcomes previous limitations and also breaks the anonymity of aggregation as the leaked data is identifiable and directly tied back to the clients they come from. Our design sends clients customized convolutional parameters, and the weight gradients of data points between clients remain separate even through aggregation. With FedAVG and aggregation across 100 clients, prior work can leak less than 1% of images on MNIST, CIFAR-100, and Tiny ImageNet. Using only a single training round, LOKI is able to leak 76-86% of all data samples.
Pre-NeRF 360: Enriching Unbounded Appearances for Neural Radiance Fields
Venue: arXiv cs.CV 2023
Year: 2023

Neural radiance fields (NeRF) appeared recently as a powerful tool to generate realistic views of objects and confined areas. Still, they face serious challenges with open scenes, where the camera has unrestricted movement and content can appear at any distance. In such scenarios, current NeRF-inspired models frequently yield hazy or pixelated outputs, suffer slow training times, and might display irregularities, because of the challenging task of reconstructing an extensive scene from a limited number of images. We propose a new framework to boost the performance of NeRF-based architectures yielding significantly superior outcomes compared to the prior work. Our solution overcomes several obstacles that plagued earlier versions of NeRF, including handling multiple video inputs, selecting keyframes, and extracting poses from real-world frames that are ambiguous and symmetrical. Furthermore, we applied our framework, dubbed as "Pre-NeRF 360", to enable the use of the Nutrition5k dataset in NeRF and introduce an updated version of this dataset, known as the N5k360 dataset.
DG-Trans: Dual-level Graph Transformer for Spatiotemporal Incident Impact Prediction on Traffic Networks
Venue: arXiv cs.LG 2023
Year: 2023

The prompt estimation of traffic incident impacts can guide commuters in their trip planning and improve the resilience of transportation agencies' decision-making on resilience. However, it is more challenging than node-level and graph-level forecasting tasks, as it requires extracting the anomaly subgraph or sub-time-series from dynamic graphs. In this paper, we propose DG-Trans, a novel traffic incident impact prediction framework, to foresee the impact of traffic incidents through dynamic graph learning. The proposed framework contains a dual-level spatial transformer and an importance-score-based temporal transformer, and the performance of this framework is justified by two newly constructed benchmark datasets. The dual-level spatial transformer removes unnecessary edges between nodes to isolate the affected subgraph from the other nodes. Meanwhile, the importance-score-based temporal transformer identifies abnormal changes in node features, causing the predictions to rely more on measurement changes after the incident occurs. Therefore, DG-Trans is equipped with dual abilities that extract spatiotemporal dependency and identify anomaly nodes affected by incidents while removing noise introduced by benign nodes. Extensive experiments on real-world datasets verify that DG-Trans outperforms the existing state-of-the-art methods, especially in extracting spatiotemporal dependency patterns and predicting traffic accident impacts. It offers promising potential for traffic incident management systems.
Designing the Metaverse: A Scoping Review to Map Current Research Effort on Ethical Implications
Venue: arXiv cs.CY 2023
Year: 2023

The metaverse and digital, virtual environments have been part of recent history as places in which people can socialize, work and spend time playing games. However, the infancy of the development of these digital, virtual environments brings some challenges that are still not fully depicted. With this article, we seek to identify and map the currently available knowledge and scientific effort to discover what principles, guidelines, laws, policies, and practices are currently in place to allow for the design of digital, virtual environments, and the metaverse. Through a scoping review, we aimed to systematically survey the existing literature and discern gaps in knowledge within the domain of metaverse research from sociological, anthropological, cultural, and experiential perspectives. The objective of this review was twofold: (1) to examine the focus of the literature studying the metaverse from various angles and (2) to formulate a research agenda for the design and development of ethical digital, virtual environments. With this paper, we identified several works and articles detailing experiments and research on the design of digital, virtual environments and metaverses. We found an increased number of publications in the year 2022. This finding, together with the fact that only a few articles were focused on the domain of ethics, culture and society shows that there is still a vast amount of work to be done to create awareness, principles and policies that could help to design safe, secure and inclusive digital, virtual environments and metaverses.
Exploring Turkish Speech Recognition via Hybrid CTC/Attention Architecture and Multi-feature Fusion Network
Venue: arXiv cs.SD 2023
Year: 2023

In recent years, End-to-End speech recognition technology based on deep learning has developed rapidly. Due to the lack of Turkish speech data, the performance of Turkish speech recognition system is poor. Firstly, this paper studies a series of speech recognition tuning technologies. The results show that the performance of the model is the best when the data enhancement technology combining speed perturbation with noise addition is adopted and the beam search width is set to 16. Secondly, to maximize the use of effective feature information and improve the accuracy of feature extraction, this paper proposes a new feature extractor LSPC. LSPC and LiGRU network are combined to form a shared encoder structure, and model compression is realized. The results show that the performance of LSPC is better than MSPC and VGGnet when only using Fbank features, and the WER is improved by 1.01% and 2.53% respectively. Finally, based on the above two points, a new multi-feature fusion network is proposed as the main structure of the encoder. The results show that the WER of the proposed feature fusion network based on LSPC is improved by 0.82% and 1.94% again compared with the single feature (Fbank feature and Spectrogram feature) extraction using LSPC. Our model achieves performance comparable to that of advanced End-to-End models.
Anomaly Detection in Aeronautics Data with Quantum-compatible Discrete Deep Generative Model
Venue: arXiv cs.LG 2023
Year: 2023

Deep generative learning cannot only be used for generating new data with statistical characteristics derived from input data but also for anomaly detection, by separating nominal and anomalous instances based on their reconstruction quality. In this paper, we explore the performance of three unsupervised deep generative models -- variational autoencoders (VAEs) with Gaussian, Bernoulli, and Boltzmann priors -- in detecting anomalies in flight-operations data of commercial flights consisting of multivariate time series. We devised two VAE models with discrete latent variables (DVAEs), one with a factorized Bernoulli prior and one with a restricted Boltzmann machine (RBM) as prior, because of the demand for discrete-variable models in machine-learning applications and because the integration of quantum devices based on two-level quantum systems requires such models. The DVAE with RBM prior, using a relatively simple -- and classically or quantum-mechanically enhanceable -- sampling technique for the evolution of the RBM's negative phase, performed better than the Bernoulli DVAE and on par with the Gaussian model, which has a continuous latent space. Our studies demonstrate the competitiveness of a discrete deep generative model with its Gaussian counterpart on anomaly-detection tasks. Moreover, the DVAE model with RBM prior can be easily integrated with quantum sampling by outsourcing its generative process to measurements of quantum states obtained from a quantum annealer or gate-model device.
Automatically Predict Material Properties with Microscopic Image Example Polymer Compatibility
Venue: arXiv cs.CV 2023
Year: 2023

Many material properties are manifested in the morphological appearance and characterized with microscopic image, such as scanning electron microscopy (SEM). Polymer miscibility is a key physical quantity of polymer material and commonly and intuitively judged by SEM images. However, human observation and judgement for the images is time-consuming, labor-intensive and hard to be quantified. Computer image recognition with machine learning method can make up the defects of artificial judging, giving accurate and quantitative judgement. We achieve automatic miscibility recognition utilizing convolution neural network and transfer learning method, and the model obtains up to 94% accuracy. We also put forward a quantitative criterion for polymer miscibility with this model. The proposed method can be widely applied to the quantitative characterization of the microstructure and properties of various materials.
Disturbance Injection under Partial Automation: Robust Imitation Learning for Long-horizon Tasks
Venue: arXiv cs.RO 2023
Year: 2023

Partial Automation (PA) with intelligent support systems has been introduced in industrial machinery and advanced automobiles to reduce the burden of long hours of human operation. Under PA, operators perform manual operations (providing actions) and operations that switch to automatic/manual mode (mode-switching). Since PA reduces the total duration of manual operation, these two action and mode-switching operations can be replicated by imitation learning with high sample efficiency. To this end, this paper proposes Disturbance Injection under Partial Automation (DIPA) as a novel imitation learning framework. In DIPA, mode and actions (in the manual mode) are assumed to be observables in each state and are used to learn both action and mode-switching policies. The above learning is robustified by injecting disturbances into the operator's actions to optimize the disturbance's level for minimizing the covariate shift under PA. We experimentally validated the effectiveness of our method for long-horizon tasks in two simulations and a real robot environment and confirmed that our method outperformed the previous methods and reduced the demonstration burden.
Graph Data Models and Relational Database Technology
Venue: arXiv cs.DB 2023
Year: 2023

Recent work on database application development platforms has sought to include a declarative formulation of a conceptual data model in the application code, using annotations or attributes. Some recent work has used metadata to include the details of such formulations in the physical database, and this approach brings significant advantages in that the model can be enforced across a range of applications for a single database. In previous work, we have discussed the advantages for enterprise integration of typed graph data models (TGM), which can play a similar role in graphical databases, leveraging the existing support for the unified modelling language UML. Ideally, the integration of systems designed with different models, for example, graphical and relational database, should also be supported. In this work, we implement this approach, using metadata in a relational database management system (DBMS).
VMCML: Video and Music Matching via Cross-Modality Lifting
Venue: arXiv cs.CV 2023
Year: 2023

We propose a content-based system for matching video and background music. The system aims to address the challenges in music recommendation for new users or new music give short-form videos. To this end, we propose a cross-modal framework VMCML that finds a shared embedding space between video and music representations. To ensure the embedding space can be effectively shared by both representations, we leverage CosFace loss based on margin-based cosine similarity loss. Furthermore, we establish a large-scale dataset called MSVD, in which we provide 390 individual music and the corresponding matched 150,000 videos. We conduct extensive experiments on Youtube-8M and our MSVD datasets. Our quantitative and qualitative results demonstrate the effectiveness of our proposed framework and achieve state-of-the-art video and music matching performance.
Designing Theory-Driven Analytics-Enhanced Self-Regulated Learning Applications
Venue: arXiv cs.CY 2023
Year: 2023

There is an increased interest in the application of learning analytics (LA) to promote self-regulated learning (SRL). A variety of LA dashboards and indicators were proposed to support different crucial SRL processes, such as planning, awareness, self-reflection, self-monitoring, and feedback. However, the design of these dashboards and indicators is often without reference to theories in learning science, human-computer interaction (HCI), and information visualization (InfoVis). Moreover, there is a lack of theoretically sound frameworks to guide the systematic design and development of LA dashboards and indicators to scaffold SRL. This chapter seeks to explore theoretical underpinnings of the design of LA-enhanced SRL applications, drawing from the fields of learning science, HCI, and InfoVis. We first present the Student-Centered Learning Analytics-enhanced Self-Regulated Learning (SCLA-SRL) methodology for building theory-driven LA-enhanced SRL applications for and with learners. We then put this methodology into practice by designing and developing LA indicators to support novice programmers' SRL in a higher education context.
Region-wise matching for image inpainting based on adaptive weighted low-rank decomposition
Venue: arXiv cs.CV 2023
Year: 2023

Digital image inpainting is an interpolation problem, inferring the content in the missing (unknown) region to agree with the known region data such that the interpolated result fulfills some prior knowledge. Low-rank and nonlocal self-similarity are two important priors for image inpainting. Based on the nonlocal self-similarity assumption, an image is divided into overlapped square target patches (submatrices) and the similar patches of any target patch are reshaped as vectors and stacked into a patch matrix. Such a patch matrix usually enjoys a property of low rank or approximately low rank, and its missing entries are recoveried by low-rank matrix approximation (LRMA) algorithms. Traditionally, $n$ nearest neighbor similar patches are searched within a local window centered at a target patch. However, for an image with missing lines, the generated patch matrix is prone to having entirely-missing rows such that the downstream low-rank model fails to reconstruct it well. To address this problem, we propose a region-wise matching (RwM) algorithm by dividing the neighborhood of a target patch into multiple subregions and then search the most similar one within each subregion. A non-convex weighted low-rank decomposition (NC-WLRD) model for LRMA is also proposed to reconstruct all degraded patch matrices grouped by the proposed RwM algorithm. We solve the proposed NC-WLRD model by the alternating direction method of multipliers (ADMM) and analyze the convergence in detail. Numerous experiments on line inpainting (entire-row/column missing) demonstrate the superiority of our method over other competitive inpainting algorithms. Unlike other low-rank-based matrix completion methods and inpainting algorithms, the proposed model NC-WLRD is also effective for removing random-valued impulse noise and structural noise (stripes).
Doppler-Division Multiplexing for MIMO OFDM Joint Sensing and Communications
Venue: arXiv cs.IT 2023
Year: 2023

A promising waveform candidate for future joint sensing and communication systems is orthogonal frequencydivision multiplexing (OFDM). For such systems, supporting multiple transmit antennas requires multiplexing methods for the generation of orthogonal transmit signals, where equidistant subcarrier interleaving (ESI) is the most popular multiplexing method. In this work, we analyze a multiplexing method called Doppler-division multiplexing (DDM). This method applies a phase shift from OFDM symbol to OFDM symbol to separate signals transmitted by different Tx antennas along the velocity axis of the range-Doppler map. While general properties of DDM for the task of radar sensing are analyzed in this work, the main focus lies on the implications of DDM on the communication task. It will be shown that for DDM, the channels observed in the communication receiver are heavily timevarying, preventing any meaningful transmission of data when not taken into account. In this work, a communication system designed to combat these time-varying channels is proposed, which includes methods for data estimation, synchronization, and channel estimation. Bit error ratio (BER) simulations demonstrate the superiority of this communications system compared to a system utilizing ESI.
On Fair Allocation of Indivisible Goods to Submodular Agents
Venue: arXiv cs.GT 2023
Year: 2023

We consider the problem of fair allocation of indivisible goods to agents with submodular valuation functions, where agents may have either equal entitlements or arbitrary (possibly unequal) entitlements. We focus on share-based fairness notions, specifically, the maximin share (MMS) for equal entitlements and the anyprice share (APS) for arbitrary entitlements, and design allocation algorithms that give each agent a bundle of value at least some constant fraction of her share value. For the equal entitlement case (and submodular valuations), Ghodsi, Hajiaghayi, Seddighin, Seddighin, and Yami [EC 2018] designed a polynomial-time algorithm for $\frac{1}{3}$-maximin-fair allocation. We improve this result in two different ways. We consider the general case of arbitrary entitlements, and present a polynomial time algorithm that guarantees submodular agents $\frac{1}{3}$ of their APS. For the equal entitlement case, we improve the approximation ratio and obtain $\frac{10}{27}$-maximin-fair allocations. Our algorithms are based on designing strategies for a certain bidding game that was previously introduced by Babaioff, Ezra and Feige [EC 2021].
DevelSet: Deep Neural Level Set for Instant Mask Optimization
Venue: arXiv cs.CV 2023
Year: 2023

With the feature size continuously shrinking in advanced technology nodes, mask optimization is increasingly crucial in the conventional design flow, accompanied by an explosive growth in prohibitive computational overhead in optical proximity correction (OPC) methods. Recently, inverse lithography technique (ILT) has drawn significant attention and is becoming prevalent in emerging OPC solutions. However, ILT methods are either time-consuming or in weak performance of mask printability and manufacturability. In this paper, we present DevelSet, a GPU and deep neural network (DNN) accelerated level set OPC framework for metal layer. We first improve the conventional level set-based ILT algorithm by introducing the curvature term to reduce mask complexity and applying GPU acceleration to overcome computational bottlenecks. To further enhance printability and fast iterative convergence, we propose a novel deep neural network delicately designed with level set intrinsic principles to facilitate the joint optimization of DNN and GPU accelerated level set optimizer. Experimental results show that DevelSet framework surpasses the state-of-the-art methods in printability and boost the runtime performance achieving instant level (around 1 second).
BlockChain and Decentralized Apps
Venue: arXiv cs.CR 2023
Year: 2023

Blockchain, the backbone of Bitcoin, has recently gained a lot of attention. Blockchain functions as an immutable record that enables decentralized transactions. Blockchain-based applications are sprouting up in a variety of industries, including financial services, reputation systems, and the Internet of Things (IoT), among others. However, many hurdles of blockchain technology, including scalability and security issues, have to be overcome. Many industries, including finance, medicine, manufacturing, and education, use blockchain applications to capitalize on this technology's unique set of properties. Blockchain technology (BT) has the potential to improve trustworthiness, collaboration, organization, identity, credibility, and transparency. We provide an overview of blockchain architecture, various different kinds of blockchain as well as information about the Decentralized apps which are also known as Dapps. This paper provides an in-depth look at blockchain technology
Do Backdoors Assist Membership Inference Attacks?
Venue: arXiv cs.CR 2023
Year: 2023

When an adversary provides poison samples to a machine learning model, privacy leakage, such as membership inference attacks that infer whether a sample was included in the training of the model, becomes effective by moving the sample to an outlier. However, the attacks can be detected because inference accuracy deteriorates due to poison samples. In this paper, we discuss a \textit{backdoor-assisted membership inference attack}, a novel membership inference attack based on backdoors that return the adversary's expected output for a triggered sample. We found three crucial insights through experiments with an academic benchmark dataset. We first demonstrate that the backdoor-assisted membership inference attack is unsuccessful. Second, when we analyzed loss distributions to understand the reason for the unsuccessful results, we found that backdoors cannot separate loss distributions of training and non-training samples. In other words, backdoors cannot affect the distribution of clean samples. Third, we also show that poison and triggered samples activate neurons of different distributions. Specifically, backdoors make any clean sample an inlier, contrary to poisoning samples. As a result, we confirm that backdoors cannot assist membership inference.