OpenRaiser/Intern-Atlas huggingface dataset:
Semi-MAE: Masked Autoencoders for Semi-supervised Vision Transformers
Venue: arXiv cs.CV 2023
Year: 2023

Vision Transformer (ViT) suffers from data scarcity in semi-supervised learning (SSL). To alleviate this issue, inspired by masked autoencoder (MAE), which is a data-efficient self-supervised learner, we propose Semi-MAE, a pure ViT-based SSL framework consisting of a parallel MAE branch to assist the visual representation learning and make the pseudo labels more accurate. The MAE branch is designed as an asymmetric architecture consisting of a lightweight decoder and a shared-weights encoder. We feed the weakly-augmented unlabeled data with a high masking ratio to the MAE branch and reconstruct the missing pixels. Semi-MAE achieves 75.9% top-1 accuracy on ImageNet with 10% labels, surpassing prior state-of-the-art in semi-supervised image classification. In addition, extensive experiments demonstrate that Semi-MAE can be readily used for other ViT models and masked image modeling methods.
Freeform Islamic Geometric Patterns
Venue: arXiv cs.GR 2023
Year: 2023

Islamic geometric patterns are a rich and venerable ornamental tradition. Many classic designs feature periodic arrangements of rosettes: star shapes surrounded by rings of hexagonal petals. We present a new technique for generating 'freeform' compositions of rosettes: finite designs that freely mix rosettes of unusual sizes while retaining the aesthetics of traditional patterns. We use a circle packing as a scaffolding for developing a patch of polygons and fill each polygon with a motif based on established constructions from Islamic art.
Automated Misconfiguration Repair of Configurable Cyber-Physical Systems with Search: an Industrial Case Study on Elevator Dispatching Algorithms
Venue: arXiv cs.SE 2023
Year: 2023

Real-world Cyber-Physical Systems (CPSs) are usually configurable. Through parameters, it is possible to configure, select or unselect different system functionalities. While this provides high flexibility, it also becomes a source for failures due to misconfigurations. The large number of parameters these systems have and the long test execution time in this context due to the use of simulation-based testing make the manual repair process a cumbersome activity. Subsequently, in this context, automated repairing methods are paramount. In this paper, we propose an approach to automatically repair CPSs' misconfigurations. Our approach is evaluated with an industrial CPS case study from the elevation domain. Experiments with a real building and data obtained from operation suggests that our approach outperforms a baseline algorithm as well as the state of the practice (i.e., manual repair carried out by domain experts).
Analyzing I/O Performance of a Hierarchical HPC Storage System for Distributed Deep Learning
Venue: arXiv cs.DC 2023
Year: 2023

Today, deep learning is an essential technology for our life. To solve more complex problems with deep learning, both sizes of training datasets and neural networks are increasing. To train a model with large datasets and networks, distributed deep neural network (DDNN) training technique is necessary. For large-scale DDNN training, HPC clusters are a promising computation environment. In large-scale DDNN on HPC clusters, I/O performance is critical because it is becoming a bottleneck. Most flagship-class HPC clusters have hierarchical storage systems. For designing future HPC storage systems, it is necessary to quantify the performance improvement effect of the hierarchical storage system on the workloads. This paper demonstrates the quantitative performance analysis of the hierarchical storage system for DDNN workload in a flagship-class supercomputer. Our analysis shows how much performance improvement and volume increment of the storage will be required to meet the performance goal.
Federated Learning for Data Streams
Venue: arXiv cs.LG 2023
Year: 2023

Federated learning (FL) is an effective solution to train machine learning models on the increasing amount of data generated by IoT devices and smartphones while keeping such data localized. Most previous work on federated learning assumes that clients operate on static datasets collected before training starts. This approach may be inefficient because 1) it ignores new samples clients collect during training, and 2) it may require a potentially long preparatory phase for clients to collect enough data. Moreover, learning on static datasets may be simply impossible in scenarios with small aggregate storage across devices. It is, therefore, necessary to design federated algorithms able to learn from data streams. In this work, we formulate and study the problem of \emph{federated learning for data streams}. We propose a general FL algorithm to learn from data streams through an opportune weighted empirical risk minimization. Our theoretical analysis provides insights to configure such an algorithm, and we evaluate its performance on a wide range of machine learning tasks.
Matching Using Sufficient Dimension Reduction for Heterogeneity Causal Effect Estimation
Venue: arXiv cs.DS 2023
Year: 2023

Causal inference plays an important role in under standing the underlying mechanisation of the data generation process across various domains. It is challenging to estimate the average causal effect and individual causal effects from observational data with high-dimensional covariates due to the curse of dimension and the problem of data sufficiency. The existing matching methods can not effectively estimate individual causal effect or solve the problem of dimension curse in causal inference. To address this challenge, in this work, we prove that the reduced set by sufficient dimension reduction (SDR) is a balance score for confounding adjustment. Under the theorem, we propose to use an SDR method to obtain a reduced representation set of the original covariates and then the reduced set is used for the matching method. In detail, a non-parametric model is used to learn such a reduced set and to avoid model specification errors. The experimental results on real-world datasets show that the proposed method outperforms the compared matching methods. Moreover, we conduct an experiment analysis and the results demonstrate that the reduced representation is enough to balance the imbalance between the treatment group and control group individuals.
Learning Ambiguity from Crowd Sequential Annotations
Venue: arXiv cs.CL 2023
Year: 2023

Most crowdsourcing learning methods treat disagreement between annotators as noisy labelings while inter-disagreement among experts is often a good indicator for the ambiguity and uncertainty that is inherent in natural language. In this paper, we propose a framework called Learning Ambiguity from Crowd Sequential Annotations (LA-SCA) to explore the inter-disagreement between reliable annotators and effectively preserve confusing label information. First, a hierarchical Bayesian model is developed to infer ground-truth from crowds and group the annotators with similar reliability together. By modeling the relationship between the size of group the annotator involved in, the annotator's reliability and element's unambiguity in each sequence, inter-disagreement between reliable annotators on ambiguous elements is computed to obtain label confusing information that is incorporated to cost-sensitive sequence labeling. Experimental results on POS tagging and NER tasks show that our proposed framework achieves competitive performance in inferring ground-truth from crowds and predicting unknown sequences, and interpreting hierarchical clustering results helps discover labeling patterns of annotators with similar reliability.
Rényi-Ulam Games and Online Computation with Imperfect Advice
Venue: arXiv cs.DS 2023
Year: 2023

We study the nascent setting of online computation with imperfect advice, in which the online algorithm is enhanced by some prediction encoded in the form of a possibly erroneous binary string. The algorithm is oblivious to the advice error, but defines a desired tolerance, namely an upper bound on the number of erroneous advice bits it can tolerate. This is a model that generalizes the untrusted advice model [Angelopoulos et al. ITCS 2020], in which the performance of the algorithm is only evaluated at the extreme values of error (namely, if the advice has either no errors, or if it is generated adversarially). In this work, we establish connections between games with a lying responder, also known as Rényi-Ulam games, and the design and analysis of online algorithms with imperfect advice. Specifically, we demonstrate how to obtain upper and lower bounds on the competitive ratio for well-studied online problems such as time-series search, online bidding, and fractional knapsack. Our techniques provide the first lower bounds for online problems in this model. We also highlight and exploit connections between competitive analysis with imperfect advice and fault-tolerance in multiprocessor systems. Last, we show how to waive the dependence on the tolerance parameter, by means of resource augmentation and robustification.
Text sampling strategies for predicting missing bibliographic links
Venue: arXiv cs.LG 2023
Year: 2023

The paper proposes various strategies for sampling text data when performing automatic sentence classification for the purpose of detecting missing bibliographic links. We construct samples based on sentences as semantic units of the text and add their immediate context which consists of several neighboring sentences. We examine a number of sampling strategies that differ in context size and position. The experiment is carried out on the collection of STEM scientific papers. Including the context of sentences into samples improves the result of their classification. We automatically determine the optimal sampling strategy for a given text collection by implementing an ensemble voting when classifying the same data sampled in different ways. Sampling strategy taking into account the sentence context with hard voting procedure leads to the classification accuracy of 98% (F1-score). This method of detecting missing bibliographic links can be used in recommendation engines of applied intelligent information systems.
Augmenting data-driven models for energy systems through feature engineering: A Python framework for feature engineering
Venue: arXiv cs.LG 2023
Year: 2023

Data-driven modeling is an approach in energy systems modeling that has been gaining popularity. In data-driven modeling, machine learning methods such as linear regression, neural networks or decision-tree based methods are being applied. While these methods do not require domain knowledge, they are sensitive to data quality. Therefore, improving data quality in a dataset is beneficial for creating machine learning-based models. The improvement of data quality can be implemented through preprocessing methods. A selected type of preprocessing is feature engineering, which focuses on evaluating and improving the quality of certain features inside the dataset. Feature engineering methods include methods such as feature creation, feature expansion, or feature selection. In this work, a Python framework containing different feature engineering methods is presented. This framework contains different methods for feature creation, expansion and selection; in addition, methods for transforming or filtering data are implemented. The implementation of the framework is based on the Python library scikit-learn. The framework is demonstrated on a case study of a use case from energy demand prediction. A data-driven model is created including selected feature engineering methods. The results show an improvement in prediction accuracy through the engineered features.
Episodes Discovery Recommendation with Multi-Source Augmentations
Venue: arXiv cs.IR 2023
Year: 2023

Recommender systems (RS) commonly retrieve potential candidate items for users from a massive number of items by modeling user interests based on historical interactions. However, historical interaction data is highly sparse, and most items are long-tail items, which limits the representation learning for item discovery. This problem is further augmented by the discovery of novel or cold-start items. For example, after a user displays interest in bitcoin financial investment shows in the podcast space, a recommender system may want to suggest, e.g., a newly released blockchain episode from a more technical show. Episode correlations help the discovery, especially when interaction data of episodes is limited. Accordingly, we build upon the classical Two-Tower model and introduce the novel Multi-Source Augmentations using a Contrastive Learning framework (MSACL) to enhance episode embedding learning by incorporating positive episodes from numerous correlated semantics. Extensive experiments on a real-world podcast recommendation dataset from a large audio streaming platform demonstrate the effectiveness of the proposed framework for user podcast exploration and cold-start episode recommendation.
Tight Distortion Bounds for Distributed Single-Winner Metric Voting on a Line
Venue: arXiv cs.GT 2023
Year: 2023

We consider the distributed single-winner metric voting problem on a line, where agents and alternative are represented by points on the line of real numbers, the agents are partitioned into disjoint districts, and the goal is to choose a single winning alternative in a decentralized manner. In particular, the choice is done by a distributed voting mechanism which first selects a representative alternative for each district of agents and then chooses one of these representatives as the winner. In this paper, we design simple distributed mechanisms that achieve distortion at most $2+\sqrt{5}$ for the average-of-max and the max-of-average social cost objectives, matching the corresponding lower bound shown in previous work for these objectives.
Dynamic Maintenance of Monotone Dynamic Programs and Applications
Venue: arXiv cs.DS 2023
Year: 2023

Dynamic programming (DP) is one of the fundamental paradigms in algorithm design. However, many DP algorithms have to fill in large DP tables, represented by two-dimensional arrays, which causes at least quadratic running times and space usages. This has led to the development of improved algorithms for special cases when the DPs satisfy additional properties like, e.g., the Monge property or total monotonicity. In this paper, we consider a new condition which assumes (among some other technical assumptions) that the rows of the DP table are monotone. Under this assumption, we introduce a novel data structure for computing $(1+\varepsilon)$-approximate DP solutions in near-linear time and space in the static setting, and with polylogarithmic update times when the DP entries change dynamically. To the best of our knowledge, our new condition is incomparable to previous conditions and is the first which allows to derive dynamic algorithms based on existing DPs. Instead of using two-dimensional arrays to store the DP tables, we store the rows of the DP tables using monotone piecewise constant functions. This allows us to store length-$n$ DP table rows with entries in $[0,W]$ using only polylog$(n,W)$ bits, and to perform operations, such as $(\min,+)$-convolution or rounding, on these functions in polylogarithmic time. We further present several applications of our data structure. For bicriteria versions of $k$-balanced graph partitioning and simultaneous source location, we obtain the first dynamic algorithms with subpolynomial update times, as well as the first static algorithms using only near-linear time and space. Additionally, we obtain the currently fastest algorithm for fully dynamic knapsack.
An Ensemble Mobile-Cloud Computing Method for Affordable and Accurate Glucometer Readout
Venue: arXiv cs.DC 2023
Year: 2023

Despite essential efforts towards advanced wireless medical devices for regular monitoring of blood properties, many such devices are not available or not affordable for everyone in many countries. Alternatively using ordinary devices, patients ought to log data into a mobile health-monitoring manually. It causes several issues: (1) clients reportedly tend to enter unrealistic data; (2) typing values several times a day is bothersome and causes clients to leave the mobile app. Thus, there is a strong need to use now-ubiquitous smartphones, reducing error by capturing images from the screen of medical devices and extracting useful information automatically. Nevertheless, there are a few challenges in its development: (1) data scarcity has led to impractical methods with very low accuracy: to our knowledge, only small datasets are available in this case; (2) accuracy-availability tradeoff: one can execute a less accurate algorithm on a mobile phone to maintain higher availability, or alternatively deploy a more accurate and more compute-intensive algorithm on the cloud, however, at the cost of lower availability in poor/no connectivity situations. We present an ensemble learning algorithm, a mobile-cloud computing service architecture, and a simple compression technique to achieve higher availability and faster response time while providing higher accuracy by integrating cloud- and mobile-side predictions. Additionally, we propose an algorithm to generate synthetic training data which facilitates utilizing deep learning models to improve accuracy. Our proposed method achieves three main objectives: (1) 92.1% and 97.7% accuracy on two different datasets, improving previous methods by 40%, (2) reducing required bandwidth by 45x with 1% drop in accuracy, (3) and providing better availability compared to mobile-only, cloud-only, split computing, and early exit service models.
Evaluation of Induced Expert Knowledge in Causal Structure Learning by NOTEARS
Venue: arXiv cs.LG 2023
Year: 2023

Causal modeling provides us with powerful counterfactual reasoning and interventional mechanism to generate predictions and reason under various what-if scenarios. However, causal discovery using observation data remains a nontrivial task due to unobserved confounding factors, finite sampling, and changes in the data distribution. These can lead to spurious cause-effect relationships. To mitigate these challenges in practice, researchers augment causal learning with known causal relations. The goal of the paper is to study the impact of expert knowledge on causal relations in the form of additional constraints used in the formulation of the nonparametric NOTEARS. We provide a comprehensive set of comparative analyses of biasing the model using different types of knowledge. We found that (i) knowledge that corrects the mistakes of the NOTEARS model can lead to statistically significant improvements, (ii) constraints on active edges have a larger positive impact on causal discovery than inactive edges, and surprisingly, (iii) the induced knowledge does not correct on average more incorrect active and/or inactive edges than expected. We also demonstrate the behavior of the model and the effectiveness of domain knowledge on a real-world dataset.
Availability Adversarial Attack and Countermeasures for Deep Learning-based Load Forecasting
Venue: arXiv cs.LG 2023
Year: 2023

The forecast of electrical loads is essential for the planning and operation of the power system. Recently, advances in deep learning have enabled more accurate forecasts. However, deep neural networks are prone to adversarial attacks. Although most of the literature focuses on integrity-based attacks, this paper proposes availability-based adversarial attacks, which can be more easily implemented by attackers. For each forecast instance, the availability attack position is optimally solved by mixed-integer reformulation of the artificial neural network. To tackle this attack, an adversarial training algorithm is proposed. In simulation, a realistic load forecasting dataset is considered and the attack performance is compared to the integrity-based attack. Meanwhile, the adversarial training algorithm is shown to significantly improve robustness against availability attacks. All codes are available at https://github.com/xuwkk/AAA_Load_Forecast.
PMP: Privacy-Aware Matrix Profile against Sensitive Pattern Inference for Time Series
Venue: arXiv cs.LG 2023
Year: 2023

Recent rapid development of sensor technology has allowed massive fine-grained time series (TS) data to be collected and set the foundation for the development of data-driven services and applications. During the process, data sharing is often involved to allow the third-party modelers to perform specific time series data mining (TSDM) tasks based on the need of data owner. The high resolution of TS brings new challenges in protecting privacy. While meaningful information in high-resolution TS shifts from concrete point values to local shape-based segments, numerous research have found that long shape-based patterns could contain more sensitive information and may potentially be extracted and misused by a malicious third party. However, the privacy issue for TS patterns is surprisingly seldom explored in privacy-preserving literature. In this work, we consider a new privacy-preserving problem: preventing malicious inference on long shape-based patterns while preserving short segment information for the utility task performance. To mitigate the challenge, we investigate an alternative approach by sharing Matrix Profile (MP), which is a non-linear transformation of original data and a versatile data structure that supports many data mining tasks. We found that while MP can prevent concrete shape leakage, the canonical correlation in MP index can still reveal the location of sensitive long pattern. Based on this observation, we design two attacks named Location Attack and Entropy Attack to extract the pattern location from MP. To further protect MP from these two attacks, we propose a Privacy-Aware Matrix Profile (PMP) via perturbing the local correlation and breaking the canonical correlation in MP index vector. We evaluate our proposed PMP against baseline noise-adding methods through quantitative analysis and real-world case studies to show the effectiveness of the proposed method.
An Automatic Method for Generating Symbolic Expressions of Zernike Circular Polynomials
Venue: arXiv cs.SC 2023
Year: 2023

Zernike circular polynomials (ZCP) play a significant role in optics engineering. The symbolic expressions for ZCP are valuable for theoretic analysis and engineering designs. However, there are still two problems which remain open: firstly, there is a lack of sufficient mathematical formulas of the ZCP for optics designers; secondly the formulas for inter-conversion of Noll's single index and Born-Wolf's double indices of ZCP are neither uniquely determinate nor satisfactory. An automatic method for generating symbolic expressions for ZCP is proposed based on five essential factors: the new theorems for converting the single/double indices of the ZCP, the robust and effective numeric algorithms for computing key parameters of ZCP, the symbolic algorithms for generating mathematical expressions of ZCP, and meta-programming \& \LaTeX{} programming for generating the table of ZCP. The theorems, method, algorithms and system architecture proposed are beneficial to both optics design process, optics software, computer-output typesetting in publishing industry as well as STEM education.