OpenRaiser/Intern-Atlas huggingface dataset:
Make Pixels Dance: High-Dynamic Video Generation
Venue: arXiv cs.CV 2023
Year: 2023

Creating high-dynamic videos such as motion-rich actions and sophisticated visual effects poses a significant challenge in the field of artificial intelligence. Unfortunately, current state-of-the-art video generation methods, primarily focusing on text-to-video generation, tend to produce video clips with minimal motions despite maintaining high fidelity. We argue that relying solely on text instructions is insufficient and suboptimal for video generation. In this paper, we introduce PixelDance, a novel approach based on diffusion models that incorporates image instructions for both the first and last frames in conjunction with text instructions for video generation. Comprehensive experimental results demonstrate that PixelDance trained with public data exhibits significantly better proficiency in synthesizing videos with complex scenes and intricate motions, setting a new standard for video generation.
"Centralized or Decentralized?": Concerns and Value Judgments of Stakeholders in the Non-Fungible Tokens (NFTs) Market
Venue: arXiv cs.CY 2023
Year: 2023

Non-fungible tokens (NFTs) are decentralized digital tokens to represent the unique ownership of items. Recently, NFTs have been gaining popularity and at the same time bringing up issues, such as scams, racism, and sexism. Decentralization, a key attribute of NFT, contributes to some of the issues that are easier to regulate under centralized schemes, which are intentionally left out of the NFT marketplace. In this work, we delved into this centralization-decentralization dilemma in the NFT space through mixed quantitative and qualitative methods. Centralization-decentralization dilemma is the dilemma caused by the conflict between the slogan of decentralization and the interests of stakeholders. We first analyzed over 30,000 NFT-related tweets to obtain a high-level understanding of stakeholders' concerns in the NFT space. We then interviewed 15 NFT stakeholders (both creators and collectors) to obtain their in-depth insights into these concerns and potential solutions. Our findings identify concerning issues among users: financial scams, counterfeit NFTs, hacking, and unethical NFTs. We further reflected on the centralization-decentralization dilemma drawing upon the perspectives of the stakeholders in the interviews. Finally, we gave some inferences to solve the centralization-decentralization dilemma in the NFT market and thought about the future of NFT and decentralization.
Bit Cipher -- A Simple yet Powerful Word Representation System that Integrates Efficiently with Language Models
Venue: arXiv cs.CL 2023
Year: 2023

While Large Language Models (LLMs) become ever more dominant, classic pre-trained word embeddings sustain their relevance through computational efficiency and nuanced linguistic interpretation. Drawing from recent studies demonstrating that the convergence of GloVe and word2vec optimizations all tend towards log-co-occurrence matrix variants, we construct a novel word representation system called Bit-cipher that eliminates the need of backpropagation while leveraging contextual information and hyper-efficient dimensionality reduction techniques based on unigram frequency, providing strong interpretability, alongside efficiency. We use the bit-cipher algorithm to train word vectors via a two-step process that critically relies on a hyperparameter -- bits -- that controls the vector dimension. While the first step trains the bit-cipher, the second utilizes it under two different aggregation modes -- summation or concatenation -- to produce contextually rich representations from word co-occurrences. We extend our investigation into bit-cipher's efficacy, performing probing experiments on part-of-speech (POS) tagging and named entity recognition (NER) to assess its competitiveness with classic embeddings like word2vec and GloVe. Additionally, we explore its applicability in LM training and fine-tuning. By replacing embedding layers with cipher embeddings, our experiments illustrate the notable efficiency of cipher in accelerating the training process and attaining better optima compared to conventional training paradigms. Experiments on the integration of bit-cipher embedding layers with Roberta, T5, and OPT, prior to or as a substitute for fine-tuning, showcase a promising enhancement to transfer learning, allowing rapid model convergence while preserving competitive performance.
Having Difficulty Understanding Manuals? Automatically Converting User Manuals into Instructional Videos
Venue: arXiv cs.HC 2023
Year: 2023

While users tend to perceive instructional videos as an experience rather than a lesson with a set of instructions, instructional videos are more effective and appealing than textual user manuals and eliminate the ambiguity in text-based descriptions. However, most software vendors only offer document manuals that describe how to install and use their software, leading burden for non-professionals to comprehend the instructions. In this paper, we present a framework called M2V to generate instructional videos automatically based on the provided instructions and images in user manuals. M2V is a two-step framework. First, an action sequence is extracted from the given user manual via natural language processing and computer vision techniques. Second, M2V operates the software sequentially based on the extracted actions; meanwhile, the operation procedure is recorded into an instructional video. We evaluate the usability of automatically generated instructional videos via user studies and an online survey. The evaluation results show, with our toolkit, the generated instructional videos can better assist non-professional end users with the software operations. Moreover, more than 85% of survey participants prefer to use the instructional videos rather than the original user manuals.
Orca 2: Teaching Small Language Models How to Reason
Venue: arXiv cs.AI 2023
Year: 2023

Orca 1 learns from rich signals, such as explanation traces, allowing it to outperform conventional instruction-tuned models on benchmarks like BigBench Hard and AGIEval. In Orca 2, we continue exploring how improved training signals can enhance smaller LMs' reasoning abilities. Research on training small LMs has often relied on imitation learning to replicate the output of more capable models. We contend that excessive emphasis on imitation may restrict the potential of smaller models. We seek to teach small LMs to employ different solution strategies for different tasks, potentially different from the one used by the larger model. For example, while larger models might provide a direct answer to a complex task, smaller models may not have the same capacity. In Orca 2, we teach the model various reasoning techniques (step-by-step, recall then generate, recall-reason-generate, direct answer, etc.). More crucially, we aim to help the model learn to determine the most effective solution strategy for each task. We evaluate Orca 2 using a comprehensive set of 15 diverse benchmarks (corresponding to approximately 100 tasks and over 36,000 unique prompts). Orca 2 significantly surpasses models of similar size and attains performance levels similar or better to those of models 5-10x larger, as assessed on complex tasks that test advanced reasoning abilities in zero-shot settings. make Orca 2 weights publicly available at aka.ms/orca-lm to support research on the development, evaluation, and alignment of smaller LMs
CLIPSwarm: Converting text into formations of robots
Venue: arXiv cs.RO 2023
Year: 2023

We present CLIPSwarm, an algorithm to generate robot swarm formations from natural language descriptions. CLIPSwarm receives an input text and finds the position of the robots to form a shape that corresponds to the given text. To do so, we implement a variation of the Montecarlo particle filter to obtain a matching formation iteratively. In every iteration, we generate a set of new formations and evaluate their Clip Similarity with the given text, selecting the best formations according to this metric. This metric is obtained using Clip, [1], an existing foundation model trained to encode images and texts into vectors within a common latent space. The comparison between these vectors determines how likely the given text describes the shapes. Our initial proof of concept shows the potential of this solution to generate robot swarm formations just from natural language descriptions and demonstrates a novel application of foundation models, such as CLIP, in the field of multi-robot systems. In this first approach, we create formations using a Convex-Hull approach. Next steps include more robust and generic representation and optimization steps in the process of obtaining a suitable swarm formation.
The Weighted HOM-Problem over Fields
Venue: arXiv cs.FL 2023
Year: 2023

The HOM-problem, which asks whether the image of a regular tree language under a tree homomorphism is again regular, is known to be decidable. Since then, weighted versions of this problem for different semirings have also been investigated. In this paper, we prove the weighted HOM-problem for all fields decidable, provided that the tree homomorphism is tetris-free (a condition that generalizes injectivity). To this end, we reduce the problem to a property of the device representing the homomorphic image in question; to prove this property decidable, we then derive a pumping lemma for such devices from the well-known pumping lemma for regular tree series over fields, proved by Berstel and Reutenauer in 1982.
The Persian Piano Corpus: A Collection Of Instrument-Based Feature Extracted Data Considering Dastgah
Venue: arXiv cs.SD 2023
Year: 2023

The research in the field of music is rapidly growing, and this trend emphasizes the need for comprehensive data. Though researchers have made an effort to contribute their own datasets, many data collections lack the requisite inclusivity for comprehensive study because they are frequently focused on particular components of music or other specific topics. We have endeavored to address data scarcity by employing an instrument-based approach to provide a complete corpus related to the Persian piano. Our piano corpus includes relevant labels for Persian music mode (Dastgah) and comprehensive metadata, allowing for utilization in various popular research areas. The features extracted from 2022 Persian piano pieces in The Persian Piano Corpus (PPC) have been collected and made available to researchers, aiming for a more thorough understanding of Persian music and the role of the piano in it in subsequent steps.
Compositional Fusion of Signals in Data Embedding
Venue: arXiv cs.LG 2023
Year: 2023

Embeddings in AI convert symbolic structures into fixed-dimensional vectors, effectively fusing multiple signals. However, the nature of this fusion in real-world data is often unclear. To address this, we introduce two methods: (1) Correlation-based Fusion Detection, measuring correlation between known attributes and embeddings, and (2) Additive Fusion Detection, viewing embeddings as sums of individual vectors representing attributes. Applying these methods, word embeddings were found to combine semantic and morphological signals. BERT sentence embeddings were decomposed into individual word vectors of subject, verb and object. In the knowledge graph-based recommender system, user embeddings, even without training on demographic data, exhibited signals of demographics like age and gender. This study highlights that embeddings are fusions of multiple signals, from Word2Vec components to demographic hints in graph embeddings.
Introducing NCL-SM: A Fully Annotated Dataset of Images from Human Skeletal Muscle Biopsies
Venue: arXiv cs.CV 2023
Year: 2023

Single cell analysis of skeletal muscle (SM) tissue is a fundamental tool for understanding many neuromuscular disorders. For this analysis to be reliable and reproducible, identification of individual fibres within microscopy images (segmentation) of SM tissue should be precise. There is currently no tool or pipeline that makes automatic and precise segmentation and curation of images of SM tissue cross-sections possible. Biomedical scientists in this field rely on custom tools and general machine learning (ML) models, both followed by labour intensive and subjective manual interventions to get the segmentation right. We believe that automated, precise, reproducible segmentation is possible by training ML models. However, there are currently no good quality, publicly available annotated imaging datasets available for ML model training. In this paper we release NCL-SM: a high quality bioimaging dataset of 46 human tissue sections from healthy control subjects and from patients with genetically diagnosed muscle pathology. These images include $>$ 50k manually segmented muscle fibres (myofibres). In addition we also curated high quality myofibres and annotated reasons for rejecting low quality myofibres and regions in SM tissue images, making this data completely ready for downstream analysis. This, we believe, will pave the way for development of a fully automatic pipeline that identifies individual myofibres within images of tissue sections and, in particular, also classifies individual myofibres that are fit for further analysis.
Collectively Simplifying Trajectories in a Database: A Query Accuracy Driven Approach
Venue: arXiv cs.DB 2023
Year: 2023

Increasing and massive volumes of trajectory data are being accumulated that may serve a variety of applications, such as mining popular routes or identifying ridesharing candidates. As storing and querying massive trajectory data is costly, trajectory simplification techniques have been introduced that intuitively aim to reduce the sizes of trajectories, thus reducing storage and speeding up querying, while preserving as much information as possible. Existing techniques rely mainly on hand-crafted error measures when deciding which point to drop when simplifying a trajectory. While the hope may be that such simplification affects the subsequent usability of the data only minimally, the usability of the simplified data remains largely unexplored. Instead of using error measures that indirectly may to some extent yield simplified trajectories with high usability, we adopt a direct approach to simplification and present the first study of query accuracy driven trajectory simplification, where the direct objective is to achieve a simplified trajectory database that preserves the query accuracy of the original database as much as possible. Specifically, we propose a multi-agent reinforcement learning based solution with two agents working cooperatively to collectively simplify trajectories in a database while optimizing query usability. Extensive experiments on four real-world trajectory datasets show that the solution is capable of consistently outperforming baseline solutions over various query types and dynamics.
Unraveling the "Anomaly" in Time Series Anomaly Detection: A Self-supervised Tri-domain Solution
Venue: arXiv cs.LG 2023
Year: 2023

The ongoing challenges in time series anomaly detection (TSAD), notably the scarcity of anomaly labels and the variability in anomaly lengths and shapes, have led to the need for a more efficient solution. As limited anomaly labels hinder traditional supervised models in TSAD, various SOTA deep learning techniques, such as self-supervised learning, have been introduced to tackle this issue. However, they encounter difficulties handling variations in anomaly lengths and shapes, limiting their adaptability to diverse anomalies. Additionally, many benchmark datasets suffer from the problem of having explicit anomalies that even random functions can detect. This problem is exacerbated by ill-posed evaluation metrics, known as point adjustment (PA), which can result in inflated model performance. In this context, we propose a novel self-supervised learning based Tri-domain Anomaly Detector (TriAD), which addresses these challenges by modeling features across three data domains - temporal, frequency, and residual domains - without relying on anomaly labels. Unlike traditional contrastive learning methods, TriAD employs both inter-domain and intra-domain contrastive loss to learn common attributes among normal data and differentiate them from anomalies. Additionally, our approach can detect anomalies of varying lengths by integrating with a discord discovery algorithm. It is worth noting that this study is the first to reevaluate the deep learning potential in TSAD, utilizing both rigorously designed datasets (i.e., UCR Archive) and evaluation metrics (i.e., PA%K and affiliation). Through experimental results on the UCR dataset, TriAD achieves an impressive three-fold increase in PA%K based F1 scores over SOTA deep learning models, and 50% increase of accuracy as compared to SOTA discord discovery algorithms.
Submeter-level Land Cover Mapping of Japan
Venue: arXiv cs.CV 2023
Year: 2023

Deep learning has shown promising performance in submeter-level mapping tasks; however, the annotation cost of submeter-level imagery remains a challenge, especially when applied on a large scale. In this paper, we present the first submeter-level land cover mapping of Japan with eight classes, at a relatively low annotation cost. We introduce a human-in-the-loop deep learning framework leveraging OpenEarthMap, a recently introduced benchmark dataset for global submeter-level land cover mapping, with a U-Net model that achieves national-scale mapping with a small amount of additional labeled data. By adding a small amount of labeled data of areas or regions where a U-Net model trained on OpenEarthMap clearly failed and retraining the model, an overall accuracy of 80\% was achieved, which is a nearly 16 percentage point improvement after retraining. Using aerial imagery provided by the Geospatial Information Authority of Japan, we create land cover classification maps of eight classes for the entire country of Japan. Our framework, with its low annotation cost and high-accuracy mapping results, demonstrates the potential to contribute to the automatic updating of national-scale land cover mapping using submeter-level optical remote sensing data. The mapping results will be made publicly available.
Systematic Analysis of Security and Vulnerabilities in Miniapps
Venue: arXiv cs.CR 2023
Year: 2023

The past few years have witnessed a boom of miniapps, as lightweight applications, miniapps are of great importance in the mobile internet sector. Consequently, the security of miniapps can directly impact compromising the integrity of sensitive data, posing a potential threat to user privacy. However, after a thorough review of the various research efforts in miniapp security, we found that their actions in researching the safety of miniapp web interfaces are limited. This paper proposes a triad threat model focusing on users, servers and attackers to mitigate the security risk of miniapps. By following the principle of least privilege and the direction of permission consistency, we design a novel analysis framework for the security risk assessment of miniapps by this model. Then, we analyzed the correlation between the security risk assessment and the threat model associated with the miniapp. This analysis led to identifying potential scopes and categorisations with security risks. In the case study, we identify nine major categories of vulnerability issues, such as SQL injection, logical vulnerabilities and cross-site scripting. We also assessed a total of 50,628 security risk hazards and provided specific examples.
PIMSAB: A Processing-In-Memory System with Spatially-Aware Communication and Bit-Serial-Aware Computation
Venue: arXiv cs.AR 2023
Year: 2023

Bit-serial Processing-In-Memory (PIM) is an attractive paradigm for accelerator architectures, for parallel workloads such as Deep Learning (DL), because of its capability to achieve massive data parallelism at a low area overhead and provide orders-of-magnitude data movement savings by moving computational resources closer to the data. While many PIM architectures have been proposed, improvements are needed in communicating intermediate results to consumer kernels, for communication between tiles at scale, for reduction operations, and for efficiently performing bit-serial operations with constants. We present PIMSAB, a scalable architecture that provides spatially aware communication network for efficient intra-tile and inter-tile data movement and provides efficient computation support for generally inefficient bit-serial compute patterns. Our architecture consists of a massive hierarchical array of compute-enabled SRAMs (CRAMs) and is codesigned with a compiler to achieve high utilization. The key novelties of our architecture are: (1) providing efficient support for spatially-aware communication by providing local H-tree network for reductions, by adding explicit hardware for shuffling operands, and by deploying systolic broadcasting, and (2) taking advantage of the divisible nature of bit-serial computations through adaptive precision, bit-slicing and efficient handling of constant operations. When compared against a similarly provisioned modern Tensor Core GPU (NVIDIA A100), across common DL kernels and an end-to-end DL network (Resnet18), PIMSAB outperforms the GPU by 3x, and reduces energy by 4.2x. We compare PIMSAB with similarly provisioned state-of-the-art SRAM PIM (Duality Cache) and DRAM PIM (SIMDRAM) and observe a speedup of 3.7x and 3.88x respectively.
Attention-based Multi-fidelity Machine Learning Model for Computational Fractional Flow Reserve Assessment
Venue: arXiv cs.CE 2023
Year: 2023

Coronary Artery Disease (CAD) is one of the most common forms of heart disease, which is caused by a buildup of atherosclerotic plaque (known as stenosis) in the coronary arteries, leading to insufficient supplement of blood, oxygen, and nutrients to the heart. Fractional Flow Reserve (FFR), measuring the pressure ratio between the aorta and distal coronary artery, is an invasive physiologic gold standard for assessing the severity of coronary artery stenosis. Despite its benefits, invasive FFR assessment is still underutilized due to its high cost, time-consuming, experimental variability, and increased risk to patients. In this study, an attention-based multi-fidelity machine learning model (AttMulFid) is proposed for computationally efficient and accurate FFR assessment with uncertainty measurement. Within AttMulFid, an autoencoder is utilized to intelligently select geometric features from coronary arteries, with additional attention on the key area. Results show that the geometric features are able to represent the entirety of the geometric information and intelligently allocate attention based on crucial properties of geometry. Furthermore, the AttMulFid is a feasible approach for non-invasive, rapid, and accurate FFR assessment (with 0.002s/simulation).