OpenRaiser/Intern-Atlas huggingface dataset:
6G-enabled Edge AI for Metaverse: Challenges, Methods, and Future Research Directions
Venue: arXiv cs.NI 2022
Year: 2022

6G-enabled edge intelligence opens up a new era of Internet of Everything and makes it possible to interconnect people-devices-cloud anytime, anywhere. More and more next-generation wireless network smart service applications are changing our way of life and improving our quality of life. As the hottest new form of next-generation Internet applications, Metaverse is striving to connect billions of users and create a shared world where virtual and reality merge. However, limited by resources, computing power, and sensory devices, Metaverse is still far from realizing its full vision of immersion, materialization, and interoperability. To this end, this survey aims to realize this vision through the organic integration of 6G-enabled edge AI and Metaverse. Specifically, we first introduce three new types of edge-Metaverse architectures that use 6G-enabled edge AI to solve resource and computing constraints in Metaverse. Then we summarize technical challenges that these architectures face in Metaverse and the existing solutions. Furthermore, we explore how the edge-Metaverse architecture technology helps Metaverse to interact and share digital data. Finally, we discuss future research directions to realize the true vision of Metaverse with 6G-enabled edge AI.
xPACE and TASC Modeler: Tool support for data-driven context modeling
Venue: arXiv cs.SE 2022
Year: 2022

From a requirements engineering point of view, the elicitation of context-aware functionalities calls for context modeling, an early step that aims at understanding the application contexts and how it may influence user tasks. In practice, however, context modeling activities have been overlooked by practitioners due to their high complexity. To improve this situation, we implemented xPACE and TASC Modeler, which are tools that support the automation of context modeling based on existing contextual data. In this demonstration paper, we present our implementation of a data-driven context modeling approach, which is composed of a contextual data processor (xPACE) and a context model generator (TASC Modeler). We successfully evaluated the results provided by the tools in a software development project.
Lessons learned from replicating a study on information-retrieval based test case prioritization
Venue: arXiv cs.SE 2022
Year: 2022

Objective: In this study, we aim to replicate an artefact-based study on software testing to address the gap. We focus on (a) providing a step by step guide of the replication, reflecting on challenges when replicating artefact-based testing research, (b) Evaluating the replicated study concerning its validity and robustness of the findings. Method: We replicate a test case prioritization technique by Kwon et al. We replicated the original study using four programs, two from the original study and two new programs. The replication study was implemented using Python to support future replications. Results: Various general factors facilitating replications are identified, such as: (1) the importance of documentation; (2) the need of assistance from the original authors; (3) issues in the maintenance of open source repositories (e.g., concerning needed software dependencies); (4) availability of scripts. We also raised several observations specific to the study and its context, such as insights from using different mutation tools and strategies for mutant generation. Conclusion: We conclude that the study by Kwon et al. is replicable for small and medium programs and could be automated to facilitate software practitioners, given the availability of required information.
Improving Differential-Neural Distinguisher Model For DES, Chaskey, and PRESENT
Venue: arXiv cs.CR 2022
Year: 2022

In CRYPTO'19, Gohr proposed a new cryptanalysis strategy using machine learning algorithms. Combining the differential-neural distinguisher with a differential path and integrating the advanced key recovery procedure, Gohr achieved a 12-round key recovery attack on Speck32/64. Chen and Yu improved prediction accuracy of differential-neural distinguisher considering derived features from multiple-ciphertext pairs instead of single-ciphertext pairs. By modifying the kernel size of initial convolutional layer to capture more dimensional information, the prediction accuracy of differential-neural distinguisher can be improved for for three reduced symmetric ciphers. For DES, we improve the prediction accuracy of (5-6)-round differential-neural distinguisher and train a new 7-round differential-neural distinguisher. For Chaskey, we improve the prediction accuracy of (3-4)-round differential-neural distinguisher. For PRESENT, we improve the prediction accuracy of (6-7)-round differential-neural distinguisher. The source codes are available in https://drive.google.com/drive/folders/1i0RciZlGZsEpCyW-wQAy7zzJeOLJNWqL?usp=sharing.
Creating good quality meshes from smooth implicit surfaces
Venue: arXiv cs.GR 2022
Year: 2022

Visualization of implicit surfaces is an actively researched topic. While raytracing can produce high quality images, it is not well suited for creating a quick preview of the surface. Indirect algorithms (e.g. Marching Cubes) create an easily renderable triangle mesh, but the result is often not sufficiently well-structured for a good approximation of differential surface quantities (normals, curvatures, etc.). Post-processing methods usually have a considerable computational overhead, and high quality is not guaranteed. We propose a tessellation algorithm to create nearly isotropic meshes, using multi-sided implicit surfaces.
Scalable Training of Language Models using JAX pjit and TPUv4
Venue: arXiv cs.LG 2022
Year: 2022

Modern large language models require distributed training strategies due to their size. The challenges of efficiently and robustly training them are met with rapid developments on both software and hardware frontiers. In this technical report, we explore challenges and design decisions associated with developing a scalable training framework, and present a quantitative analysis of efficiency improvements coming from adopting new software and hardware solutions.
Fast Few-shot Debugging for NLU Test Suites
Venue: arXiv cs.CL 2022
Year: 2022

We study few-shot debugging of transformer based natural language understanding models, using recently popularized test suites to not just diagnose but correct a problem. Given a few debugging examples of a certain phenomenon, and a held-out test set of the same phenomenon, we aim to maximize accuracy on the phenomenon at a minimal cost of accuracy on the original test set. We examine several methods that are faster than full epoch retraining. We introduce a new fast method, which samples a few in-danger examples from the original training set. Compared to fast methods using parameter distance constraints or Kullback-Leibler divergence, we achieve superior original accuracy for comparable debugging accuracy.
Character-focused Video Thumbnail Retrieval
Venue: arXiv cs.CV 2022
Year: 2022

We explore retrieving character-focused video frames as candidates for being video thumbnails. To evaluate each frame of the video based on the character(s) present in it, characters (faces) are evaluated in two aspects: Facial-expression: We train a CNN model to measure whether a face has an acceptable facial expression for being in a video thumbnail. This model is trained to distinguish faces extracted from artworks/thumbnails, from faces extracted from random frames of videos. Prominence and interactions: Character(s) in the thumbnail should be important character(s) in the video, to prevent the algorithm from suggesting non-representative frames as candidates. We use face clustering to identify the characters in the video, and form a graph in which the prominence (frequency of appearance) of the character(s), and their interactions (co-occurrence) are captured. We use this graph to infer the relevance of the characters present in each candidate frame. Once every face is scored based on the two criteria above, we infer frame level scores by combining the scores for all the faces within a frame.
A Distant Supervision Corpus for Extracting Biomedical Relationships Between Chemicals, Diseases and Genes
Venue: arXiv cs.CL 2022
Year: 2022

We introduce ChemDisGene, a new dataset for training and evaluating multi-class multi-label document-level biomedical relation extraction models. Our dataset contains 80k biomedical research abstracts labeled with mentions of chemicals, diseases, and genes, portions of which human experts labeled with 18 types of biomedical relationships between these entities (intended for evaluation), and the remainder of which (intended for training) has been distantly labeled via the CTD database with approximately 78\% accuracy. In comparison to similar preexisting datasets, ours is both substantially larger and cleaner; it also includes annotations linking mentions to their entities. We also provide three baseline deep neural network relation extraction models trained and evaluated on our new dataset.
Fix Bugs with Transformer through a Neural-Symbolic Edit Grammar
Venue: arXiv cs.LG 2022
Year: 2022

We introduce NSEdit (neural-symbolic edit), a novel Transformer-based code repair method. Given only the source code that contains bugs, NSEdit predicts an editing sequence that can fix the bugs. The edit grammar is formulated as a regular language, and the Transformer uses it as a neural-symbolic scripting interface to generate editing programs. We modify the Transformer and add a pointer network to select the edit locations. An ensemble of rerankers are trained to re-rank the editing sequences generated by beam search. We fine-tune the rerankers on the validation set to reduce over-fitting. NSEdit is evaluated on various code repair datasets and achieved a new state-of-the-art accuracy ($24.04\%$) on the Tufano small dataset of the CodeXGLUE benchmark. NSEdit performs robustly when programs vary from packages to packages and when buggy programs are concrete. We conduct detailed analysis on our methods and demonstrate the effectiveness of each component.
Joint Coreset Construction and Quantization for Distributed Machine Learning
Venue: arXiv cs.LG 2022
Year: 2022

Coresets are small, weighted summaries of larger datasets, aiming at providing provable error bounds for machine learning (ML) tasks while significantly reducing the communication and computation costs. To achieve a better trade-off between ML error bounds and costs, we propose the first framework to incorporate quantization techniques into the process of coreset construction. Specifically, we theoretically analyze the ML error bounds caused by a combination of coreset construction and quantization. Based on that, we formulate an optimization problem to minimize the ML error under a fixed budget of communication cost. To improve the scalability for large datasets, we identify two proxies of the original objective function, for which efficient algorithms are developed. For the case of data on multiple nodes, we further design a novel algorithm to allocate the communication budget to the nodes while minimizing the overall ML error. Through extensive experiments on multiple real-world datasets, we demonstrate the effectiveness and efficiency of our proposed algorithms for a variety of ML tasks. In particular, our algorithms have achieved more than 90% data reduction with less than 10% degradation in ML performance in most cases.
LSTM-Autoencoder based Anomaly Detection for Indoor Air Quality Time Series Data
Venue: arXiv cs.LG 2022
Year: 2022

Anomaly detection for indoor air quality (IAQ) data has become an important area of research as the quality of air is closely related to human health and well-being. However, traditional statistics and shallow machine learning-based approaches in anomaly detection in the IAQ area could not detect anomalies involving the observation of correlations across several data points (i.e., often referred to as long-term dependences). We propose a hybrid deep learning model that combines LSTM with Autoencoder for anomaly detection tasks in IAQ to address this issue. In our approach, the LSTM network is comprised of multiple LSTM cells that work with each other to learn the long-term dependences of the data in a time-series sequence. Autoencoder identifies the optimal threshold based on the reconstruction loss rates evaluated on every data across all time-series sequences. Our experimental results, based on the Dunedin CO2 time-series dataset obtained through a real-world deployment of the schools in New Zealand, demonstrate a very high and robust accuracy rate (99.50%) that outperforms other similar models.
Control-oriented meta-learning
Venue: arXiv cs.RO 2022
Year: 2022

Real-time adaptation is imperative to the control of robots operating in complex, dynamic environments. Adaptive control laws can endow even nonlinear systems with good trajectory tracking performance, provided that any uncertain dynamics terms are linearly parameterizable with known nonlinear features. However, it is often difficult to specify such features a priori, such as for aerodynamic disturbances on rotorcraft or interaction forces between a manipulator arm and various objects. In this paper, we turn to data-driven modeling with neural networks to learn, offline from past data, an adaptive controller with an internal parametric model of these nonlinear features. Our key insight is that we can better prepare the controller for deployment with control-oriented meta-learning of features in closed-loop simulation, rather than regression-oriented meta-learning of features to fit input-output data. Specifically, we meta-learn the adaptive controller with closed-loop tracking simulation as the base-learner and the average tracking error as the meta-objective. With both fully-actuated and underactuated nonlinear planar rotorcraft subject to wind, we demonstrate that our adaptive controller outperforms other controllers trained with regression-oriented meta-learning when deployed in closed-loop for trajectory tracking control.
Proof Theory of Skew Non-Commutative MILL
Venue: arXiv cs.LO 2022
Year: 2022

Monoidal closed categories naturally model NMILL, non-commutative multiplicative intuitionistic linear logic: the monoidal unit and tensor interpret the multiplicative verum and conjunction; the internal hom interprets linear implication. In recent years, the weaker notion of (left) skew monoidal closed category has been proposed by Ross Street, where the three structural laws of left and right unitality and associativity are not required to be invertible, they are merely natural transformations with a specific orientation. A question arises: is it possible to find a logic which is naturally modelled by skew monoidal closed categories? We answer positively by introducing a cut-free sequent calculus for a skew version of NMILL that is a presentation of the free skew monoidal closed category. We study the proof-theoretic semantics of the sequent calculus by identifying a calculus of derivations in normal form, obtained from an adaptation of Andreoli's focusing technique to the skew setting. The resulting focused sequent calculus peculiarly employs a system of tags for keeping track of new formulae appearing in the antecedent and appropriately reducing non-deterministic choices in proof search. Focusing solves the coherence problem for skew monoidal closed categories by exhibiting an effective procedure for deciding equality of maps in the free such category.
Decidability of Intuitionistic Sentential Logic with Identity via Sequent Calculus
Venue: arXiv cs.LO 2022
Year: 2022

The aim of our paper is twofold: firstly we present a sequent calculus for an intuitionistic non-Fregean logic ISCI, which is based on the calculus presented in the paper by Chlebowski and Leszczynska-Jasion, 'An Investigation into Intuitionistic Logic with Identity' (Bulletin of the Section of Logic 48(4), p. 259-283, 2019) and, secondly, we discuss the problem of decidability of ISCI via the obtained system. The original calculus from the mentioned paper did not provide the decidability result for ISCI. There are two problems to be solved in order to obtain this result: the so called loops characteristic for intuitionistic logic and the lack of the subformula property due to the form of the identity-dedicated rules. We discuss possible routes to overcome these problems: we consider a weaker version of the subformula property, guarded by the complexity of formulas which can be included within it; we also present a proof-search procedure such that when it fails, then there exists a countermodel (in Kripke semantics for ISCI).
Superconnexivity Reconsidered
Venue: arXiv cs.LO 2022
Year: 2022

We reconsider the idea of superconnexivity, an idea that has not received much attention so far. We inspect more closely the problems with the proposal that are responsible for this disregard. However, we also suggest a slight modification of the idea that has a much better chance of delivering the desired results, which we call super-bot-connexivity.
Another Combination of Classical and Intuitionistic Conditionals
Venue: arXiv cs.LO 2022
Year: 2022

On the one hand, classical logic is an extremely successful theory, even if not being perfect. On the other hand, intuitionistic logic is, without a doubt, one of the most important non-classical logics. But, how can proponents of one logic view the other logic? In this paper, we focus on one of the directions, namely how classicists can view intuitionistic logic. To this end, we introduce an expansion of positive intuitionistic logic, both semantically and proof-theoretically, and establish soundness and strong completeness. Moreover, we discuss the interesting status of disjunction, and the possibility of combining classical logic and minimal logic. We also compare our system with the system of Caleiro and Ramos.
On the Expressive Power of the Normal Form for Branching-Time Temporal Logics
Venue: arXiv cs.FL 2022
Year: 2022

With the emerging applications that involve complex distributed systems branching-time specifications are specifically important as they reflect dynamic and non-deterministic nature of such applications. We describe the expressive power of a simple yet powerful branching-time specification framework -- branching-time normal form (BNF), which has been developed as part of clausal resolution for branching-time temporal logics. We show the encoding of Buchi Tree Automata in the language of the normal form, thus representing, syntactically, tree automata in a high-level way. Thus we can treat BNF as a normal form for the latter. These results enable us (1) to translate given problem specifications into the normal form and apply as a verification method a deductive reasoning technique -- the clausal temporal resolution; (2) to apply one of the core components of the resolution method -- the loop searching to extract, syntactically, hidden invariants in a wide range of complex temporal specifications.
A Logic for Paraconsistent Transition Systems
Venue: arXiv cs.LO 2022
Year: 2022

Modelling complex information systems often entails the need for dealing with scenarios of inconsistency in which several requirements either reinforce or contradict each other. In this kind of scenarios, arising e.g. in knowledge representation, simulation of biological systems, or quantum computation, inconsistency has to be addressed in a precise and controlled way. This paper generalises Belnap-Dunn four-valued logic, introducing paraconsistent transition systems (PTS), endowed with positive and negative accessibility relations, and a metric space over the lattice of truth values, and their modal logic.
GPT-NeoX-20B: An Open-Source Autoregressive Language Model
Venue: arXiv cs.CL 2022
Year: 2022

We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge, the largest dense autoregressive model that has publicly available weights at the time of submission. In this work, we describe \model{}'s architecture and training and evaluate its performance on a range of language-understanding, mathematics, and knowledge-based tasks. We find that GPT-NeoX-20B is a particularly powerful few-shot reasoner and gains far more in performance when evaluated five-shot than similarly sized GPT-3 and FairSeq models. We open-source the training and evaluation code, as well as the model weights, at https://github.com/EleutherAI/gpt-neox.
Multi-label topic classification for COVID-19 literature with Bioformer
Venue: arXiv cs.CL 2022
Year: 2022

We describe Bioformer team's participation in the multi-label topic classification task for COVID-19 literature (track 5 of BioCreative VII). Topic classification is performed using different BERT models (BioBERT, PubMedBERT, and Bioformer). We formulate the topic classification task as a sentence pair classification problem, where the title is the first sentence, and the abstract is the second sentence. Our results show that Bioformer outperforms BioBERT and PubMedBERT in this task. Compared to the baseline results, our best model increased micro, macro, and instance-based F1 score by 8.8%, 15.5%, 7.4%, respectively. Bioformer achieved the highest micro F1 and macro F1 scores in this challenge. In post-challenge experiments, we found that pretraining of Bioformer on COVID-19 articles further improves the performance.
Multimodal spatiotemporal graph neural networks for improved prediction of 30-day all-cause hospital readmission
Venue: arXiv cs.LG 2022
Year: 2022

Measures to predict 30-day readmission are considered an important quality factor for hospitals as accurate predictions can reduce the overall cost of care by identifying high risk patients before they are discharged. While recent deep learning-based studies have shown promising empirical results on readmission prediction, several limitations exist that may hinder widespread clinical utility, such as (a) only patients with certain conditions are considered, (b) existing approaches do not leverage data temporality, (c) individual admissions are assumed independent of each other, which is unrealistic, (d) prior studies are usually limited to single source of data and single center data. To address these limitations, we propose a multimodal, modality-agnostic spatiotemporal graph neural network (MM-STGNN) for prediction of 30-day all-cause hospital readmission that fuses multimodal in-patient longitudinal data. By training and evaluating our methods using longitudinal chest radiographs and electronic health records from two independent centers, we demonstrate that MM-STGNN achieves AUROC of 0.79 on both primary and external datasets. Furthermore, MM-STGNN significantly outperforms the current clinical reference standard, LACE+ score (AUROC=0.61), on the primary dataset. For subset populations of patients with heart and vascular disease, our model also outperforms baselines on predicting 30-day readmission (e.g., 3.7 point improvement in AUROC in patients with heart disease). Lastly, qualitative model interpretability analysis indicates that while patients' primary diagnoses were not explicitly used to train the model, node features crucial for model prediction directly reflect patients' primary diagnoses. Importantly, our MM-STGNN is agnostic to node feature modalities and could be utilized to integrate multimodal data for triaging patients in various downstream resource allocation tasks.