OpenRaiser/Intern-Atlas huggingface dataset:
Waiting, Banning, and Embracing: An Empirical Analysis of Adapting Policies for Generative AI in Higher Education
Venue: arXiv cs.CY 2023
Year: 2023

Generative AI tools such as ChatGPT have recently gained significant attention in higher education. This study aims to understand how universities establish policies regarding the use of AI tools and explore the factors that influence their decisions. Our study examines ChatGPT policies implemented at universities around the world, including their existence, content, and issuance dates. Specifically, we analyzed the top 500 universities according to the 2022 QS World University Rankings. Our findings indicate that there is significant variation in university policies. Less than one-third of the universities included in the study had implemented ChatGPT policies. Of the universities with ChatGPT policies, approximately 67 percent embraced ChatGPT in teaching and learning, more than twice the number of universities that banned it. The majority of the universities that ban the use of ChatGPT in assessments allow individual instructors to deviate from this restrictive policy. Our empirical analysis identifies several factors that are significantly and positively correlated with a university's likelihood of having a ChatGPT policy, including the university's academic reputation score, being in an English-speaking country, and the general public attitudes toward ChatGPT. In addition, we found that a university's likelihood of having a ban policy is positively associated with faculty student ratio, citations, and the English-speaking country dummy, while negatively associated with the number of peer universities within the same country that have banned ChatGPT. We discuss the challenges faced by universities based our empirical findings.
Instant Representation Learning for Recommendation over Large Dynamic Graphs
Venue: arXiv cs.IR 2023
Year: 2023

Recommender systems are able to learn user preferences based on user and item representations via their historical behaviors. To improve representation learning, recent recommendation models start leveraging information from various behavior types exhibited by users. In real-world scenarios, the user behavioral graph is not only multiplex but also dynamic, i.e., the graph evolves rapidly over time, with various types of nodes and edges added or deleted, which causes the Neighborhood Disturbance. Nevertheless, most existing methods neglect such streaming dynamics and thus need to be retrained once the graph has significantly evolved, making them unsuitable in the online learning environment. Furthermore, the Neighborhood Disturbance existing in dynamic graphs deteriorates the performance of neighbor-aggregation based graph models. To this end, we propose SUPA, a novel graph neural network for dynamic multiplex heterogeneous graphs. Compared to neighbor-aggregation architecture, SUPA develops a sample-update-propagate architecture to alleviate neighborhood disturbance. Specifically, for each new edge, SUPA samples an influenced subgraph, updates the representations of the two interactive nodes, and propagates the interaction information to the sampled subgraph. Furthermore, to train SUPA incrementally online, we propose InsLearn, an efficient workflow for single-pass training of large dynamic graphs. Extensive experimental results on six real-world datasets show that SUPA has a good generalization ability and is superior to sixteen state-of-the-art baseline methods. The source code is available at https://github.com/shatter15/SUPA.
Experience Filter: Using Past Experiences on Unseen Tasks or Environments
Venue: arXiv cs.RO 2023
Year: 2023

One of the bottlenecks of training autonomous vehicle (AV) agents is the variability of training environments. Since learning optimal policies for unseen environments is often very costly and requires substantial data collection, it becomes computationally intractable to train the agent on every possible environment or task the AV may encounter. This paper introduces a zero-shot filtering approach to interpolate learned policies of past experiences to generalize to unseen ones. We use an experience kernel to correlate environments. These correlations are then exploited to produce policies for new tasks or environments from learned policies. We demonstrate our methods on an autonomous vehicle driving through T-intersections with different characteristics, where its behavior is modeled as a partially observable Markov decision process (POMDP). We first construct compact representations of learned policies for POMDPs with unknown transition functions given a dataset of sequential actions and observations. Then, we filter parameterized policies of previously visited environments to generate policies to new, unseen environments. We demonstrate our approaches on both an actual AV and a high-fidelity simulator. Results indicate that our experience filter offers a fast, low-effort, and near-optimal solution to create policies for tasks or environments never seen before. Furthermore, the generated new policies outperform the policy learned using the entire data collected from past environments, suggesting that the correlation among different environments can be exploited and irrelevant ones can be filtered out.
Short Answer Grading Using One-shot Prompting and Text Similarity Scoring Model
Venue: arXiv cs.CL 2023
Year: 2023

In this study, we developed an automated short answer grading (ASAG) model that provided both analytic scores and final holistic scores. Short answer items typically consist of multiple sub-questions, and providing an analytic score and the text span relevant to each sub-question can increase the interpretability of the automated scores. Furthermore, they can be used to generate actionable feedback for students. Despite these advantages, most studies have focused on predicting only holistic scores due to the difficulty in constructing dataset with manual annotations. To address this difficulty, we used large language model (LLM)-based one-shot prompting and a text similarity scoring model with domain adaptation using small manually annotated dataset. The accuracy and quadratic weighted kappa of our model were 0.67 and 0.71 on a subset of the publicly available ASAG dataset. The model achieved a substantial improvement over the majority baseline.
Recommending Bug Assignment Approaches for Individual Bug Reports: An Empirical Investigation
Venue: arXiv cs.SE 2023
Year: 2023

Multiple approaches have been proposed to automatically recommend potential developers who can address bug reports. These approaches are typically designed to work for any bug report submitted to any software project. However, we conjecture that these approaches may not work equally well for all the reports in a project. We conducted an empirical study to validate this conjecture, using three bug assignment approaches applied on 2,249 bug reports from two open source systems. We found empirical evidence that validates our conjecture, which led us to explore the idea of identifying and applying the best-performing approach for each bug report to obtain more accurate developer recommendations. We conducted an additional study to assess the feasibility of this idea using machine learning. While we found a wide margin of accuracy improvement for this approach, it is far from achieving the maximum possible improvement and performs comparably to baseline approaches. We discuss potential reasons for these results and conjecture that the assignment approaches may not capture important information about the bug assignment process that developers perform in practice. The results warrant future research in understanding how developers assign bug reports and improving automated bug report assignment
Parity Calibration
Venue: arXiv cs.LG 2023
Year: 2023

In a sequential regression setting, a decision-maker may be primarily concerned with whether the future observation will increase or decrease compared to the current one, rather than the actual value of the future observation. In this context, we introduce the notion of parity calibration, which captures the goal of calibrated forecasting for the increase-decrease (or "parity") event in a timeseries. Parity probabilities can be extracted from a forecasted distribution for the output, but we show that such a strategy leads to theoretical unpredictability and poor practical performance. We then observe that although the original task was regression, parity calibration can be expressed as binary calibration. Drawing on this connection, we use an online binary calibration method to achieve parity calibration. We demonstrate the effectiveness of our approach on real-world case studies in epidemiology, weather forecasting, and model-based control in nuclear fusion.
Exact Distributed Stochastic Block Partitioning
Venue: arXiv cs.DC 2023
Year: 2023

Stochastic block partitioning (SBP) is a community detection algorithm that is highly accurate even on graphs with a complex community structure, but its inherently serial nature hinders its widespread adoption by the wider scientific community. To make it practical to analyze large real-world graphs with SBP, there is a growing need to parallelize and distribute the algorithm. The current state-of-the-art distributed SBP algorithm is a divide-and-conquer approach that limits communication between compute nodes until the end of inference. This leads to the breaking of computational dependencies, which causes convergence issues as the number of compute nodes increases, and when the graph is sufficiently sparse. In this paper, we introduce EDiSt - an exact distributed stochastic block partitioning algorithm. Under EDiSt, compute nodes periodically share community assignments during inference. Due to this additional communication, EDiSt improves upon the divide-and-conquer algorithm by allowing it to scale out to a larger number of compute nodes without suffering from convergence issues, even on sparse graphs. We show that EDiSt provides speedups of up to 23.8X over the divide-and-conquer approach, and speedups up to 38.0X over shared memory parallel SBP when scaled out to 64 compute nodes.
Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts
Venue: arXiv cs.AR 2023
Year: 2023

Computer vision researchers are embracing two promising paradigms: Vision Transformers (ViTs) and Multi-task Learning (MTL), which both show great performance but are computation-intensive, given the quadratic complexity of self-attention in ViT and the need to activate an entire large MTL model for one task. M$^3$ViT is the latest multi-task ViT model that introduces mixture-of-experts (MoE), where only a small portion of subnetworks ("experts") are sparsely and dynamically activated based on the current task. M$^3$ViT achieves better accuracy and over 80% computation reduction but leaves challenges for efficient deployment on FPGA. Our work, dubbed Edge-MoE, solves the challenges to introduce the first end-to-end FPGA accelerator for multi-task ViT with a collection of architectural innovations, including (1) a novel reordering mechanism for self-attention, which requires only constant bandwidth regardless of the target parallelism; (2) a fast single-pass softmax approximation; (3) an accurate and low-cost GELU approximation; (4) a unified and flexible computing unit that is shared by almost all computational layers to maximally reduce resource usage; and (5) uniquely for M$^3$ViT, a novel patch reordering method to eliminate memory access overhead. Edge-MoE achieves 2.24x and 4.90x better energy efficiency comparing with GPU and CPU, respectively. A real-time video demonstration is available online, along with our open-source code written using High-Level Synthesis.
Epistemic Graph: A Plug-And-Play Module For Hybrid Representation Learning
Venue: arXiv cs.CV 2023
Year: 2023

In recent years, deep models have achieved remarkable success in various vision tasks. However, their performance heavily relies on large training datasets. In contrast, humans exhibit hybrid learning, seamlessly integrating structured knowledge for cross-domain recognition or relying on a smaller amount of data samples for few-shot learning. Motivated by this human-like epistemic process, we aim to extend hybrid learning to computer vision tasks by integrating structured knowledge with data samples for more effective representation learning. Nevertheless, this extension faces significant challenges due to the substantial gap between structured knowledge and deep features learned from data samples, encompassing both dimensions and knowledge granularity. In this paper, a novel Epistemic Graph Layer (EGLayer) is introduced to enable hybrid learning, enhancing the exchange of information between deep features and a structured knowledge graph. Our EGLayer is composed of three major parts, including a local graph module, a query aggregation model, and a novel correlation alignment loss function to emulate human epistemic ability. Serving as a plug-and-play module that can replace the standard linear classifier, EGLayer significantly improves the performance of deep models. Extensive experiments demonstrates that EGLayer can greatly enhance representation learning for the tasks of cross-domain recognition and few-shot learning, and the visualization of knowledge graphs can aid in model interpretation.
Graph Reasoning for Question Answering with Triplet Retrieval
Venue: arXiv cs.CL 2023
Year: 2023

Answering complex questions often requires reasoning over knowledge graphs (KGs). State-of-the-art methods often utilize entities in questions to retrieve local subgraphs, which are then fed into KG encoder, e.g. graph neural networks (GNNs), to model their local structures and integrated into language models for question answering. However, this paradigm constrains retrieved knowledge in local subgraphs and discards more diverse triplets buried in KGs that are disconnected but useful for question answering. In this paper, we propose a simple yet effective method to first retrieve the most relevant triplets from KGs and then rerank them, which are then concatenated with questions to be fed into language models. Extensive results on both CommonsenseQA and OpenbookQA datasets show that our method can outperform state-of-the-art up to 4.6% absolute accuracy.
DiffSketching: Sketch Control Image Synthesis with Diffusion Models
Venue: arXiv cs.CV 2023
Year: 2023

Creative sketch is a universal way of visual expression, but translating images from an abstract sketch is very challenging. Traditionally, creating a deep learning model for sketch-to-image synthesis needs to overcome the distorted input sketch without visual details, and requires to collect large-scale sketch-image datasets. We first study this task by using diffusion models. Our model matches sketches through the cross domain constraints, and uses a classifier to guide the image synthesis more accurately. Extensive experiments confirmed that our method can not only be faithful to user's input sketches, but also maintain the diversity and imagination of synthetic image results. Our model can beat GAN-based method in terms of generation quality and human evaluation, and does not rely on massive sketch-image datasets. Additionally, we present applications of our method in image editing and interpolation.
Reviewing BPMN as a Modeling Notation for CACAO Security Playbooks
Venue: arXiv cs.CR 2023
Year: 2023

As cyber systems become increasingly complex and cybersecurity threats become more prominent, defenders must prepare, coordinate, automate, document, and share their response methodologies to the extent possible. The CACAO standard was developed to satisfy the above requirements, providing a common machine-readable framework and schema for documenting cybersecurity operations processes, including defensive tradecraft and tactics, techniques, and procedures. Although this approach is compelling, a remaining limitation is that CACAO provides no native modeling notation for graphically representing playbooks, which is crucial for simplifying their creation, modification, and understanding. In contrast, the industry is familiar with BPMN, a standards-based modeling notation for business processes that has also found its place in representing cybersecurity processes. This research examines BPMN and CACAO and explores the feasibility of using the BPMN modeling notation to represent CACAO security playbooks graphically. The results indicate that mapping CACAO and BPMN is attainable at an abstract level; however, conversion from one encoding to another introduces a degree of complexity due to the multiple ways CACAO constructs can be represented in BPMN and the extensions required in BPMN to support CACAO fully.
Clip21: Error Feedback for Gradient Clipping
Venue: arXiv cs.LG 2023
Year: 2023

Motivated by the increasing popularity and importance of large-scale training under differential privacy (DP) constraints, we study distributed gradient methods with gradient clipping, i.e., clipping applied to the gradients computed from local information at the nodes. While gradient clipping is an essential tool for injecting formal DP guarantees into gradient-based methods [1], it also induces bias which causes serious convergence issues specific to the distributed setting. Inspired by recent progress in the error-feedback literature which is focused on taming the bias/error introduced by communication compression operators such as Top-$k$ [2], and mathematical similarities between the clipping operator and contractive compression operators, we design Clip21 -- the first provably effective and practically useful error feedback mechanism for distributed methods with gradient clipping. We prove that our method converges at the same $\mathcal{O}\left(\frac{1}{K}\right)$ rate as distributed gradient descent in the smooth nonconvex regime, which improves the previous best $\mathcal{O}\left(\frac{1}{\sqrt{K}}\right)$ rate which was obtained under significantly stronger assumptions. Our method converges significantly faster in practice than competing methods.
Cross Encoding as Augmentation: Towards Effective Educational Text Classification
Venue: arXiv cs.CL 2023
Year: 2023

Text classification in education, usually called auto-tagging, is the automated process of assigning relevant tags to educational content, such as questions and textbooks. However, auto-tagging suffers from a data scarcity problem, which stems from two major challenges: 1) it possesses a large tag space and 2) it is multi-label. Though a retrieval approach is reportedly good at low-resource scenarios, there have been fewer efforts to directly address the data scarcity problem. To mitigate these issues, here we propose a novel retrieval approach CEAA that provides effective learning in educational text classification. Our main contributions are as follows: 1) we leverage transfer learning from question-answering datasets, and 2) we propose a simple but effective data augmentation method introducing cross-encoder style texts to a bi-encoder architecture for more efficient inference. An extensive set of experiments shows that our proposed method is effective in multi-label scenarios and low-resource tags compared to state-of-the-art models.
Back to Patterns: Efficient Japanese Morphological Analysis with Feature-Sequence Trie
Venue: arXiv cs.CL 2023
Year: 2023

Accurate neural models are much less efficient than non-neural models and are useless for processing billions of social media posts or handling user queries in real time with a limited budget. This study revisits the fastest pattern-based NLP methods to make them as accurate as possible, thus yielding a strikingly simple yet surprisingly accurate morphological analyzer for Japanese. The proposed method induces reliable patterns from a morphological dictionary and annotated data. Experimental results on two standard datasets confirm that the method exhibits comparable accuracy to learning-based baselines, while boasting a remarkable throughput of over 1,000,000 sentences per second on a single modern CPU. The source code is available at https://www.tkl.iis.u-tokyo.ac.jp/~ynaga/jagger/
Space MIMO: Direct Unmodified Handheld to Multi-Satellite Communication
Venue: arXiv cs.IT 2023
Year: 2023

This paper examines the uplink transmission of a single-antenna handsheld user to a cluster of satellites, with a focus on utilizing the inter-satellite links to enable cooperative signal detection. Two cases are studied: one with full CSI and the other with partial CSI between satellites. The two cases are compared in terms of capacity, overhead, and bit error rate. Additionally, the impact of channel estimation error is analyzed in both designs, and robust detection techniques are proposed to handle channel uncertainty up to a certain level. The performance of each case is demonstrated, and a comparison is made with conventional satellite communication schemes where only one satellite can connect to a user. The results of our study reveal that the proposed constellation with a total of 3168 satellites in orbit can enable a capacity of 800 Mbits/sec through cooperation of $12$ satellites with and occupied bandwidth of 500 MHz. In contrast, conventional satellite communication approaches with the same system parameters yield a significantly lower capacity of less than 150 Mbits/sec for the nearest satellite.
Multi-source adversarial transfer learning based on similar source domains with local features
Venue: arXiv cs.CV 2023
Year: 2023

Transfer learning leverages knowledge from other domains and has been successful in many applications. Transfer learning methods rely on the overall similarity of the source and target domains. However, in some cases, it is impossible to provide an overall similar source domain, and only some source domains with similar local features can be provided. Can transfer learning be achieved? In this regard, we propose a multi-source adversarial transfer learning method based on local feature similarity to the source domain to handle transfer scenarios where the source and target domains have only local similarities. This method extracts transferable local features between a single source domain and the target domain through a sub-network. Specifically, the feature extractor of the sub-network is induced by the domain discriminator to learn transferable knowledge between the source domain and the target domain. The extracted features are then weighted by an attention module to suppress non-transferable local features while enhancing transferable local features. In order to ensure that the data from the target domain in different sub-networks in the same batch is exactly the same, we designed a multi-source domain independent strategy to provide the possibility for later local feature fusion to complete the key features required. In order to verify the effectiveness of the method, we made the dataset "Local Carvana Image Masking Dataset". Applying the proposed method to the image segmentation task of the proposed dataset achieves better transfer performance than other multi-source transfer learning methods. It is shown that the designed transfer learning method is feasible for transfer scenarios where the source and target domains have only local similarities.