OpenRaiser/Intern-Atlas huggingface dataset:
Student Teacher Interaction While Learning Computer Science: Early Results from an Experiment on Undergraduates
Venue: arXiv cs.HC 2023
Year: 2023

The scope of this paper was to find out how the students in Computer Science perceive different teaching styles and how the teaching style impacts the learning desire and interest in the course. To find out, we designed and implemented an experiment in which the same groups of students (86 students) were exposed to different teaching styles (presented by the same teacher at a difference of two weeks between lectures). We tried to minimize external factors' impact by carefully selecting the dates (close ones), having the courses in the same classroom and on the same day of the week, at the same hour, and checking the number and the complexity of the introduced items to be comparable. We asked for students' feedback and we define a set of countable body signs for their involvement in the course. The results were comparable by both metrics (body language) and text analysis results, students prefer a more interactive course, with a relaxing atmosphere, and are keener to learn in these conditions.
Tendencies in Database Learning for Undergraduate Students: Learning In-Depth or Getting the Work Done?
Venue: arXiv cs.DB 2023
Year: 2023

This study explores and analyzes the learning tendencies of second-year students enrolled in different lines of study related to the Databases course. There were 79 answers collected from 191 enrolled students that were analyzed and interpreted using thematic analysis. The participants in the study provided two sets of answers, anonymously collected (at the beginning and at the end of the course), thus allowing us to have clear data regarding their interests and to find out their tendencies. We looked into their expectations and if they were met; we concluded that the students want to learn only database basics. Their main challenges were related to the course homework. We combined the information and the answers related to 1) other database-related topics that they would like to learn, 2) how they plan to use the acquired information, and 3) overall interest in learning other database-related topics. The conclusion was that students prefer learning only the basic information that could help them achieve their goals: creating an application or using it at work. For these students, Getting the work done is preferred to Learning in-depth.
Exploring and Characterizing Large Language Models For Embedded System Development and Debugging
Venue: arXiv cs.SE 2023
Year: 2023

Large language models (LLMs) have shown remarkable abilities to generate code, however their ability to develop software for embedded systems, which requires cross-domain knowledge of hardware and software has not been studied. In this paper we develop an extensible, open source hardware-in-the-loop framework to systematically evaluate leading LLMs (GPT-3.5, GPT-4, PaLM 2) to assess their capabilities and limitations for embedded system development. We observe through our study that even when these tools fail to produce working code, they consistently generate helpful reasoning about embedded design tasks. We leverage this finding to study how human programmers interact with these tools, and develop an human-AI based software engineering workflow for building embedded systems. Our evaluation platform for verifying LLM generated programs uses sensor actuator pairs for physical evaluation. We compare all three models with N=450 experiments and find surprisingly that GPT-4 especially shows an exceptional level of cross-domain understanding and reasoning, in some cases generating fully correct programs from a single prompt. In N=50 trials, GPT-4 produces functional I2C interfaces 66% of the time. GPT-4 also produces register-level drivers, code for LoRa communication, and context-specific power optimizations for an nRF52 program resulting in over 740x current reduction to 12.2uA. We also characterize the models' limitations to develop a generalizable human-AI workflow for using LLMs in embedded system development. We evaluate our workflow with 15 users including novice and expert programmers. We find that our workflow improves productivity for all users and increases the success rate for building a LoRa environmental sensor from 25% to 100%, including for users with zero hardware or C/C++ experience.
Novel Categories Discovery Via Constraints on Empirical Prediction Statistics
Venue: arXiv cs.CV 2023
Year: 2023

Novel Categories Discovery (NCD) aims to cluster novel data based on the class semantics of known classes using the open-world partial class space annotated dataset. As an alternative to the traditional pseudo-labeling-based approaches, we leverage the connection between the data sampling and the provided multinoulli (categorical) distribution of novel classes. We introduce constraints on individual and collective statistics of predicted novel class probabilities to implicitly achieve semantic-based clustering. More specifically, we align the class neuron activation distributions under Monte-Carlo sampling of novel classes in large batches by matching their empirical first-order (mean) and second-order (covariance) statistics with the multinoulli distribution of the labels while applying instance information constraints and prediction consistency under label-preserving augmentations. We then explore a directional statistics-based probability formation that learns the mixture of Von Mises-Fisher distribution of class labels in a unit hypersphere. We demonstrate the discriminative ability of our approach to realize semantic clustering of novel samples in image, video, and time-series modalities. We perform extensive ablation studies regarding data, networks, and framework components to provide better insights. Our approach maintains 94%, 93%, 85%, and 93% (approx.) classification accuracy in labeled data while achieving 90%, 84%, 72%, and 75% (approx.) clustering accuracy for novel categories in Cifar10, UCF101, MPSC-ARL, and SHAR datasets that match state-of-the-art approaches without any external clustering.
Opacity of Parametric Discrete Event Systems: Models, Decidability, and Algorithms
Venue: arXiv cs.FL 2023
Year: 2023

Finite automata (FAs) model is a popular tool to characterize discrete event systems (DESs) due to its succinctness. However, for some complex systems, it is difficult to describe the necessary details by means of FAs model. In this paper, we consider a kind of extended finite automata (EFAs) in which each transition carries a predicate over state and event parameters. We also consider a type of simplified EFAs, called Event-Parameters EFAs (EP-EFAs), where the state parameters are removed. Based upon these two parametric models, we investigate the problem of opacity analysis for parametric DESs. First of all, it is shown that EFAs model is more expressive than EP-EFAs model. Secondly, it is proved that the opacity properties for EFAs are undecidable in general. Moreover, the decidable opacity properties for EP-EFAs are investigated. We present the verification algorithms for current-state opacity, initial-state opacity and infinite-step opacity, and then discuss the complexity. This paper establishes a preliminary theory for the opacity of parametric DESs, which lays a foundation for the opacity analysis of complex systems.
On Delay Performance in Mega Satellite Networks with Inter-Satellite Links
Venue: arXiv cs.IT 2023
Year: 2023

Utilizing Low Earth Orbit (LEO) satellite networks equipped with Inter-Satellite Links (ISL) is envisioned to provide lower delay compared to traditional optical networks. However, LEO satellites have constrained energy resources as they rely on solar energy in their operations. Thus requiring special consideration when designing network topologies that do not only have low-delay link paths but also low-power consumption. In this paper, we study different satellite constellation types and network typologies and propose a novel power-efficient topology. As such, we compare three common satellite architectures, namely; (i) the theoretical random constellation, the widely deployed (ii) Walker-Delta, and (iii) Walker-Star constellations. The comparison is performed based on both the power efficiency and end-to-end delay. The results show that the proposed algorithm outperforms long-haul ISL paths in terms of energy efficiency with only a slight hit to delay performance relative to the conventional ISL topology.
Feature selection simultaneously preserving both class and cluster structures
Venue: arXiv cs.LG 2023
Year: 2023

When a data set has significant differences in its class and cluster structure, selecting features aiming only at the discrimination of classes would lead to poor clustering performance, and similarly, feature selection aiming only at preserving cluster structures would lead to poor classification performance. To the best of our knowledge, a feature selection method that simultaneously considers class discrimination and cluster structure preservation is not available in the literature. In this paper, we have tried to bridge this gap by proposing a neural network-based feature selection method that focuses both on class discrimination and structure preservation in an integrated manner. In addition to assessing typical classification problems, we have investigated its effectiveness on band selection in hyperspectral images. Based on the results of the experiments, we may claim that the proposed feature/band selection can select a subset of features that is good for both classification and clustering.
Applying HCAI in developing effective human-AI teaming: A perspective from human-AI joint cognitive systems
Venue: arXiv cs.AI 2023
Year: 2023

Research and application have used human-AI teaming (HAT) as a new paradigm to develop AI systems. HAT recognizes that AI will function as a teammate instead of simply a tool in collaboration with humans. Effective human-AI teams need to be capable of taking advantage of the unique abilities of both humans and AI while overcoming the known challenges and limitations of each member, augmenting human capabilities, and raising joint performance beyond that of either entity. The National AI Research and Strategic Plan 2023 update has recognized that research programs focusing primarily on the independent performance of AI systems generally fail to consider the functionality that AI must provide within the context of dynamic, adaptive, and collaborative teams and calls for further research on human-AI teaming and collaboration. However, there has been debate about whether AI can work as a teammate with humans. The primary concern is that adopting the "teaming" paradigm contradicts the human-centered AI (HCAI) approach, resulting in humans losing control of AI systems. This article further analyzes the HAT paradigm and the debates. Specifically, we elaborate on our proposed conceptual framework of human-AI joint cognitive systems (HAIJCS) and apply it to represent HAT under the HCAI umbrella. We believe that HAIJCS may help adopt HAI while enabling HCAI. The implications and future work for HAIJCS are also discussed. Insights: AI has led to the emergence of a new form of human-machine relationship: human-AI teaming (HAT), a paradigmatic shift in human-AI systems; We must follow a human-centered AI (HCAI) approach when applying HAT as a new design paradigm; We propose a conceptual framework of human-AI joint cognitive systems (HAIJCS) to represent and implement HAT for developing effective human-AI teaming
Fairness-Aware Graph Neural Networks: A Survey
Venue: arXiv cs.LG 2023
Year: 2023

Graph Neural Networks (GNNs) have become increasingly important due to their representational power and state-of-the-art predictive performance on many fundamental learning tasks. Despite this success, GNNs suffer from fairness issues that arise as a result of the underlying graph data and the fundamental aggregation mechanism that lies at the heart of the large class of GNN models. In this article, we examine and categorize fairness techniques for improving the fairness of GNNs. Previous work on fair GNN models and techniques are discussed in terms of whether they focus on improving fairness during a preprocessing step, during training, or in a post-processing phase. Furthermore, we discuss how such techniques can be used together whenever appropriate, and highlight the advantages and intuition as well. We also introduce an intuitive taxonomy for fairness evaluation metrics including graph-level fairness, neighborhood-level fairness, embedding-level fairness, and prediction-level fairness metrics. In addition, graph datasets that are useful for benchmarking the fairness of GNN models are summarized succinctly. Finally, we highlight key open problems and challenges that remain to be addressed.
Fault Monitoring in Passive Optical Networks using Machine Learning Techniques
Venue: arXiv cs.LG 2023
Year: 2023

Passive optical network (PON) systems are vulnerable to a variety of failures, including fiber cuts and optical network unit (ONU) transmitter/receiver failures. Any service interruption caused by a fiber cut can result in huge financial losses for service providers or operators. Identifying the faulty ONU becomes difficult in the case of nearly equidistant branch terminations because the reflections from the branches overlap, making it difficult to distinguish the faulty branch given the global backscattering signal. With increasing network size, the complexity of fault monitoring in PON systems increases, resulting in less reliable monitoring. To address these challenges, we propose in this paper various machine learning (ML) approaches for fault monitoring in PON systems, and we validate them using experimental optical time domain reflectometry (OTDR) data.
Achieving the Exactly Optimal Privacy-Utility Trade-Off with Low Communication Cost via Shared Randomness
Venue: arXiv cs.IT 2023
Year: 2023

We consider a discrete distribution estimation problem under a local differential privacy (LDP) constraint in the presence of shared randomness. By exploiting the shared randomness, we suggest a new method for constructing LDP schemes which achieve the exactly optimal privacy-utility trade-off (PUT) with the communication cost of less than or equal to the input data size for any privacy regime. The main idea is to decompose a block design scheme by Park et al. (2023), based on the combinatorial concept called resolution. The LDP scheme decomposed from a block design scheme is called a resolution of the block design scheme, and it achieves the same PUT as the original block design scheme while requiring a less communication cost. We provide two resolutions of an exactly PUT-optimal block design scheme, called the Baranyai's resolution and the cyclic shift resolution, both requiring the communication cost of less than or equal to the input data size. In particular, we show that the Baranyai's resolution achieves the minimum communication cost among all the PUT-optimal resolutions of block design schemes. One drawback of the Baranyai's resolution is that it can be obtained through a recursive algorithm in general. In contrast, the cyclic shift resolution has an explicit structure, but its communication cost can be larger than that of Baranyai's resolution. To complement this, we also suggest resolutions of other block design schemes achieving the optimal PUT for some privacy budgets, which require the minimum communication cost as the Baranyai's resolution and have explicit structures as the cyclic shift resolution.
What is the meaning of proofs? A Fregean distinction in proof-theoretic semantics
Venue: arXiv cs.LO 2023
Year: 2023

The origins of proof-theoretic semantics lie in the question of what constitutes the meaning of the logical connectives and its response: the rules of inference that govern the use of the connective. However, what if we go a step further and ask about the meaning of a proof as a whole? In this paper we address this question and lay out a framework to distinguish sense and denotation of proofs. Two questions are central here. First of all, if we have two (syntactically) different derivations, does this always lead to a difference, firstly, in sense, and secondly, in denotation? The other question is about the relation between different kinds of proof systems (here: natural deduction vs. sequent calculi) with respect to this distinction. Do the different forms of representing a proof necessarily correspond to a difference in how the inferential steps are given? In our framework it will be possible to identify denotation as well as sense of proofs not only within one proof system but also between different kinds of proof systems. Thus, we give an account to distinguish a mere syntactic divergence from a divergence in meaning and a divergence in meaning from a divergence of proof objects analogous to Frege's distinction for singular terms and sentences.
Autonomy 2.0: The Quest for Economies of Scale
Venue: arXiv cs.RO 2023
Year: 2023

With the advancement of robotics and AI technologies in the past decade, we have now entered the age of autonomous machines. In this new age of information technology, autonomous machines, such as service robots, autonomous drones, delivery robots, and autonomous vehicles, rather than humans, will provide services. In this article, through examining the technical challenges and economic impact of the digital economy, we argue that scalability is both highly necessary from a technical perspective and significantly advantageous from an economic perspective, thus is the key for the autonomy industry to achieve its full potential. Nonetheless, the current development paradigm, dubbed Autonomy 1.0, scales with the number of engineers, instead of with the amount of data or compute resources, hence preventing the autonomy industry to fully benefit from the economies of scale, especially the exponentially cheapening compute cost and the explosion of available data. We further analyze the key scalability blockers and explain how a new development paradigm, dubbed Autonomy 2.0, can address these problems to greatly boost the autonomy industry.
Hybrid Successive Interference Cancellation and Power Adaptation: a Win-Win Strategy for Robust Uplink NOMA Transmission
Venue: arXiv cs.IT 2023
Year: 2023

The aim of this paper is to reveal the importance of hybrid successive interference cancellation (SIC) and power adaptation (PA) for improving transmission robustness of uplink non-orthogonal multiple access (NOMA). Particularly, a cognitive radio inspired uplink NOMA communication scenario is considered, where one primary user is allocated one dedicated resource block, while M secondary users compete with each other to be opportunistically served by using the same resource block of the primary user. Two novel schemes are proposed for the considered scenario, namely hybrid SIC with PA (HSIC-PA) scheme and fixed SIC with PA (FSIC-PA) scheme. Both schemes can ensure that the secondary users are served without degrading the transmission reliability of the primary user compared to conventional orthogonal multiple access (OMA) based schemes. Rigorous analytical results are presented to evaluate the performance of the proposed two schemes. It is shown that both schemes can avoid outage probability error floors without any constraints on users' target rates in the high SNR regime. Furthermore, it is shown that the diversity gain achieved by the HSIC-PA scheme is M, while that of the FISC-PA scheme is only 1. Numerical results are provided to verify the developed analytical results and also demonstrate the superior performance achieved by the proposed schemes by comparing with the existing HSIC without PA (HSIC-NPA) scheme. The presented simulation results also show that HSIC-PA scheme performs the best among the three schemes, which indicates the importance of the combination of HSIC and PA for improving transmission robustness.
A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation
Venue: arXiv cs.CL 2023
Year: 2023

Recently developed large language models have achieved remarkable success in generating fluent and coherent text. However, these models often tend to 'hallucinate' which critically hampers their reliability. In this work, we address this crucial problem and propose an approach that actively detects and mitigates hallucinations during the generation process. Specifically, we first identify the candidates of potential hallucination leveraging the model's logit output values, check their correctness through a validation procedure, mitigate the detected hallucinations, and then continue with the generation process. Through extensive experiments with GPT-3.5 (text-davinci-003) on the 'article generation task', we first demonstrate the individual efficacy of our detection and mitigation techniques. Specifically, the detection technique achieves a recall of ~88% and the mitigation technique successfully mitigates 57.6% of the correctly detected hallucinations. Importantly, our mitigation technique does not introduce new hallucinations even in the case of incorrectly detected hallucinations, i.e., false positives. Then, we show that the proposed active detection and mitigation approach successfully reduces the hallucinations of the GPT-3.5 model from 47.5% to 14.5% on average. We further demonstrate the effectiveness and wide applicability of our approach through additional studies including performance on different types of questions (multi-hop and false premise questions) and with another LLM from a different model family (Vicuna). In summary, our work contributes to improving the reliability and trustworthiness of large language models, a crucial step en route to enabling their widespread adoption in real-world applications.
Internet Localization of Multi-Party Relay Users: Inherent Friction Between Internet Services and User Privacy
Venue: arXiv cs.NI 2023
Year: 2023

Internet privacy is increasingly important on the modern Internet. Users are looking to control the trail of data that they leave behind on the systems that they interact with. Multi-Party Relay (MPR) architectures lower the traditional barriers to adoption of privacy enhancing technologies on the Internet. MPRs are unique from legacy architectures in that they are able to offer privacy guarantees without paying significant performance penalties. Apple's iCloud Private Relay is a recently deployed MPR service, creating the potential for widespread consumer adoption of the architecture. However, many current Internet-scale systems are designed based on assumptions that may no longer hold for users of privacy enhancing systems like Private Relay. There are inherent tensions between systems that rely on data about users -- estimated location of a user based on their IP address, for example -- and the trend towards a more private Internet. This work studies a core function that is widely used to control network and application behavior, IP geolocation, in the context of iCloud Private Relay usage. We study the location accuracy of popular IP geolocation services compared against the published location dataset that Apple publicly releases to explicitly aid in geolocating PR users. We characterize geolocation service performance across a number of dimensions, including different countries, IP version, infrastructure provider, and time. Our findings lead us to conclude that existing approaches to IP geolocation (e.g., frequently updated databases) perform inadequately for users of the MPR architecture. For example, we find median location errors >1,000 miles in some countries for IPv4 addresses using IP2Location. Our findings lead us to conclude that new, privacy-focused, techniques for inferring user location may be required as privacy becomes a default user expectation on the Internet.