OpenRaiser/Intern-Atlas huggingface dataset:
Long-Form Speech Translation through Segmentation with Finite-State Decoding Constraints on Large Language Models
Venue: arXiv cs.CL 2023
Year: 2023

One challenge in speech translation is that plenty of spoken content is long-form, but short units are necessary for obtaining high-quality translations. To address this mismatch, we adapt large language models (LLMs) to split long ASR transcripts into segments that can be independently translated so as to maximize the overall translation quality. We overcome the tendency of hallucination in LLMs by incorporating finite-state constraints during decoding; these eliminate invalid outputs without requiring additional training. We discover that LLMs are adaptable to transcripts containing ASR errors through prompt-tuning or fine-tuning. Relative to a state-of-the-art automatic punctuation baseline, our best LLM improves the average BLEU by 2.9 points for English-German, English-Spanish, and English-Arabic TED talk translation in 9 test sets, just by improving segmentation.
Redefining Access to Large Audiovisual Archives through Embodied Experiences in Immersive Environments: Creativity & Cognition 2022 -- Graduate Student Symposium
Venue: arXiv cs.HC 2023
Year: 2023

Audiovisual archives are the mnemonic archives of the 21st century, with important cultural institutions increasingly digitizing their video collections. However, these remain mostly inaccessible, due to the sheer amount of content combined with the lack of innovative forms of engagement through compelling frameworks for their exploration. The present research therefore aims at redefining access to large video collections through embodied experiences in immersive environments. The author claims that, once users are empowered to be actors of the experience rather than mere spectators, their creativity is stimulated and narrative can emerge.
Automatic Sensor-free Affect Detection: A Systematic Literature Review
Venue: arXiv cs.HC 2023
Year: 2023

Emotions and other affective states play a pivotal role in cognition and, consequently, the learning process. It is well-established that computer-based learning environments (CBLEs) that can detect and adapt to students' affective states can enhance learning outcomes. However, practical constraints often pose challenges to the deployment of sensor-based affect detection in CBLEs, particularly for large-scale or long-term applications. As a result, sensor-free affect detection, which exclusively relies on logs of students' interactions with CBLEs, emerges as a compelling alternative. This paper provides a comprehensive literature review on sensor-free affect detection. It delves into the most frequently identified affective states, the methodologies and techniques employed for sensor development, the defining attributes of CBLEs and data samples, as well as key research trends. Despite the field's evident maturity, demonstrated by the consistent performance of the models and the application of advanced machine learning techniques, there is ample scope for future research. Potential areas for further exploration include enhancing the performance of sensor-free detection models, amassing more samples of underrepresented emotions, and identifying additional emotions. There is also a need to refine model development practices and methods. This could involve comparing the accuracy of various data collection techniques, determining the optimal granularity of duration, establishing a shared database of action logs and emotion labels, and making the source code of these models publicly accessible. Future research should also prioritize the integration of models into CBLEs for real-time detection, the provision of meaningful interventions based on detected emotions, and a deeper understanding of the impact of emotions on learning.
Eye Tracking for Tele-robotic Surgery: A Comparative Evaluation of Head-worn Solutions
Venue: arXiv cs.HC 2023
Year: 2023

Purpose: Metrics derived from eye-gaze-tracking and pupillometry show promise for cognitive load assessment, potentially enhancing training and patient safety through user-specific feedback in tele-robotic surgery. However, current eye-tracking solutions' effectiveness in tele-robotic surgery is uncertain compared to everyday situations due to close-range interactions causing extreme pupil angles and occlusions. To assess the effectiveness of modern eye-gaze-tracking solutions in tele-robotic surgery, we compare the Tobii Pro 3 Glasses and Pupil Labs Core, evaluating their pupil diameter and gaze stability when integrated with the da Vinci Research Kit (dVRK). Methods: The study protocol includes a nine-point gaze calibration followed by pick-and-place task using the dVRK and is repeated three times. After a final calibration, users view a 3x3 grid of AprilTags, focusing on each marker for 10 seconds, to evaluate gaze stability across dVRK-screen positions with the L2-norm. Different gaze calibrations assess calibration's temporal deterioration due to head movements. Pupil diameter stability is evaluated using the FFT from the pupil diameter during the pick-and-place tasks. Users perform this routine with both head-worn eye-tracking systems. Results: Data collected from ten users indicate comparable pupil diameter stability. FFTs of pupil diameters show similar amplitudes in high-frequency components. Tobii Glasses show more temporal gaze stability compared to Pupil Labs, though both eye trackers yield a similar 4cm error in gaze estimation without an outdated calibration. Conclusion: Both eye trackers demonstrate similar stability of the pupil diameter and gaze, when the calibration is not outdated, indicating comparable eye-tracking and pupillometry performance in tele-robotic surgery settings.
Enhancing Abstractiveness of Summarization Models through Calibrated Distillation
Venue: arXiv cs.CL 2023
Year: 2023

Sequence-level knowledge distillation reduces the size of Seq2Seq models for more efficient abstractive summarization. However, it often leads to a loss of abstractiveness in summarization. In this paper, we propose a novel approach named DisCal to enhance the level of abstractiveness (measured by n-gram overlap) without sacrificing the informativeness (measured by ROUGE) of generated summaries. DisCal exposes diverse pseudo summaries with two supervision to the student model. Firstly, the best pseudo summary is identified in terms of abstractiveness and informativeness and used for sequence-level distillation. Secondly, their ranks are used to ensure the student model to assign higher prediction scores to summaries with higher ranks. Our experiments show that DisCal outperforms prior methods in abstractive summarization distillation, producing highly abstractive and informative summaries.
Comparative Analysis of Machine Learning Algorithms for Solar Irradiance Forecasting in Smart Grids
Venue: arXiv cs.LG 2023
Year: 2023

The increasing global demand for clean and environmentally friendly energy resources has caused increased interest in harnessing solar power through photovoltaic (PV) systems for smart grids and homes. However, the inherent unpredictability of PV generation poses problems associated with smart grid planning and management, energy trading and market participation, demand response, reliability, etc. Therefore, solar irradiance forecasting is essential for optimizing PV system utilization. This study proposes the next-generation machine learning algorithms such as random forests, Extreme Gradient Boosting (XGBoost), Light Gradient Boosted Machine (lightGBM) ensemble, CatBoost, and Multilayer Perceptron Artificial Neural Networks (MLP-ANNs) to forecast solar irradiance. Besides, Bayesian optimization is applied to hyperparameter tuning. Unlike tree-based ensemble algorithms that select the features intrinsically, MLP-ANN needs feature selection as a separate step. The simulation results indicate that the performance of the MLP-ANNs improves when feature selection is applied. Besides, the random forest outperforms the other learning algorithms.
Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks
Venue: arXiv cs.CL 2023
Year: 2023

Large Language Models (LLMs) evaluation is a patchy and inconsistent landscape, and it is becoming clear that the quality of automatic evaluation metrics is not keeping up with the pace of development of generative models. We aim to improve the understanding of current models' performance by providing a preliminary and hybrid evaluation on a range of open and closed-source generative LLMs on three NLP benchmarks: text summarisation, text simplification and grammatical error correction (GEC), using both automatic and human evaluation. We also explore the potential of the recently released GPT-4 to act as an evaluator. We find that ChatGPT consistently outperforms many other popular models according to human reviewers on the majority of metrics, while scoring much more poorly when using classic automatic evaluation metrics. We also find that human reviewers rate the gold reference as much worse than the best models' outputs, indicating the poor quality of many popular benchmarks. Finally, we find that GPT-4 is capable of ranking models' outputs in a way which aligns reasonably closely to human judgement despite task-specific variations, with a lower alignment in the GEC task.
Plausibility Processing in Transformer Language Models: Focusing on the Role of Attention Heads in GPT
Venue: arXiv cs.CL 2023
Year: 2023

The goal of this paper is to explore how Transformer language models process semantic knowledge, especially regarding the plausibility of noun-verb relations. First, I demonstrate GPT2 exhibits a higher degree of similarity with humans in plausibility processing compared to other Transformer language models. Next, I delve into how knowledge of plausibility is contained within attention heads of GPT2 and how these heads causally contribute to GPT2's plausibility processing ability. Through several experiments, it was found that: i) GPT2 has a number of attention heads that detect plausible noun-verb relationships; ii) these heads collectively contribute to the Transformer's ability to process plausibility, albeit to varying degrees; and iii) attention heads' individual performance in detecting plausibility does not necessarily correlate with how much they contribute to GPT2's plausibility processing ability.
GPTutor: an open-source AI pair programming tool alternative to Copilot
Venue: arXiv cs.HC 2023
Year: 2023

This paper presents the latest progress of GPTutor: a ChatGPT-powered programming tool extension in Visual Studio Code. The emergence of Large Language Models (LLMs) has improved software development efficiency, but their performance can be hindered by training data limitations and prompt design issues. Existing LLM development tools often operate as black boxes, with users unable to view the prompts used and unable to improve performance by correcting prompts when errors occur. To address the aforementioned issues, GPTutor was introduced as an open-source AI pair programming tool, offering an alternative to Copilot. GPTutor empowers users to customize prompts for various programming languages and scenarios, with support for 120+ human languages and 50+ programming languages. Users can fine-tune prompts to correct the errors from LLM for precision and efficient code generation. At the end of the paper, we underscore GPTutor's potential through examples, including demonstrating its proficiency in interpreting and generating Sui-Move, a newly introduced smart contract language, using prompt engineering.
Beamforming Design for the Distributed RISs-aided THz Communications with Double-Layer True Time Delays
Venue: arXiv cs.IT 2023
Year: 2023

In this paper, we investigate the reconfigurable intelligent surface (RIS)-aided terahertz (THz) communication system with the sparse radio frequency chains antenna structure at the base station (BS). To overcome the beam split of the BS, different from the conventional single-layer true-time-delay (TTD) scheme, we propose a double-layer TTD scheme that can effectively reduce the number of large-range delay devices, which involve additional insertion loss and amplification circuitry. Next, we analyze the system performance under the proposed double-layer TTD scheme. To relieve the beam split of the RIS, we consider multiple distributed RISs to replace an ultra-large size RIS. Based on this, we formulate an achievable rate maximization problem for the distributed RISs-aided THz communications via jointly optimizing the hybrid analog/digital beamforming, time delays of the double-layer TTD network and reflection coefficients of RISs. Considering the practical hardware limitation, the finite-resolution phase shift, time delay and reflection phase are constrained. To solve the formulated problem, we first design an analog beamforming scheme including optimizing phase shift and time delay based on the RISs' locations. Then, an alternatively optimization algorithm is proposed to obtain the digital beamforming and reflection coefficients based on the minimum mean square error and coordinate update techniques. Finally, simulation results show the effectiveness of the proposed scheme.
Toward Generative Data Augmentation for Traffic Classification
Venue: arXiv cs.LG 2023
Year: 2023

Data Augmentation (DA)-augmenting training data with synthetic samples-is wildly adopted in Computer Vision (CV) to improve models performance. Conversely, DA has not been yet popularized in networking use cases, including Traffic Classification (TC). In this work, we present a preliminary study of 14 hand-crafted DAs applied on the MIRAGE19 dataset. Our results (i) show that DA can reap benefits previously unexplored in TC and (ii) foster a research agenda on the use of generative models to automate DA design.
DER Pricing Power in the Presence of Multi-Location Consumers with Load Migration Capabilities
Venue: arXiv cs.GT 2023
Year: 2023

Renewable distributed energy resources (DERs) have the potential to provide multi-location electricity consumers (MLECs) with electricity at prices lower than those offered by the grid using behind-the-meter advantages. This study examines the pricing power of such DER owners in a local environment with few competitors and how it depends on the MLEC's ability to migrate a portion of the load between locations. We simulate a dynamic game between an MLEC and the local DER owners, where the MLEC is modeled as a cost-minimizer and the DER owners as strategic profit maximizers. We show that, when the MLEC is inflexible, the DER owners' optimal behavior is to offer their electricity close to maximal prices, that is, at the grid price level. However, when the MLEC can migrate a fraction of the load to the other locations, the prices offered by the DER owners quickly decrease to the minimum level, that is, the DERs' grid feed-in tariffs quickly decrease to a lower level, depending on the load migration capability.
Adversarial Image Generation by Spatial Transformation in Perceptual Colorspaces
Venue: arXiv cs.CV 2023
Year: 2023

Deep neural networks are known to be vulnerable to adversarial perturbations. The amount of these perturbations are generally quantified using $L_p$ metrics, such as $L_0$, $L_2$ and $L_\infty$. However, even when the measured perturbations are small, they tend to be noticeable by human observers since $L_p$ distance metrics are not representative of human perception. On the other hand, humans are less sensitive to changes in colorspace. In addition, pixel shifts in a constrained neighborhood are hard to notice. Motivated by these observations, we propose a method that creates adversarial examples by applying spatial transformations, which creates adversarial examples by changing the pixel locations independently to chrominance channels of perceptual colorspaces such as $YC_{b}C_{r}$ and $CIELAB$, instead of making an additive perturbation or manipulating pixel values directly. In a targeted white-box attack setting, the proposed method is able to obtain competitive fooling rates with very high confidence. The experimental evaluations show that the proposed method has favorable results in terms of approximate perceptual distance between benign and adversarially generated images. The source code is publicly available at https://github.com/ayberkydn/stadv-torch
Smooth Number Message Authentication Code in the IoT Landscape
Venue: arXiv cs.CR 2023
Year: 2023

This paper presents the Smooth Number Message Authentication Code (SNMAC) for the context of lightweight IoT devices. The proposal is based on the use of smooth numbers in the field of cryptography, and investigates how one can use them to improve the security and performance of various algorithms or security constructs. The literature findings suggest that current IoT solutions are viable and promising, yet they should explore the potential usage of smooth numbers. The methodology involves several processes, including the design, implementation, and results evaluation. After introducing the algorithm, provides a detailed account of the experimental performance analysis of the SNMAC solution, showcasing its efficiency in real-world scenarios. Furthermore, the paper also explores the security aspects of the proposed SNMAC algorithm, offering valuable insights into its robustness and applicability for ensuring secure communication within IoT environments.
Linguistically Motivated Sign Language Segmentation
Venue: arXiv cs.CL 2023
Year: 2023

Sign language segmentation is a crucial task in sign language processing systems. It enables downstream tasks such as sign recognition, transcription, and machine translation. In this work, we consider two kinds of segmentation: segmentation into individual signs and segmentation into phrases, larger units comprising several signs. We propose a novel approach to jointly model these two tasks. Our method is motivated by linguistic cues observed in sign language corpora. We replace the predominant IO tagging scheme with BIO tagging to account for continuous signing. Given that prosody plays a significant role in phrase boundaries, we explore the use of optical flow features. We also provide an extensive analysis of hand shapes and 3D hand normalization. We find that introducing BIO tagging is necessary to model sign boundaries. Explicitly encoding prosody by optical flow improves segmentation in shallow models, but its contribution is negligible in deeper models. Careful tuning of the decoding algorithm atop the models further improves the segmentation quality. We demonstrate that our final models generalize to out-of-domain video content in a different signed language, even under a zero-shot setting. We observe that including optical flow and 3D hand normalization enhances the robustness of the model in this context.
Automatic Pronunciation Assessment -- A Review
Venue: arXiv cs.CL 2023
Year: 2023

Pronunciation assessment and its application in computer-aided pronunciation training (CAPT) have seen impressive progress in recent years. With the rapid growth in language processing and deep learning over the past few years, there is a need for an updated review. In this paper, we review methods employed in pronunciation assessment for both phonemic and prosodic. We categorize the main challenges observed in prominent research trends, and highlight existing limitations, and available resources. This is followed by a discussion of the remaining challenges and possible directions for future work.
Advancing Requirements Engineering through Generative AI: Assessing the Role of LLMs
Venue: arXiv cs.SE 2023
Year: 2023

Requirements Engineering (RE) is a critical phase in software development including the elicitation, analysis, specification, and validation of software requirements. Despite the importance of RE, it remains a challenging process due to the complexities of communication, uncertainty in the early stages and inadequate automation support. In recent years, large-language models (LLMs) have shown significant promise in diverse domains, including natural language processing, code generation, and program understanding. This chapter explores the potential of LLMs in driving RE processes, aiming to improve the efficiency and accuracy of requirements-related tasks. We propose key directions and SWOT analysis for research and development in using LLMs for RE, focusing on the potential for requirements elicitation, analysis, specification, and validation. We further present the results from a preliminary evaluation, in this context.
Continual Invariant Risk Minimization
Venue: arXiv cs.LG 2023
Year: 2023

Empirical risk minimization can lead to poor generalization behavior on unseen environments if the learned model does not capture invariant feature representations. Invariant risk minimization (IRM) is a recent proposal for discovering environment-invariant representations. IRM was introduced by Arjovsky et al. (2019) and extended by Ahuja et al. (2020). IRM assumes that all environments are available to the learning system at the same time. With this work, we generalize the concept of IRM to scenarios where environments are observed sequentially. We show that existing approaches, including those designed for continual learning, fail to identify the invariant features and models across sequentially presented environments. We extend IRM under a variational Bayesian and bilevel framework, creating a general approach to continual invariant risk minimization. We also describe a strategy to solve the optimization problems using a variant of the alternating direction method of multiplier (ADMM). We show empirically using multiple datasets and with multiple sequential environments that the proposed methods outperform or is competitive with prior approaches.
Local Differential Privacy for Number of Paths and Katz Centrality
Venue: arXiv cs.SI 2023
Year: 2023

In this paper, we give an algorithm to publish the number of paths and Katz centrality under the local differential privacy (LDP), providing a thorough theoretical analysis. Although various works have already introduced subgraph counting algorithms under LDP, they have primarily concentrated on subgraphs of up to five nodes. The challenge in extending this to larger subgraphs is the cumulative and exponential growth of noise as the subgraph size increases in any publication under LDP. We address this issue by proposing an algorithm to publish the number of paths that start at every node in the graph, leading to an algorithm that publishes the Katz centrality of all nodes. This algorithm employs multiple rounds of communication and the clipping technique. Both our theoretical and experimental assessments indicate that our algorithm exhibits acceptable bias and variance, considerably less than an algorithm that bypasses clipping. Furthermore, our Katz centrality estimation is able to recall up to 90% of the nodes with the highest Katz centrality.