Miracle-dz/UniFAR-FTUs huggingface dataset:
Enhancing miniBERT: Exploring Methods to Improve BERT Performance

This paper investigates BERT’s performance in fine-tuning tasks related to semantic classification, paraphrase detection, and semantic similarity. We propose four meth-ods to enhance BERT: implementing a ConvBERT-inspired mixed attention block, cosine similarity fine-tuning, progressive stacking, and multitask learning. Our results indicate that incorporating ConvBERT’s mixed attention block marginally improves the model’s performance, which we attribute to the use of pretrained weights from BERT-base-uncased. However, employing cosine similarity during fine-tuning effectively reduces overfitting. Despite its potential, multitask learning as an additional enhancement only slightly improved the results. This lack of improvement may stem from the diverse nature of the tasks, potentially requiring more specialized model architectures or optimization strategies for simultaneous handling of multiple tasks.
What are Adapters Really Efficient At?

Adapters have been positioned as a parameter-001 efficient fine-tuning (PEFT) approach, whereby 002 a minimal number of parameters are added to 003 the model and fine-tuned. However, adapters 004 have not been sufficiently analyzed to under-005 stand if PEFT translates to benefits in train-006 ing/deployment efficiency and maintainabil-007 ity/extensibility. Through extensive experi-008 ments on many adapters, tasks, and languages 009 in supervised and cross-lingual zero-shot set-010 tings, we clearly show that for Natural Lan-011 guage Understanding tasks, the parameter ef-012 ficiency in adapters does not translate to effi-013 ciency gains compared to full fine-tuning of 014 models. More precisely, adapters are relatively 015 expensive to train and have slightly higher de-016 ployment latency. Furthermore, the maintain-017 ability /extensibility benefits of adapters can be 018 achieved with simpler approaches like multi-019 task training via full fine-tuning, which also 020 provide relatively faster training times. We, 021 therefore, recommend that for moderately sized 022 models practitioners should rely on full fine-023 tuning or multi-task training rather than using 024 adapters. 025
CohExplore: Visually Supporting Students in Exploring Text Cohesion

A cohesive text allows readers to follow the described ideas and events. Exploring cohesion in text might aid students enhancing their academic writing. We introduce CohExplore, which promotes exploring and reflecting on cohesion of a given text by visualizing computed cohesion-related metrics on an overview and detailed level. Detected topics are color-coded, semantic similarity is shown via lines, while connectives and co-references in a paragraph are encoded using text decoration. Demonstrating the system, we share insights about a student-authored text.
Student Achievement Prediction Model based on TLBO-BP

Today, educational prediction analysis has become an important tool for educational institutions to analyze students, and the performance of students in educational prediction analysis is a critical feedback. However, there are serious challenges in dealing with multifactor datasets to improve the convergence and accuracy of predicting student performance. Therefore, this article comprehensively analyzes machine learning technologies, analyzes the community activities of middle-term students, and explores the impact of community activities on middle-aged students' practical skills. This paper uses Random Forest and Pearson Correlation Coefficient to analyze the relevance of the impact of community activities on student hands-on performance and use Teaching and Learning Algorithms (TLBO) to optimize Back propagation (BP) neural networks applied to predict students' future trends. The results show that the TLBO-BP model can more accurately predict dynamic changes in student performance and predict high accuracy in simple patterns.
Platform Search Design and Market Power

On the Amazon.com marketplace, both Amazon and small businesses compete in offering retail products. However, Amazon chooses what products consumers see when they search. Products sold by Amazon may have a better position compared to small business products, but the effects on consumers and sellers are unclear. Policymakers have expressed antitrust concerns about this, suspecting “self-preferencing” and “gatekeeper” market power. To study this, I develop a model where heterogeneous consumers search for differentiated products arranged on an acyclic graph (i.e., tree). Firms price in response to consumer search and how their products are arranged—highlighting how search design determines market structure. The model endogenizes consideration set formation and recovers the correlated distribution of consumer preferences and search costs. Estimated on Amazon data, I show that not accounting for product arrangement (e.g., search results and BuyBox) leads to incorrect price elasticity estimates. I provide three results on market power and antitrust policies using counterfactual product arrangements. (i) To isolate the effect of Amazon’s position advantage, I remove it through a “neutral” product arrangement. Profits shift from Amazon to small businesses, confirming Amazon’s sizable market power. However, consumers reduce their search intensity in response to reduced option value and welfare decreases. This suggests Amazon’s incentives and consumers’ preferences are aligned, weakening the claim of self-preferencing. (ii) Banning the platform owner from also being a seller reduces consumer welfare; prices increase even though product variety is unaffected. (iii) I propose an alternate policy, splitting the platform into an Amazon side and a small-business side. Giving consumers the ability to search for and “support small businesses” would alleviate the market power imbalance without harming consumers. ∗I thank Rob Porter, Mar Reguant, Gaston Illanes and Vivek Bhattacharya for their invaluable guidance and insightful advice. I also thank Brett Gordon, Matt Leisten, Ilya Morozov, William Rogerson, Anna Tuchman and Caio Waisman for helpful feedback and comments. All errors are my own. †Email: tai.lam@u.northwestern. edu
Toward Interactive Dictation

Voice dictation is an increasingly important text input modality. Existing systems that allow both dictation and editing-by-voice restrict their command language to flat templates invoked by trigger words. In this work, we study the feasibility of allowing users to interrupt their dictation with spoken editing commands in open-ended natural language. We introduce a new task and dataset, TERTiUS, to experiment with such systems. To support this flexibility in real-time, a system must incrementally segment and classify spans of speech as either dictation or command, and interpret the spans that are commands. We experiment with using large pre-trained language models to predict the edited text, or alternatively, to predict a small text-editing program. Experiments show a natural trade-off between model accuracy and latency: a smaller model achieves 30% end-state accuracy with 1.3 seconds of latency, while a larger model achieves 55% end-state accuracy with 7 seconds of latency.
Analysis of Nonverbal Qualities in Dialogue to Improve Turn-taking of Spoken Dialogue Systems

Speech is one of the most important means of communication for humans. With computer systems becoming more and more part of our daily lives, it therefore seems reasonable to develop computer programs that can communicate with human users via speech. These programs are known as spoken dialogue systems. A central part of human dialogue is the organisation of speech into alternating turns. Previous research has shown that humans are highly skilled in anticipating the end of the previous turn to prepare a timely response and keep the conversation fluid. Replicating this behavior is thus also important for enabling natural, fluent and responsive interaction with spoken dialogue systems. This dissertation puts the focus on the analysis of nonverbal qualities in dialogue to help bridge this gap. In particular, the influence of speaker personality types on nonverbal qualities is examined. Results of this analysis could help to adapt turn-taking models to the personality type of their users to simulate more natural conversation. Furthermore, in order to examine the predictive power of nonverbal qualities on turn-taking, a logistic regression model is presented that makes continuous turn-taking decisions based on nonverbal qualities. To achieve this, a multimodal dataset of spoken English task-based dialogues is utilised. Next to annotations of the turns and other nonverbal qualities such as gaze and laughter, the dataset also contains personality scores for the participants based on the Big Five model. This dissertation finds evidence that the openness trait of the Big Five model, influences the total time spoken in a dialogue, the average time between turns and the amount of times a person gets interrupted during dialogue. There is also evidence that the extroversion factor influences the amount of gaps in speech a person leaves during dialogue. The presented continuous turn-taking model does not outperform a last-known value baseline.
Adversarial Transfer Learning for Continuous Natural Language Representation

As one of the most effective tools in modern natural language processing, transfer learning has been bottle-necked by the finetuning process. In order to address this challenge, we implement the Adversarial Regularization and Bregman Proximal Point Optimization (SMART) and demonstrate its success in finetuning improvements on downstream tasks such as sentiment analysis. By careful hyperparameter tuning, we’ve gained 10.4% improvement in accuracy over the baseline min-BERT model. In addition to SMART, we’ve also studied another strategy, the Contrastive Learning with Self-Guidance (CLSG), in pursuit of further improvement. However, through careful reasoning we found that the CLSG approach necessitates huge batch size and is therefore infeasible for the downstream finetuning tasks on hand.
Investigating BERT Model’s Abilities in Multi-Task Learning and Methods for Performance Improvement

In recent years, multi-task learning (MTL) has emerged as a promising approach to enhance the performance of deep learning models by leveraging shared information across related tasks. However, the ability of popular large language models like BERT to handle multiple tasks simultaneously remains under-explored. In this paper, we investigate the performance of BERT (Devlin et al., 2019) on MTL and propose a series of methods to improve its effectiveness in this setting. We conduct experiments on three natural language processing tasks: sentiment analysis, paraphrase detection, and semantic textual similarity. Our proposed method demonstrates improvements over baseline BERT models and other MTL strategies, highlighting its potential for a wide range of applications. Meanwhile, we had findings contrary to our expectation, which outlines future work that should be conducted.
Improving the Delivery of Discharge Instructions on the Postpartum Unit by Increasing the Nurses' Knowledge

The field of nursing prides itself on two critical functions: safety and patient education. One of the mechanisms for both education and safety is the discharge process. Improvements in the discharge process have been shown to increase patient comprehension and safety. This study is grounded in the (Barach & Johnson, 2006) healthcare organizational framework called the 5 Ps microsystems assessment to explore the impacts and improvements in the discharge instruction process on a postpartum unit in a large organization. It was found that HCAHPS scores regarding discharge instructions were 55-60%, which is below what is considered satisfactory. Utilizing the microsystems assessment, a gap was found in different discharge instruction delivery by staff nurses on the postpartum unit and a lack of knowledge of key elements to meet discharge instructions standard practice and policy. A focus on three of the five Ps: the professionals, the patients, and the processes, led the study of the quality improvement project. Staff nurse surveys revealed that staff nurses needed further education and training to increase competency to follow the organization’s discharge policy. Education was provided to the nurses on what is expected to be explained during discharge instructions to be best practices. Once the nurses received more training and education there was a direct correlation in the improvement of patients’ recall of the discharge instructions. The patients’ recall of discharge instructions improved to 64% after education. This project explored that continued education, standardization of delivery of instructions, and increased accountability will close gaps in nursing staff knowledge, and delivery of discharge instructions, and increase patient comprehension and safety.
DANGER: A Framework of Danger-Aware Novel Dataset Generator Extension for Robustness Test of Machine Learning

Benchmark datasets for autonomous driving, such as KITTI, Argoverse, or Waymo are realistic, but they are designed to be too idealistic. These datasets do not contain errors, difficult driving maneuvers, or other corner cases. We propose a framework for perturbing autonomous vehicle datasets, the DANGER framework, which generates edge-case images on top of current autonomous driving datasets. The input to DANGER is a photorealistic datasets from real driving scenarios. We present the DANGER algorithm for vehicle position manipulation and the interface towards the renderer module, and present primitive generation cases applied to the virtual KITTI dataset. Our experiments prove that DANGER can be used as a framework for enlarging the current dataset to cover generative corner cases.
We do it BERTer: Comparison of Finetuning Methods to Improve Sentence Embeddings

We aim to contribute to the growing body of work that finetunes Bidirectional Encoder Representations for Transformers (BERT) (3) for natural language understanding (NLU) tasks by creating a more efficient task-specific model for (1) sentiment analysis, (2) paraphrase detection, and (3) semantic textual similarity. In this paper, we implement a model that achieves an overall accuracy of 0.622 on the test leaderboard, which is an improvement on the traditional BERT model’s overall accuracy. In doing so, we demonstrate that an ensemble method combining multi-task finetuning, cosine similarity, the AdamW optimizer
Impossibly Good Experts and How to Follow Them

, a training method that uses the estimated value function of a policy trained from expert demonstrations to guide exploration for a second reinforcement learning agent. We have shown that this algorithm outperforms several distillation baselines that incorporate both environmental reward and expert demonstrations on a set of challenging minigrid and vizdoom environments.
Hyrax: Fail-in-Place Server Operation in Cloud Platforms

Today’s cloud platforms handle server hardware failures by shutting down the affected server and only turning it back on-line once it has been repaired by a technician. At cloud scale, this all-or-nothing operating model is becoming increasingly unsustainable. This model is also at odds with technology trends, such as the need for new cooling technology. This paper introduces Hyrax, a datacenter stack that enables compute servers with failed components to continue hosting VMs while hiding the underlying degraded capacity and performance. A key enabler of Hyrax is a novel model of changes in memory interleaving when deactivating faulty memory modules. Experiments on cloud production servers show that Hyrax overcomes common hardware failures without impacting peak VM performance. In large-scale simulations with production traces, Hyrax reduces server repair requirements by 50-60% without impacting VM scheduling.
Reinforcement Learning Can Be More Efficient with Multiple Rewards

Reward design is one of the most critical and challenging aspects when formulating a task as a reinforcement learning (RL) problem. In practice, it often takes several attempts of reward speciﬁca-tion and learning with it in order to ﬁnd one that leads to sample-efﬁcient learning of the desired behavior. Instead, in this work, we study whether directly incorporating multiple alternate reward formulations of the same task in a single agent can lead to faster learning. We analyze multi-reward extensions of action-elimination algorithms and prove more favorable instance-dependent regret bounds compared to their single-reward counter-parts, both in multi-armed bandits and in tabular Markov decision processes. Our bounds scale for each state-action pair with the inverse of the largest gap among all reward functions. This suggests that learning with multiple rewards can indeed be more sample-efﬁcient, as long as the re-wards agree on an optimal policy. We further prove that when rewards do not agree, multi-reward action elimination in multi-armed bandits still learns a policy that is good across all reward functions.
Patch Specifications via Product Programs

Code patches are the basic blocks of software evolution and several testing and analysis techniques have been proposed to validate them. However, due to lack of specifications, most of these techniques focus on generic errors, such as crashes. In this vision paper, we propose to adopt product programs as a practical means of writing patch specifications that could be checked using existing testing techniques. Future work will investigate the feasibility of automatically generating product programs for real-world code patches, the ease of writing useful patch specifications, and the integration of such patch specifications with existing testing techniques.
Data Augmentation for Multi-task BERT models

BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained architecture that improves on previous models to create richer, context-dependent representations of words Devlin et al. (2018) More concretely, the hidden layers from the BERT model can provide contextual word embeddings for multiple subtasks, such as sentiment classification, paraphrase detection, and semantic similarity scoring Kokkinos and Potamianos (2017). This project is an exploration of methods to improve BERT architectures for this multi-task classification task, namely by scaling up the number of hidden layers of the model and by experimenting with a variety of dataset augmentation techniques
miniBERT and Multitasking: An Architectural Analysis

BERT, the seminal transformer architecture outlined in Devlin et al. (2018), has galvanized the progress of recent language models, paving the way for generational technologies such as the recent advancement of Generative Pre-trained Trans-formers (GPT) models (OpenAI, 2023). This paper investigates the performance of BERT’s architecture, implementing a simplified version of BERT referenced canonically as ’miniBERT’. Leveraging miniBERT, three sentence embedding tasks will be given to our model to assess multitasking performance and effective architectures. Our findings suggest that miniBERT embeddings capture significant underlying understanding of utterances, though they are limited due to their lack of cultural and context dependent comprehension. Then, through an architectural exploration this paper suggests that the architectures which effectively capitalize on BERT embeddings need not be unmanageably complex. Finally, we outline promising future explorations to improve model accuracy and give insight into future potential architectures to contrast the associated difficulties of our multitasking performance.
FloralSurf : Space-Filling Geodesic Ornaments

We propose a method to generate floral patterns on manifolds without relying on parametrizations. Taking inspiration from the literature on procedural space-filling vegetation, these patterns are made of non-intersecting ornaments that are grown on the surface by repeatedly adding different types of decorative elements, until the whole surface is covered. Each decorative element is defined by a set of geodesic Bézier splines and a set of growth points from which to continue growing the ornaments. Ornaments are grown in a greedy fashion, one decorative element at a time. At each step, we analyze a set of candidates, and retain the one that maximizes surface coverage, while ensuring that it does not intersect other ornaments. All operations in our method are performed in the intrinsic metric of the surface, thus ensuring that the derived decorations have good coverage, with neither distortions nor discontinuities, and can be grown on complex surfaces. In our method, users control the decorations by selecting the size and shape of the decorative elements and the position of the growth points. We demonstrate decorations that vary in the length of the ornaments’ lines, and the number, scale and orientation of the placed decorations. We show that these patterns mimic closely the design of hand-drawn objects. Our algorithm supports any manifold surface represented as triangle meshes. In particular, we demonstrate patterns generated on surfaces with high genus, with and without borders and holes, and that can include a mixture of thin and large features.
Back to the 1980s or Not? The Drivers of Real and Inflation Risks in Treasury Bonds *

This paper explains changes in nominal and real bond risks in a New Keynesian model of monetary policy, where habit formation preferences generate endogenously time-varying risk premia. In the 1979.Q4-2001.Q1 calibration, a strong monetary policy response to volatile supply shocks leads to “stagflations”, the inflation-output correlation is negative
A N I NTELLIGENT A ND AI-D RIVEN A PPLICATION T O P ROMOTE M ENTAL W ELLNESS A ND S ELF C ARE U SING N ATURAL L ANGUAGE P ROCESSING A ND G AMIFICATION

Poor mental health and mental illness have become more widespread in recent years, resulting in a greater need to expand the accessibility of mental health treatment. To address this issue, this study proposes an application to provide a widely appealing and easily accessible alternative to traditional mental health treatment by implementing an artificial intelligence chatbot and gamification of self-care. The AI chatbot serves as a companion, counselor, and self-help advisor for individuals who are unable to seek professional human help, while gamification incentivizes users to maintain self-care habits which can improve both physical and mental well-being. The application uses Unity as the basis for its gameplay elements, and Inworld AI to simulate realistic and dynamic conversations through natural language processing. In an experiment involving the application, results demonstrated a greater reduction of anxiety and depression symptoms among application users than among non-users. These results highlight the positive impact of AI chatbots and gamification on overall mental health.