Reviewerly/exHarmony huggingface dataset:
Multiclass objects detection algorithm using DarkNet-53 and DenseNet for intelligent vehicles

Abstract Intelligent vehicles should not only be able to detect various obstacles, but also identify their categories so as to take an appropriate protection and intervention. However, the scenarios of object detection are usually complex and changeable, so how to balance the relationship between accuracy and speed is a difficult task of object detection. This paper proposes a multi-object detection algorithm using DarkNet-53 and dense convolution network (DenseNet) to further ensure maximum information flow between layers. Three 8-layer dense blocks are used to replace the last three downsampling layers in DarkNet-53 structure, so that the network can make full use of multi-layer convolution features before prediction. The loss function of coordinate prediction error in YOLOv3 is further improved to improve the detection accuracy. Extensive experiments are conducted on the public KITTI and Pascal VOC datasets, and the results demonstrate that the proposed algorithm has better robustness, and the network model is more suitable for the traffic scene in the real driving environment and has better adaptability to the objects with long distance, small size and partial occlusion.
Cyber-Social Research: Emerging Paradigms for Interventionist Education Research in the Postdigital Era

This chapter explores the processes of research-informed educational intervention and change at a time when much teaching and learning is pervasively mediated by digital tools and environments. Drawing on theories and practices of design research and agile research, we propose a model that we term ‘cyber-social research’. For this, we draw inspiration from ‘agile’ software development processes, adapting these to interventionist projects designed for education reform in the postdigital era. We outline a theory of agile research and present a case study of an educational intervention that uses agile, cyber-social research methods. For this, we describe a project that designs and evaluates the effectiveness of an innovative, browser-based concept mapping and peer review tool. This example demonstrates the significance of cyber-social research methods that harness the affordances of the digital, integrating agile and design research principles for learning in the postdigital age.
A new mixed reality tool for training in minimally invasive robotic-assisted surgery

Robotic-assisted surgery (RAS) is developing an increasing role in surgical practice. Therefore, it is of the utmost importance to introduce this paradigm into surgical training programs. However, the steep learning curve of RAS remains a problem that hinders the development and widespread use of this surgical paradigm. For this reason, it is important to be able to train surgeons in the use of RAS procedures. RAS involves distinctive features that makes its learning different to other minimally invasive surgical procedures. One of these features is that the surgeons operate using a stereoscopic console. Therefore, it is necessary to perform RAS training stereoscopically. This article presents a mixed-reality (MR) tool for the stereoscopic visualization, annotation and collaborative display of RAS surgical procedures. The tool is an MR application because it can display real stereoscopic content and augment it with virtual elements (annotations) properly registered in 3D and tracked over time. This new tool allows the registration of surgical procedures, teachers (experts) and students (trainees), so that the teacher can share a set of videos with their students, annotate them with virtual information and use a shared virtual pointer with the students. The students can visualize the videos within a web environment using their personal mobile phones or a desktop stereo system. The use of the tool has been assessed by a group of 15 surgeons during a robotic-surgery master's course. The results show that surgeons consider that this tool can be very useful in RAS training.
Performance investigation of two-stage detection techniques using traffic light detection dataset

&lt;span&gt;Using a camera to monitor an object or a group of objects over time is the process of object detection. It can be used for a variety of things, including security and surveillance, video communication, traffic light detection (TLD), object detection from compressed video in public places. In recent times, object tracking has become a popular topic in computer science particularly, the data science community, thanks to the usage of deep learning (DL) in artificial intelligence (AI). DL which convolutional neural network (CNN) as one of its techniques usually used two-stage detection methods in TLD. Despite all successes recorded in TLD through the use of two-stage detection methods, there is no study that has analyzed these methods in experimental research, studying the strength and witnesses by the researchers. Based on the needs this study analyses the applications of DL techniques in TLD. We implemented object detection for TLD using 5 two-stage detection methods with the traffic light dataset using a Jupyter notebook and the sklearn libraries. We present the achievements of two-stage detection methods in TLD, going by standard performance metrics used, FASTER-CNN was the best in detection accuracy, F1-score, precision and recall with 0.89, 0.93, 0.83 and 0.90 respectively.&lt;/span&gt;
Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation

Multimodal Sarcasm Explanation (MuSE) is a new yet challenging task, which aims to generate a natural language sentence for a multimodal social post (an image as well as its caption) to explain why it contains sarcasm.Although the existing pioneer study has achieved great success with the BART backbone, it overlooks the gap between the visual feature space and the decoder semantic space, the objectlevel metadata of the image, as well as the potential external knowledge.To solve these limitations, in this work, we propose a novel mulTisource sEmantic grAph-based Multimodal sarcasm explanation scheme, named TEAM.In particular, TEAM extracts the object-level semantic meta-data instead of the traditional global visual features from the input image.Meanwhile, TEAM resorts to ConceptNet to obtain the external related knowledge concepts for the input text and the extracted object metadata.Thereafter, TEAM introduces a multisource semantic graph that comprehensively characterize the multi-source (i.e., caption, object meta-data, external knowledge) semantic relations to facilitate the sarcasm reasoning.Extensive experiments on a public released dataset MORE verify the superiority of our model over cutting-edge methods.
Tailor: A Soft-Prompt-Based Approach to Attribute-Based Controlled Text Generation

Attribute-based Controlled Text Generation (CTG) refers to generating sentences that satisfy desirable attributes (e.g., emotions and topics).Existing work usually utilize fine-tuning or resort to extra attribute classifiers, yet suffer from increases in storage and inference time.To address these concerns, we explore attribute-based CTG in a parameter-efficient manner.In short, the proposed Tailor represents each attribute as a pre-trained continuous vector (i.e., single-attribute prompt), which guides the generation of a fixed pre-trained language model (PLM) to satisfy a pre-specified attribute.These prompts can be simply concatenated as a whole for multi-attribute CTG without any re-training.Nevertheless, this may raise problems of fluency downgrading and position sensitivity.To solve this, Tailor provides two solutions to enhance the combination.The former contains a multi-attribute prompt mask and a re-indexing position sequence to bridge the gap between the training (one singleattribute prompt for each task) and the testing stage (concatenating two prompts).The latter introduces a trainable prompt connector to further enhance the combinations.Experiments demonstrate that, only requiring 0.08% extra training parameters of the GPT-2, Tailor can achieve effective and general improvements on eleven attribute-specific generation tasks.
BADGE: Speeding Up BERT Inference after Deployment via Block-wise Bypasses and Divergence-based Early Exiting

Early exiting can reduce the average latency of pre-trained language models (PLMs) via its adaptive inference mechanism and work with other inference speed-up methods like model pruning, thus drawing much attention from the industry.In this work, we propose a novel framework, BADGE, which consists of two offthe-shelf methods for improving PLMs' early exiting.We first address the issues of training a multi-exit PLM, the backbone model for early exiting.We propose the novel architecture of block-wise bypasses, which can alleviate the conflicts in jointly training multiple intermediate classifiers and thus improve the overall performances of multi-exit PLM while introducing negligible additional flops to the model.Second, we propose a novel divergence-based early exiting (DGE) mechanism, which obtains early exiting signals by comparing the predicted distributions among the current layer and the previous layers' exits.Extensive experiments on three proprietary datasets and three GLUE benchmark tasks demonstrate that our method can obtain a better speedup-performance tradeoff than the existing baseline methods.
Evaluating Open-Domain Question Answering in the Era of Large Language Models

Lexical matching remains the de facto evaluation method for open-domain question answering (QA).Unfortunately, lexical matching fails completely when a plausible candidate answer does not appear in the list of gold answers, which is increasingly the case as we shift from extractive to generative models.The recent success of large language models (LLMs) for QA aggravates lexical matching failures since candidate answers become longer, thereby making matching with the gold answers even more challenging.Without accurate evaluation, the true progress in open-domain QA remains unknown.In this paper, we conduct a thorough analysis of various open-domain QA models, including LLMs, by manually evaluating their answers on a subset of NQ-OPEN, a popular benchmark.Our assessments reveal that while the true performance of all models is significantly underestimated, the performance of the InstructGPT (zero-shot) LLM increases by nearly +60%, making it on par with existing top models, and the InstructGPT (few-shot) model actually achieves a new state-of-the-art on NQ-OPEN.We also find that more than 50% of lexical matching failures are attributed to semantically equivalent answers.We further demonstrate that regex matching ranks QA models consistent with human judgments, although still suffering from unnecessary strictness.Finally, we demonstrate that automated evaluation models are a reasonable surrogate for lexical matching in some circumstances, but not for long-form answers generated by LLMs.The automated models struggle in detecting hallucinations in LLM answers and are thus unable to evaluate LLMs.At this time, there appears to be no substitute for human evaluation. 1What was the city of Beijing previously known as? PekingJicheng
MultiTool-CoT: GPT-3 Can Use Multiple External Tools with Chain of Thought Prompting

Large language models (LLMs) have achieved impressive performance on various reasoning tasks.To further improve the performance, we propose MultiTool-CoT, a novel framework that leverages chain-of-thought (CoT) prompting to incorporate multiple external tools, such as a calculator and a knowledge retriever, during the reasoning process.We apply MultiTool-CoT to the Task 2 dataset of NumGLUE, which requires both numerical reasoning and domain-specific knowledge.The experiments show that our method significantly outperforms strong baselines and achieves state-of-the-art performance. 1
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets

The development of large language models (LLMs) such as ChatGPT 1 has brought a lot of attention recently.However, their evaluation in the benchmark academic datasets remains under-explored due to the difficulty of evaluating the generative outputs produced by this model against the ground truth.In this paper, we aim to present a thorough evaluation of ChatGPT's performance on diverse academic datasets, covering tasks like questionanswering, text summarization, code generation, commonsense reasoning, mathematical problem-solving, machine translation, bias detection, and ethical considerations.Specifically, we evaluate ChatGPT across 140 tasks and analyze 255K responses it generates in these datasets.This makes our work the largest evaluation of ChatGPT in NLP benchmarks.In short, our study aims to validate the strengths and weaknesses of ChatGPT in various tasks and provide insights for future research using LLMs.We also report a new emergent ability to follow multi-query instructions that we mostly found in ChatGPT and other instruction-tuned models.Our extensive evaluation shows that even though ChatGPT is capable of performing a wide variety of tasks, and may obtain impressive performance in several benchmark datasets, it is still far from achieving the ability to reliably solve many challenging tasks.By providing a thorough assessment of ChatGPT's performance across diverse NLP tasks, this paper sets the stage for a targeted deployment of ChatGPT-like LLMs in real-world applications.* First three authors contributed equally. 1 https://chat.openai.com
Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions

Large language models (LLMs) can be used to generate text data for training and evaluating other models.However, creating highquality datasets with LLMs can be challenging.In this work, we explore human-AI partnerships to facilitate high diversity and accuracy in LLM-based text data generation.We first examine two approaches to diversify text generation: 1) logit suppression, which minimizes the generation of languages that have already been frequently generated, and 2) temperature sampling, which flattens the token sampling probability.We found that diversification approaches can increase data diversity but often at the cost of data accuracy (i.e., text and labels being appropriate for the target domain).To address this issue, we examined two human interventions, 1) label replacement (LR), correcting misaligned labels, and 2) out-of-scope filtering (OOSF), removing instances that are out of the user's domain of interest or to which no considered label applies.With oracle studies, we found that LR increases the absolute accuracy of models trained with diversified datasets by 14.4%.Moreover, we found that some models trained with data generated with LR interventions outperformed LLM-based few-shot classification.In contrast, OOSF was not effective in increasing model accuracy, implying the need for future work in human-in-the-loop text data generation.
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor

Instruction tuning enables pretrained language models to perform new tasks from inferencetime natural language descriptions.These approaches rely on vast amounts of human supervision in the form of crowdsourced datasets or user interactions.In this work, we introduce Unnatural Instructions: a large dataset of creative and diverse instructions, collected with virtually no human labor.We collect 64,000 examples by prompting a language model with three seed examples of instructions and eliciting a fourth.This set is then expanded by prompting the model to rephrase each instruction, creating a total of approximately 240,000 examples of instructions, inputs, and outputs.Experiments show that despite containing a fair amount of noise, training on Unnatural Instructions rivals the effectiveness of training on open-source manually-curated datasets, surpassing the performance of models such as T0++ and Tk-Instruct across various benchmarks.These results demonstrate the potential of model-generated data as a cost-effective alternative to crowdsourcing for dataset expansion and diversification.
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation

Human evaluation is the foundation upon which the evaluation of both summarization systems and automatic metrics rests.However, existing human evaluation studies for summarization either exhibit a low inter-annotator agreement or have insufficient scale, and an in-depth analysis of human evaluation is lacking.Therefore, we address the shortcomings of existing summarization evaluation along the following axes: (1) We propose a modified summarization salience protocol, Atomic Content Units (ACUs), which is based on fine-grained semantic units and allows for a high interannotator agreement.(2) We curate the Robust Summarization Evaluation (RoSE) benchmark, a large human evaluation dataset consisting of 22,000 summary-level annotations over 28 top-performing systems on three datasets.(3) We conduct a comparative study of four human evaluation protocols, underscoring potential confounding factors in evaluation setups.(4) We evaluate 50 automatic metrics and their variants using the collected human annotations across evaluation protocols and demonstrate how our benchmark leads to more statistically stable and significant results.The metrics we benchmarked include recent methods based on large language models (LLMs), GPTScore and G-Eval.Furthermore, our findings have important implications for evaluating LLMs, as we show that LLMs adjusted by human feedback (e.g., GPT-3.5) may overfit unconstrained human evaluation, which is affected by the annotators' prior, input-agnostic preferences, calling for more robust, targeted evaluation methods. Statistical Power-High statistical power is difficult to reach for human evaluation of similar-performing systems.§4.1-Increasing the sample size of human evaluation effectively raises statistical power. Summary Length-Summaries from different summarization systems show a large difference in average length.§4.2 -Difference in summary length is not well-reflected by automatic evaluation metrics.-Reference-free and reference-based human evaluation results have a near-zero correlation. Evaluation-Reference-free human evaluation strongly correlates with input-agnostic, annotator preference.Protocol Comparison -Annotator's input-agnostic preference has a strong positive correlation with summary lengths.§5.2 -Annotator's input-agnostic preference does not favor reference summaries.-Compared to smaller, fine-tuned models, zero-shot large language models (e.g.GPT-3) perform better under reference-free evaluation, but worse under reference-based evaluation.Evaluating -A higher-powered human evaluation dataset can lead to a more robust automatic metric evaluation, as shown by a tighter confidence interval and higher statistical power of metric evaluation.Automatic Metrics -Automatic metric performance differs greatly under different human evaluation protocols.§6.1 & §6.2 -Automatic metrics show relatively strong system-level correlation and moderate summary-level correlation with our robust human evaluation protocol.
LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion

We present LLM-BL E N D E R, an ensembling framework designed to attain consistently superior performance by leveraging the diverse strengths of multiple open-source large language models (LLMs).Our framework consists of two modules: PAIRRANKER and GEN-FUSER, addressing the observation that optimal LLMs for different examples can significantly vary.PAIRRANKER employs a specialized pairwise comparison method to distinguish subtle differences between candidate outputs.It jointly encodes the input text and a pair of candidates, using cross-attention encoders to determine the superior one.Our results demonstrate that PAIRRANKER exhibits the highest correlation with ChatGPT-based ranking.Then, GENFUSER aims to merge the top-ranked candidates, generating an improved output by capitalizing on their strengths and mitigating their weaknesses.To facilitate largescale evaluation, we introduce a benchmark dataset, MixInstruct, which is a mixture of multiple instruction datasets featuring oracle pairwise comparisons.Our LLM-BL E N D E R significantly outperform individual LLMs and baseline methods across various metrics, establishing a substantial performance gap.
Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models

To recognize and mitigate harms from large language models (LLMs), we need to understand the prevalence and nuances of stereotypes in LLM outputs.Toward this end, we present Marked Personas, a prompt-based method to measure stereotypes in LLMs for intersectional demographic groups without any lexicon or data labeling.Grounded in the sociolinguistic concept of markedness (which characterizes explicitly linguistically marked categories versus unmarked defaults), our proposed method is twofold: 1) prompting an LLM to generate personas, i.e., natural language descriptions, of the target demographic group alongside personas of unmarked, default groups; 2) identifying the words that significantly distinguish personas of the target group from corresponding unmarked ones.We find that the portrayals generated by GPT-3.5 and GPT-4 contain higher rates of racial stereotypes than human-written portrayals using the same prompts.The words distinguishing personas of marked (non-white, non-male) groups reflect patterns of othering and exoticizing these demographics.An intersectional lens further reveals tropes that dominate portrayals of marginalized groups, such as tropicalism and the hypersexualization of minoritized women.These representational harms have concerning implications for downstream applications like story generation.
Dynamic Transformers Provide a False Sense of Efficiency

Despite much success in natural language processing (NLP), pre-trained language models typically lead to a high computational cost during inference.Multi-exit is a mainstream approach to address this issue by making a tradeoff between efficiency and accuracy, where the saving of computation comes from an early exit.However, whether such saving from earlyexiting is robust remains unknown.Motivated by this, we first show that directly adapting existing adversarial attack approaches targeting model accuracy cannot significantly reduce inference efficiency.To this end, we propose a simple yet effective attacking framework, SAME, a novel slowdown attack framework on multi-exit models, which is specially tailored to reduce the efficiency of the multi-exit models.By leveraging the multi-exit models' design characteristics, we utilize all internal predictions to guide the adversarial sample generation instead of merely considering the final prediction.Experiments on the GLUE benchmark show that SAME can effectively diminish the efficiency gain of various multi-exit models by 80% on average, convincingly validating its effectiveness and generalization ability.
Event Extraction as Question Generation and Answering

Recent work on Event Extraction has reframed the task as Question Answering (QA), with promising results.The advantage of this approach is that it addresses the error propagation issue found in traditional token-based classification approaches by directly predicting event arguments without extracting candidates first.However, the questions are typically based on fixed templates and they rarely leverage contextual information such as relevant arguments.In addition, prior QA-based approaches have difficulty handling cases where there are multiple arguments for the same role.In this paper, we propose QGA-EE, which enables a Question Generation (QG) model to generate questions that incorporate rich contextual information instead of using fixed templates.We also propose dynamic templates to assist the training of QG model.Experiments show that QGA-EE outperforms all prior single-task-based models on the ACE05 English dataset. 1