Title: NewsBench: A Systematic Evaluation Framework for Assessing Editorial Capabilities of Large Language Models in Chinese Journalism
Benchmark: NewsBench
Summary: The paper introduces a benchmark to evaluate LLM capabilities in Chinese journalism, with a focus on writing proficiency and safety adherence. It also proposes several GPT-4 based automated evaluation protocols and uses the benchmark to evaluate popular LLMs that can handle Chinese.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: journalistic editorial tasks
Phenomenon defined by authors: Yes
Phenomenon definition: "this paper introduces NewsBench, a systematic evaluation framework which is focused on assessing the editorial capabilities of LLMs for not only journalistic writing proficiency but also safety adherence. For journalistic writing proficiency, we focus on language fluency, logical coherence, style alignment, and instruction fulfilment, while for safety adherence we consider six facets including civil language, bias and discrimination, personal privacy, social harm, journalistic ethics, and illegal activities."
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Headline Generation, Summarization, Continuation of Writing, Expansion of Writing, and Style Refinement
Task item definition: For MCQ, prompts consist of instructions, context, and choices. For short answer questions, it consists of instruction and context. 
Task source: Expert-crafted task examples (e.g. hand-written examples)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 1,267
Dataset metadata available: Yes
Dataset metadata detail: human-written answers and explanations, domain
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric subscores detail: MCQ vs short answer questions, different facets of safety adherence (e.g., ethics, privacy, bias)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / 
New taxonomy: Specific Application / Profesisonal Domains