Title: Bi-Phone: Modeling Inter Language Phonetic Influences in Text
Benchmark: FunGLUE
Summary: Many users are forced to use the web in a language they’re not fluent in (the second language (L2) ), often resulting in text errors influenced by their native language (L1). This work introduces Bi-Phone, a model that uses phoneme confusions between L1 and L2 to generate realistic corrupted text, evaluates its impact on language models with the new FunGLUE benchmark, and proposes a phoneme prediction task to improve model robustness.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Authors measure robustness to L1-L2 phonetic interference in natural language understanding (NLU).
Phenomenon defined by authors: Yes
Phenomenon definition: It is defined as the influence of a speaker's native language (L1) on their written use of a second language (L2), particularly phoneme-shift-based misspellings that arise due to difficulty distinguishing or producing certain L2 sounds that do not exist or differ in L1.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The goal is to evaluate the robustness of NLU models to phonetic misspellings caused by L1-L2 (native-second language) interference. 
Task item definition: A single item in the FunGLUE task dataset is a modified version of a SuperGLUE example, where one or more words in a key field (e.g., question, hypothesis, premise) have been replaced with phonetically plausible misspellings generated by the Bi-Phone model to simulate L1-L2 interference. Each item retains the original structure of the SuperGLUE task (e.g., a question and answer pair, or a premise and hypothesis) along with the original label.
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: N/A (should be the same as superglue)
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: No, no link is provided
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: No
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding