VERIFY: A Novel Multi-Domain Dataset Grounding LTL in Contextual Natural Language via Provable Intermediate Logic
Authors: Paapa Kwesi Quansah
Pablo Rivas
Ernest Bonnah
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Linear Temporal Logic
model checking
formal verification
Primary area: datasets and benchmarks
TL;DR: dataset for translating between LTL and natural language

Abstract: Bridging the gap between the formal precision of system specifications and the nuances of human language is critical for reliable engineering, robotics, and AI safety, but it remains a major bottleneck. Prior efforts in grounding formal logic remain fragmented, resulting in datasets that are very small-scale (~2-5k examples), domain-specific, or translate logic into overly technical forms rather than context-rich natural language (NL). Thus, failing to adequately bridge formal methods and practical NLP. To address this gap, we introduce VERIFY, the first large-scale dataset meticulously designed to unify these elements. This dataset contains more than 200k+ rigorously generated triplets, each comprising a Linear Temporal Logic (LTL) formula, a structured, human-readable 'Intermediate Technical Language' (ITL) representation designed as a bridge between logic and text, and a domain-specific NL description contextualized across 13 diverse domains. VERIFY's construction pipeline ensures high fidelity: LTL formulas are enumerated and verified via model checking, mapped to the novel ITL representation using a provably complete formal grammar, and then translated into context-aware NL via LLM-driven generation. We guarantee data quality through extensive validation protocols, i.e., manual expert verification of 10,000 diverse samples. Furthermore, automated semantic consistency checks judged by Llama 3.3 confirmed an estimated >97% semantic correctness. From the initial experiments, we demonstrate VERIFY's scalability, logical complexity, and contextual diversity, significantly challenging standard models such as T5 and Llama 3.

Published as a conference paper at ICLR 2026
VERIFY: A NOVEL MULTI-DOMAIN DATASET GROUNDING LTL IN CONTEXTUAL NATURAL LAN- GUAGE VIA PROVABLE INTERMEDIATE LOGIC
Paapa Kwesi Quansah, Ernest Bonnah,
Pablo Rivas Perea
Baylor University, Waco, Texas
beaapa_quansahi Senest_bonnee pablo_rivee rivassrvv_ ( @@ballor.ddu
ABSTRACT
Bridging the gap between the formal precision of system specifications and the nuances of human language is critical for reliable engineering, robotics, and Al safety, but it remains major bottleneck. Prior efforts in grounding formal logic remain fragmented, resulting in datasets that are very small-scale (~ 5k ex- amples), domain-specific, or translate logic into overly technical forms rather than context-rich natural language (NL), failing to adequately bridge formal methods and practical NLP. To address this gap, we introduce VERIFY, the first large-scale dataset designed unify these elements. This dataset contains more than 200k+ rigorously generated triplets, each comprising a Linear Temporal Logic (LTL) formula, a structured, human-readable 'Intermediate Technical Language' (TTL) representation designed as bridge between logic and text, and domain-specific NL description contextualized across 13 diverse domains. VERIFY's construction pipeline ensures high fidelity: LTL formulas are enumerated and verified via model checking, mapped to the novel ITL. representation using a provably complete formal grammar, and then translated into context-aware NI. via LL gener- ation. We guarantee data quality through extensive validation protocols, including manual expert verification of 10,000 diverse samples. Furthermore, automated semantic consistency checks judged by Llama 3.3 confirmed an estimated >97 semantic correctness. From the initial experiments, we demonstrate VERIFY's scalability, logical complexity, and contextual diversity, significantly challenging standard models such as T5 and I.lama 3.
O
Data & Code: wwt.uu..com//eeiist... wtt..b/oo//sessa//eeeiiiii
Data & Dataset Card: huggingface http:/ggww...ooooaaee///////e//a
kaggle Data & Dataset Card: Jeggd...co/dmassss//ssssaaavavir
INTRODUCTION
The increasing complexity of software systems, autonomous agents, and critical infrastructure, from financial trading algorithms and medical devices to aerospace controls and smart grids necessitates rigorous methods for specifying behavior and ensuring reliability (Bochm) 2002 Knight (002). For- mal methods provide mathematically grounded techniques for specifying, developing, and verifying such systems (Clarke 1997 Clarke et al., 2018 Holzmann 2004). Temporal logics such as Linear widely used for specifying and verifying properties of critical hardware, software, and communication systems (Rozier) 2011]. For instance, consider a spec- ification of home automation system that requires that "If any exterior door opens after 10 p.m., the security lights should immediately turn on and stay on until the door is closed." " Using LTL formalism, this requirement can be expressed G((t > 22:00 A door door_open) Despite this precision, the syntax and semantics of formal logic render specifications opaque to do- main experts and many developers, limiting the adoption of formal methods in practice (Fasterbrook et al. 1998 Dwyer et al. 1999). Conversely, system requirements are frequently documented in natural language (NL), which accessible but prone to ambiguity, incompleteness, and inconsistency, This leads to specification misunderstandings and costly errors in safety-critical contexts (Berry et al.) 2003 Hwang and Rine 1998. Großer et al. 2022). Closing this gap requires large-scale resources that systematicaly align unambiguous formal expressions with their context-rich NL counterparts (Wing) 1990). Thus, there is the need to create large-scale resources that systematically align such unambiguous formal expressions with their context-rich NI. counterparts.
Temporal Logie (LTL) (Pnueli 1977)
lights on U -doo -door_open)

---

Published as a conference paper at ICLR 2026
Progress toward this goal has been hampered by the limitations of available datasets (Andreas et al. 2013] Jiang et al.] 2024). Existing resources linking temporal logic and natural language typically fall short in several critical dimensions such as confinement to single, niche application domains as robotics commands (Scalise et al., 2018) or specific software verification patterns (Neyzov and Kuzmin 2023), inhibiting the development of cross-domain, generalizable models.
To address this critical gap, we introduce VERIFY, a large-scale dataset that unifies three levels of representation; LTL formulas, structured Intermediate Technical Language (ITL), and domain- specific natural language. VERIFY contains over 200 thousand rigorously generated triplets, each comprising: (i) a Linear Temporal Logic (LTL) formula specifying a temporal property, (ii) structured, human- readable 'Intermediate Technical Language' (ITL) representation, novel to this work, explicitly designed bridge the structural patterns of L.TI. with the syntax of natural language and (iii) domain-speciff Natural Language (NI.) description expressing the property in context.
The construction of VERIFY prioritizes both scale and fidelity. LTL formulas are systematically enumerated and formally verified for non-triviality and satisfiability using model checking (Clarke) 1997). Each verified LTL formula then mapped to our ITL representation using a formal grammar engineered to be provably complete with respect to the input LTL fragment. Finally, context-aware NL descriptions are generated using a state-of-the-ann reasoning large language model (Liu et al. 2024] Guo et al. 2025). conditioned on the LTL/ITL structure, the domain, and domain-speciff variable semantics. Crucially, data quality is guaranteed through extensive validation protocols: manual expert verification of 10,000 random samples and automated semantic consistency and correctness checks judged by an LLM-as-judge approach (using Llama 3.3) (Grattafiori et al. 22224] Son et al. 2024) across 18% of the dataset, confirming an estimated > 97% semantie integrity.
This work makes four primary contributions. First, we release the VERIFY dataset: over 200,000 LTL-ITL-NL triplets spanning 13 application domains. Second, we introduce the formalism, structured intermediate representation with provably total mapping from LTL. Third, we describe a multi-stage construction pipeline incorporating model checking, formal grammars, LLM generation, and extensive human and automated validation. Fourth, we report baseline experiments with T5, BART, Llama 3, Mistral, Code and DeepSeek Coder that establish performance benchmarks and reveal the difficulty of NL-to-LTL translation (Raffel et al. 2020 Grattafiori et al., 2024). The full dataset, ITL specification, and evaluation code are publicly released.
2 RELATED WORK
The challenge of bridging formal logical specifications and natural language descriptions is long- standing pursuit with significant implications for system design, verification, requirements engi- neering, and human-robot interaction (Wing) 1990). This section reviews prior work relevant to the VERIFY dataset, focusing on approaches for translating between Linear Temporal Logic (LTL) and natural language (NL), the landscape of existing datasets, and the limitations that motivated VERIFY'S s development.
Translating Between LTL and Natural Language: Translating formal specifications like LTL into understandable natural language, and vice-versa, has been approached using various methods, ranging from rule-based systems to modern deep learning techniques. Early work on L.TL-to-NL translation relied on attribute grammars and heuristics applied to the formula's parse tree (Brunello al. 2019). However, achieving truly fluent, context-aware, and unambiguous translations that avoid common human misinterpretations (e.g., regarding temporal operators like "Until" and "Weak Until" (Greenman et al. 2022 ) using purely rule-based methods remains an open challenge (Brunello
et al. 2019). Translating NL to LTL using rule-based methods often involves complex semantic parsing pipelines, which can be incredibly brittle and difficult to scale (Kushman and Barzilay 2013). With the rise of deep learning. Neural Machine Translation (NMT) approaches have been applied to translatoo. A seminal effort by Cherukuri et al. (Cherukuri et al.) (022) demonstrated the feasibility of using OpenNMT (Klein et al. 2017)) to translate LTL formulas into English explanations, achieving high BLEU scores on a dataset augmented with variable permutations. This highlighted the potential of data-driven methods but also their dependence on sufficiently large and representative paired corpora. Similar sequence-to- to-nee-----eeeccee models have been explored for NL to LTL tasks, often
framing it translation problem (Wang) 2020 Vaezipoor et al. (021).
More recently, Large Language Models (LLMs) have shown significant promise. Pan et al. Pan et al. 2023) employs GPT-3 for paraphrasing structured English templates (derived from LTL. via
2

---

Published as a conference paper at ICLR 2026
rules/templates) to synthesize diverse NL commands for training NL LTL models data data-efficiently The Lang2LTL work also utilizes LLMs within its translation framework (Liu et al. 2023). The NL2TL project (Chen et al. 2023) uses GPT-3 to generate large dataset of "litted" NL-Tempora Logic pairs where specific details are abstracted away and fine-tunes '5 models, demonstrating LLMs' potential for both data generation and translation, partieularly when aiming for cross-domain generalization via lifted representations. Tooling efforts like the NL2LTL Python package also integrate LLMs (GPT) alongside traditional NLU engines (Rasa) translate NL into predefined LTL patterns. These works showcase the power of LLMs but often still rely on intermediate structures, specific patterns, or focus on lifted representations rather than fully contextual, grounded NL across truly diverse domains. Furthermore, work like Greenman et al.'s (Greenman et al. (022) reminds us that effective translation requires more than semantic equivalence; it demands alignment with human cognitive patterns and expectations. Their user studies revealed systematic misunderstandings when humans map LTL to English, emphasizing that automated translation must produce outputs that align with both formal semantics and user expectations to be truly effective for explanation or requirements validation.
Despite progress in translation methodologies, major bottleneck that remains is the availability of large, diverse, and suitable datasets. While several resources have been created, a review reveals significant limitations, especially concerning scale, domain diversity, contextual richness, and accessibility. A significant portion of publicly available datasets is confined to narrow application domains, primarily robotics and navigation. Examples include the datasets from Pan et al. (Pan et al. 2023). Wang et al. (Wang et al. (021)) the (Berg et al. 2020), and the grounded parts of Lang (iuu et al.] 2023). While valuable within their specific contexts, these resources lack the linguistic and conceptual diversity needed to train models that generalize beyond command and---nnoooooolo scenarios. Other datasets operate symbolic level, using abstract variable names (Cherukuri et al. 2022: Liu al. 2023), focus on specialized areas like hardware verification (Hahn et al.] 2022). While the NL2TL dataset (Chen et al.) 2023) attempts cross-domain generalization using lifted representations, it differs from providing specific, contextual groundings across varied domains.
Existing Datasets and Resources:
Borpuuugg-t--Laadrrrr work
Furthermore, many existing datasets are limited in scale, often containing only few thousand (Wang et al. 2021 Berg et al., 2020; Liu et al. 2023) or low tens of thousands (Cherukuri et al. 2022 Chen et al. 2023) of examples. This scale is often insufficient train large neural models capable of capturing the complex interplay between logical structure and linguistic variation and serves for light finetuning. Perhaps most critically, the nature of the natural language presented is often restricted. Many datasets feature imperative commands or relatively technical descriptions that closely mirror the underlying logic (Pan al. 2023 Kushman and Barzilay, 2013). rather than the richer, more descriptive, and context-dependent language typically found in real-world requirements documents or system descriptions (Beltagy et al. 2020).
Finally, accessing and utilizing these resources can be challenging due to their fragmented nature, originating from different research groups with varying formats and objectives. So, despite valuable contributions within specific niches, large-scale, multi-domain dataset featuring rich, contextual NL paired with LTL has been conspicuously absent.
Where Does VERIFY Fit In: VERIFY addresses these combined limitations. Its scale (200k+ triplets), domain breadth (13 areas), contextual NL grounding, and the novel ITL intermediate layer collectively distinguish it from all prior resources. Table summarizes the comparison.
3
THE VERIFY DATASET
This section describes VERIFY's structure, its three-layer representation framework, and key dataset statistics.
3.1 CONCEPTUAL FRAMEWORK: UNIFYING LTI, ITL, AND CONTEXTUAL NL
VERIFY is built upon three-layer representation; Linear Temporal Logic (LTL), Intermediate Technical Language (ITL), and Natural Language (NL), all grounded within specific application domains and contexts.
Linear Temporal Logic (LTL): At the core, LTL serves as the formal specification language (Pnueli 1977). It allows precise expression of properties over time using propositional variables, standard
3

---

Published as a conference paper at ICLR 2026
Table Comparison of VERIFY with existing LTL-NL datasets. "Conetts."CCCCiii "Contextual" NL refers descrip- tions grounded in domain-specifer variable definitions, as opposed to generic templates or imperative commands, A dash indicates the property is absent or not applicable.
Dataset
Scale
Domains
NI Type
Intermediate Repr.
Formal Verification
Public
Cherukuri al. Chhrukuri et al. 2022
~12
Abstract/ / symbolic
Technical
Partial
al. (Pan aCa
Robotics
Commands
Yes
Wang et al.
Wang el ai 2021
~2k
Robotics
Commands
Yes
Berg al.
(Regg ef al.
~1.5k
Navigation
Commands
Yes
Lang21.TL
Lai el
20243
2.
Robotics navigation
Commands
Yes
N1.2TI. (Chen
2056
~28k
Lifted (eross-domain)
Lifted templates
Yes
Haho et dl. (Hähin et al.
2022
~100
Hardware verification
Technical
Yes
VERIFY (ours)
200k+
13 diverse
Contextual
ITL.
< (Spot)
Yes
boolean operators (7, A, -+), and temporal modal operators. VERIFY utilizes the standard LTL operators: G ("Globally" 'Always'), F ("Finally' 'Eventually"), X ('Next'), U ("Until'), W ('Weak Until"), and R ('Release"). For instance, G(req + F(ack)) formally states that it is always the case that if request req is sent, then an acknowledgment ack must be sent back at some point in the future. This layer provides the unambiguous, machine-verifiabl meaning.
The Intermediate Technical Language (ITL): A key challenge in this whole research area is the sigmificant semantic gap between the abstract, symbolic nature of LTL and the rich, nuanced, context-dependent nature of real-world natural language. Directly mapping complex L.TI. formulas to fluent, accurate, and contextual NL is extremely difficult. This is because translations often become overly technical, template-like, or lose semantic fidelity. To address this, we introduce ITL, a novel intermediate representation designed specifically to serve LTL and NL. ITL is conceived to be more structured and less ambiguous than free-form NL, yet more numan-readable and linguistically closer to NL than raw LTL formulas. ITL preserves the logical structure of the LTL formula's abstract syntax tree while replacing symbolic operators with controlled English keywords drawn from curated library of human-readable temporal expressions. This intermediate target simplifies the translation task in both directions (LTL - NL and NI. -> LTL). Given the LTL formula: Gsssttmm censtem_rredyy ready- + (check a U check b)). . An ITL representation might be: Always(II [IF gystem_readt THEN (check_e Until check b_) This ITL form preserves the exact logical structure (always, implies, until) but uses more verbose, keyword-like operators, making the transition to or from NL more manageable than directly handling the symbolic LTL.
structural and semantic bridge between
Domain and Context: While LTL provides formal meaning and ITL offers structural bridge, generating truly relevant NL requires grounding in specific application context. An LTL formula like G(p + Fq) abstract; its meaningful NI. translation depends entirely on what p and q represent in given scenario. VERIFY incorporates this crucial grounding through two key fields associated with each triplet: (i) domain: Specifies the application area (c.g., 'Financial Services', 'Home Automation') ) and (ii) activity: Provides natural language definitions for the propositional variables used in the LTL formula within that domain's context (e.g.. p = user login attempt succeeds, q = two-factor authentication prompt is displayed).
This domain and activity information provides the essential semantic context, enabling the generation and interpretation of NL descriptions that are not generic templates but are instead specific, relevant, and interpretable within their intended domain. For instance, grounded in a financial domain, the ITL above might translate to: "It must always be the case that if the trading system reports ready, then check A must remain valid until check B is completed."
3.2 DATASET DESIGN AND STRUCTURE
The creation of VERIFY was guided by several core principles aimed at producing high-quality, impactful resource for the research community. We aimed for Logical Diversity, ensuring the dataset includes a wide spectrum of LTL formulas, varying in structure, operator usage, and nesting depth. Contextual Richness was a primary design goal; driving the generation NL that is deeply specific to the domain and variable definitions, avoiding vague purely syntactic translations. Broad Domain Coverage across 13 distinct areas was incorporated to facilitate research into domain generalization and adaptation. Verifiability and Quality were central, addressed through formal verification of
4

---

Published as a conference paper at ICLR 2026
LTL formulas, provably correct LTL-tt-ITL mapping, and stage validation of NL alignment (detailed in Section 4). Finally, Scalability was a key goal, resulting a large-scale dataset suitable for training modern deep learning models.
Data Schema:
The dataset structured as a collection of records, where each record represents complete LTL-ITLL-N. triplet with its associated context and metadata. The primary fields are described in T55
Domain Coverage: VERIFY spans 13 distinct application domains, selected to cover wide range of scenarios where formal specification and natural language descriptions interact. The domains are listed in Table [
3.3 DATASET STATISTICS
VERIFY is large-scale resource comprising over 200 thousand LTL-ITL-NL triplets. This includes substantial number of unique LTL formulas and ITL structures, reflecting diverse logical patterss. Due to the contextual generation process, the vast majority of the 200k+ NL translations are unique or near-unique within their specific domain context. The dataset features broad distribution of LTL formula complexities, ranging from simple properties involving one or two operators to complex specifications with significant nesting depths. The Natural Language descriptions also exhibit variety. Sentence lengths vary considerably depending on the complexity of the underlying logic and the specific domain context. The sample distribution based on the count of temporal operators per formula and complexity of the underlying logic and the specific domain context is shown in Appendix A We also show the sample distribution across the specific domains in the Appendix A
DATASET CONSTRUCTION METHODOLOGY
The creation of the VERIFY dataset involved rigorous, multi-stage pipeline designed to create high-quality data encompassing LTL,, ITT,, and contextual NL.. This process emphasized formal correctness, semantic consistency, contextual relevance, and scalability, incorporating automated generation, formal verification, LLM capabilities, and comprehensive quality assurance steps.
4.1
LTL FORMULA GENERATION AND VERIFICATION
The foundation of VERIFY lies in a diverse set of syntactically correct and semantically meaningful
LTL formulas.
We employed programmatic LTL formula enumerator that recursively constructs formulas up to specified maximum depth (depth 25 in our process) using standard LTL operators (G, F, X, U, R, W) and boolean connectives, applied to set of atomic propositions (p through w). The generation depth of 25 refers to the maximum recursion depth of the formula enumerator, which controls the size of the search space. The AST depth of formulas retained after verification and canonicalization is substantially lower (mean 5.0; the per-domain statistics in Table 14], because Spot's canonicalization simplifies many deeply nested structures into equivalent shorter forms. Random choices at each step ensure structural diversity in the generated formulas. To manage the vast number of potential formulas and avoid trivial duplicates, generated formulas undergo structural canonicalization process. This involves conversion Negation Normal Form (NNF), expansion of pppiications/qquivaleneesss application of associative/distribii commutative operators. A canonical hash is computed for each unique structure. Formulas are stored in an SQLite database indexed by this hash, so that only structurally distinct formulas are retained.
laws, and sorting of operands for
The primary step ensuring the semantic validity and triviality of the LTL formulas involves rigorous verification using Spot [Duret-Lutz and Poitrenaud] 2004). This critical step filters out syntactically invalid formulas that might arise from the generator or initial conversion. Successfully validated formulas, along with their canonical string representation as determined by Spot are stored back into the database
and canonical form columns). This ensures that all LTL formulas used in subsequent stages are well-formed and provides standardized representation grounded a formal verification tool.
4.2 INTERMEDIATE TECHNICAL LANGUAGE (ITL) GENERATION AND VERIFICATION
The generation of ITL from verified LTL formulas is a deterministic, rule- based process. Verified LTL formulas are first parsed into Abstract Syntax Tree (AST) representation using Spot's parsing
5

---

Published as a conference paper at ICLR 2026
capabiliiiis. This captures the precise logical and temporal structure. An AST-visitor script then walks the LTL parse tree and, at each operator node, performs an O(1) dictionary lookup of a curated list of numan-readable templates. These templates are derived from curated grammar based on common human-readable expressions for temporal concepts. For example, G(p) maps to Always and pUg maps p Until q. This recursive process yields canonical ITL. string that directly mirrrrs the LTL structure but uses more naturalistic keywords.
To ensure the integrity of the ITL generation process and the semantic equivalence between the canonical ITL and its source LTL, an automated verification step was implemented. This involves parsing the ITL text back into an LTL formula representation using a rule-based parser guided by the ITL grammar via an AST. This reconstructed LTL formula is then formally compared against the original L.TI. using Spot's built-in semantic equivalence checker. This check confirms that the ITL representation, when interpreted back through its grammar rules, retains the precise logical meaning of the source LTT.. Because the ITL. generation strictly follows the L.TL AST structure and uses defined mapping for each LTL operator, the transformation from LTL to the canonical ITL output is deterministic and structure- preseevin......iiggg This deterministic mapping forms the basis of the provably complete relationship between the source LTL and the canonical ITL representation. We prove this relationship in Appendix C
ITL Verification:
4.3 CONTEXTUAL NATURAL LANGUAGE (NL) GENERATION
To generate natural language descriptions relevant to specific application areas, we utilized large language model guided by the LTL formula, its ITL represcntation, and domain context. We used DeepSeek-R1 (Liu et al. 2024 Guo et al., 2025) for NI. generation. For each L.TIITLL pair selected from the database, a target domain was chosen using a probabilistic sampling strategy designed to balance the distribution across the 13 domains. A prompt (detailed in Appendix E.3) was constructed, instructing the LLM to act as an expert in formal methods and the target domain. The prompt provided the LTL formula and the ITL representation and explicitly requested the model generate two components within specific tags: <activity>: A natural language description defining the meaning of the atomic propositions within the context of the selected domain. <translation>: A clear, concise, and semantically accurate natural language translation of the LTL/ITL logic, incorporating the domain context provided in the <activity> tag-
4.4 QUALITY ASSURANCE AND VALIDATION
Quality assurance involved multiple stages targeting different components of each triplet. As described above, LTL. formula validity is enforced through parsing and canonicalization using the Spot library. The canonical ITL is generated via a deterministic, structure-preserving mapping from the verified LTL, ensuring structural fidelity by construction. Even then, the consistency between ITL and the original LTL was further verified using an ITL-to-LTL parser and bpst- equivalence checking.
Manual NL Check: A significant manual review conducted on the completed dataset. We randomly sampled 10,000 LTL-ITL-NL triplets and reviewed them. The review focused on: (i) Semantic Equivalence: Does the NI. translation accurately convey the precise meaning of the LTL/ITL formula, especially the temporal relationships? (ii) Contextual Relevance: Is the activity description plausible for the domain, and is the translation consistent with this context? and (iii) Linguistic Quality: Is the NL translation fluent, grammatically correct, and easily understandable? This manual check identified very low error rate (<1%), primarily consisting of minor fluency issues or occasional subtle deviations in temporal meaning, which were used to refine prompts and generation strategies iteratively.
LLM Judge (NL):
To augment manual checks and provide broader validation coverage, we employed
an automated L.Le-based judge. We utilized Llama 3.3 70B Instruct (Grattafiori et al. 2024) to evaluate a random sample making up total of 18% of the generated NI. translations. The LLM
judge was presented with the LTL formula, ITL text, and the NL translation. was prompted to assess semantic precision (especially regarding temporal operators), contextual appropriateness, and fluency, outputting a structured JSON response containing: is "s_crreeet (boolean), score (0-10 integer rating), issues (a list of identified problems), and textual reasoning for its judgment. The results from the LLM judge indicated an estimated >97% semantic correctness and consistency between the NL translations and their corresponding LTL/ITL specifications, aligning closely with the findings from

---

Published as a conference paper at ICLR 2026
Table 2: Performance on LTL/ITL-1ONL translation tasks (BERTScore FI ROUGE-L FI).
Model
LTL NL
ITL + NL
T5-base
0.62 / 0.37
0.84 / 0,41
T5-Iarge
0,67 / 00.4
0.69
BARTTlarge
0.63 / 0.39
0.78/
Mlama-3-B-IInstre (FT)
0.91 / 0.67
0.94/
Mistral-7B Hiistra-7B-Inntrucee IB-Instruct (FT)
0.88 / 0.62
0.91 0.662
(FT)
0.88 / 0.63
0.92/
the manual verification phase. We note that because both the generator (DeepSeek-R)) and the judge (Llama 3.3) are LLMs, errors that one model makes may be systematically missed by the other.
5 BENCHMARK TASKS AND EXPERIMENTS
To demonstrate the utility of VERIFY and establish baseline performance levels for future research, we conducted a set of experiments evaluate oftte-o--thh-ar- models on core translation tasks enabled by VERIFY's LTL-ITL-NL structure and probe the challenges introduced by its contextual richness, domain diversity, and logical complexity.
5.1 EXPERIMENTAL SETUP
We created standardized train, validation, and test splits for the VERIFY dataset, maintaining an approximate 80%/10%/100 ratio. These splits were stratified by domain ensure representation across all 13 areas in each set.
We evaluated diverse range of models to provide broad performance landscape; (i) Pre-trained Lequence-to-Seqqenee Models: Standard Transformer- based models, specifically T5 (t5- t5-laarge) (Raffel et al.] (2020) and BART (bart baar, (arr-large) (Lewis) ct al, 2019), were fine-tuned for each task, (ii) Instruction-
Baseline Models:
LLMs: We fine-tuned prominent
cestruction-folloiing models, including Llama 3 (Llama- 3 3-8B---- Ins ructt) (Grattafiori al.) 2024) and Mistral (Mistrrl-20 Satttaaa----nntueeeee Instruct- -v0 2) (Jiang et al. 2023), to assess the capabilities of modern LLMs on these structured tasks, and (iii) Code-Focused LLMs: Models pre-trained extensively on code, such as CodeLlama Batteeiaaa-7b--Insstt
(Rozière et al.) 2024
and DeepSeek Coder (deepseek cooerr 6. 7b instruct) (Guo et al., (024), were included, particularly for tasks involving generation of formal LTL/ITL outputs. All models were fine-tuned using standard hyperparameters optimized on the validation set (details in Appendix A).
Evaluation Metrics: We employed a suite of metrics appropriate for the different translation
directions: (i) NL Generation (LTL/ITL + NL): Primary metrics were BERTScore (Zhang et al. 2019) (for semantic similarity using DeBERTa-v3 -a-raae and ROUGE-L LLin 2004) (for lexical overlap) and (ii) Logic Generation (NL - LTL/ITL, LTL ++ ITL): Primary metrics were task- dependent: Semantic Equivalence (for NL LTL and ITL - LTL, using Spot check logical equivalence with the ground truth) and Exact Match (EM) (especially for NL - ITL and LTL - ITL). Secondary metries included Tree Edit Distance (TED) measure structural similarity, and Syntactic Correctness (percentage of outputs parsable according to the LTL/ITL grammar).
5.2 CORE TRANSLATION TASK PERFORMANCE
Tasks 1 & (LTL/ITL + NL): Generating contextual natural language from formal (LTL) or intermediate (ITL) representations, given domain and activity context. Results (Table show that modern pre-trained models achieve reasonable performance, with L.MMs generally outperforming T5/BART, particularly on semantic metrics like BERTScore. Generating NL from ITL yields comparable or slightly better results than from LTL directly for most models, suggesting ITL can be an effective input representation. Tasks 3 & 4 (NL - LTL/TT); As expected, these tasks proved significantly more challenging (Table 3). Semantic Equivalence for NL-LTL remains low across models, highlighting the difficulty of precise logical form recovery from ambiguous NL. We also do recognize that semantic equivalence as measured by Spot is a strict binary criterion: two formulas that differ by single operator score even if they are otherwise structurally identical. The low absolute
7

---

Published as a conference paper at ICLR 2026
numbers in Table 2 therefore reflect both genuine model failures and the severity of the metric. Future work might supplement semantic equivalence with partial crediit measures such as tree edit distance on the formula ASTs better characterize the distribution of errors. Code-focused LLMs showed slight advantage in generating syntactically correct outputs. Exact Match for NI-ITTL was higher than for L.TI,, potentially due to ITL's more constrained structure, but still far from perfect.
Table 3: Performance on NL-to-LTLATT translation tasks (Semantic Equiv. EM Syntactic Correctness.)
Model
NL + LTL (SemEq EM SynCor)
NL + ITL (EM/ SynCorr)
T5-large
22.3 / 2.8 66.1
2.2 11.8 68.3
Llama-3-8B-Bnn --nna------nnntteee (FT)
28.2 / 4.1 73.6
2.3 77.2
Mistral-7B-Instruet (FT)
25.6 2.9 68.4
1.6 17.9 74.5
Jama-lll--bb--nttru (FT)
/.34 71.1
3.2/ 19.2 74.8
DeepScek-Cood -odeer (FT)
31.5 5
4.1 / 18.8 79.5
Task 5 (LTTL ++ ITL):: Translating directly between the formal L.TL. (Spot canonical form) and the canonical ITL. Given the deterministic rule-based mapping used for canonical ITL generation, models achieved Exact Match scores up to 31.7% on LTL-ITL (Table 4. The ITL-LTL direction also showed high Semantic Equivalence (up to 56.4%) and corresponding Exact Match scores (up to 21.6%), confirming models can effectively learn the structural correspondence, although minor syntactic variations occasionally occurred.
Table 4: Performance on LTL+ITL translation tasks (Exact Match Semantic Equiv.)
Model
LTL ITL (EM)
ITL -> LTL (SemEq EM)
TS-large
19.3
38.6 19.0
Mlama-3-8B-Insterre (FT)
31.7
53.1 20.8
CodeLlama-77 7b-Instruct (FT)
27.9
56.4 21.6
Error Analysis. To understand why models fail at NL-LTL translation, we manually reviewed 100 incorrect predictions from the Llama-3-8B Carmmaa--BB-Inttttee Instrucc (FT) model on Task 3. Each prediction was classified into one five error categories based on the primary failure mode. [ableB]ssmmarizz the results.
Table 5: Error analysis of 100 incorrect NL-LTTL predictions from Llama-3-8B- 3-8tt-nstrtuttt (FT). Categories mutually exclusive; each prediction is assigned to its primary failure mode.
Error Category
Freq.
Description
Incorrect Logical Scope
41%
Misplaced parentheses or incorrect nesting of operator scope.
Temporal Operator Mismatch
28%
Confusion between semantically close operators, most often U/W or G/F.
Propositional Atom Error
17%
Missing or hallucinated propositional atoms relative to the activity context.
Contextual Grounding Failure
9%
Logically valid LTL that does not match the variable definitions in the activity field.
Syntactic Malformation
5%
Output cannot be parsed as a well-formed LTL for- mula.
Syntactic errors account for only 5% failures, indicating that fine-tuned LLMs reliably learn the surface grammar of LTL. The dominant failure mode is incorrect logical scope (41%), where the model produces a parseable formula whose operator nesting does not match the intended reading of the source sentence; for instance, generating (G(p 4)) U instead of G(p (q U r)).
8

---

Published as a conference paper at ICLR 2026
Temporal operator mismatch (28%) is the second most common failure: the model frequently substitutes Until (U) for Weak Until (W), conflating a liveness requirement with a safety property, a distinction that English "until" leaves ambiguous but that carries significant formal consequences. Propositional atom errors (17%) and contextual grounding failures (9%) together suggest the model loses track of the mapping between domain-specifie variable names and abstract atoms, consistent with the sharp performance drop observed when context is removed (Table 6). A fuller discussion appears in Appendix B.3
5.3 ANALYTICAL EXPERIMENTS
We performed further experiments to analyze the influence of VERIFY's specific design choices and characteristics.
Experiment A (Value of ITL:: We investigated the potential of ITL as an effective intermediate representation for logic-to-NL generation. This was assessed by comparing the performance of direct LTL + NL translation (Task 1) with the performance of ITL + NL translation (Task 2), which represents the second stage of potential two-stage pipeline (LTTL ITI. via Task 5 model, then -> NL via Task 2 model). The results, illustrated in Figure 1 demonstrate that models consistently achieve higher performance when translating ITL NL compared to translating LTL to NL. Specifically, for all evaluated models, both BERTScore FI and ROUGE-L Fl scores are notably improved when ITL the source language for NL generation as opposed to LTL. For instance, Llama-3-8B Instruct (FT) achieves ScRTS F1 of 0.91 for LTL - NL, which increases to 0.94 for ITL - NL; similarly, its ROUGE-L F1 improves from 0.67 to 0.73. T5-large shows an even more pronounced relative improvement in BERTScore F1, jumping from 0.67 (LTI. - NL) to 0.89 (ITI. + NL.. These findings support the hypothesis that ITTI. can serve as a beneficial intermediate representation, as translating from ITL to NL yields significantly better semantic accuracy and lexical overlap than direct translation from LTL to NL across a range of models.
1.2
0.22
0.M
0.18
0.1
0.2
0.63
0.6
0.1-
6.2
0.2
T020mm
en
1T BEA Noom P1
In -1 Rooee 1.
mi-ttt am Soem
Fll Ml ROW.O
Model
Figure 1: Comparison of direct LTL.-t-NL translation versus a two-stage LT----o-ILL---- pipeline, showing difference in BERTScore vs. L.T. complexity.
Experiment B (Impact of Context): Models trained for NL + LTL/ITL translation without the domain and activity context information suffered a significant performance degradation compared to models trained with full context (Table Semantic scores dropped considerably and Syntactic Correctness was much lower, confirming the importance of contextual grounding for generating meaningful and accurate translations. Note that Semantic Equivalence for NL-ITL in Table / is computed by round-tripping the predicted ITL. through the ITL--toLTT. parser and checking logical equivalence against the ground-truth LTL via Spot; this metric is substantially higher than Exact Match because many syntactically distinct ITL strings map to semantically equivalent LTL formulas. Our experiments establish initial baselines on the VERIFY dataset, demonstrating the capabilities and limitations of current models on contextual achieve strong performance LTL/ITL- - NL generation and foter-formalimm translation (LTL < ITL), significant challenges remain, particularly in parsing NL to accurate LTL specifications (NL LTL.) and generalizing across diverse domains. The results confirm the importance of contextual
tasks. While modern pre- pre-trained models

---

Published as a conference paper at ICLR 2026
Table 6: Impact of domain and activity context on NL-to-LTL/ITL translation performance (Semantic
Equiv. / EM).
Llama FT
With Context
Without Context
NL LTL
28.2 / 4.1
7.7 0.8 13.9 1.3
NL ITL
41.5 / /..3
information, the potential utility ITL for complex formulas, and VERIFY's effectiveness in representing wide spectrum of logical and domain-basse difficulties suitable for driving future research.
Additional Experiments. We conducted additional experiments to support the need for such dataset; namely, per-domain performance analysis, generalization which was tested by holding out individual domains during training and by evaluating the model entirely unseen formal logic (STL). Finally, manual error analysis was performed on incorrect NL to LTL translations to categorize common failure patterns B
CONCLUSION
Our experiments establish baselines on VERIFY that reveal asymmetry in current model capa- bilities. Generating fluent NL from LTL or ITL is achievable with fine-tuned LLMs, which reach BERTScore Fl above 0.91. The reverse direction remains far harder: the best model achieves only 31.5% semantic equivalence on NL-LLT, with error analysis (Appendix B.3) attributing 41 of failures incorreet operator scope and 28% to temporal operator confusion. These error patterns suggest that unconstrained autoregressive decoding is poorly suited to producing well-scoped logical formulas from natural language, and that structured decoding methods neurosymbolic architectures may be needed.
The ITT. layer provides measurable benefit. Models translating ITL to NL consistently outperform those translating LTL to NL, with T5-large showing the largest relative gain (BERTScore Fl from 0.67 to 0.89). This supports the use of intermediate representations as general strategy for bridging formal and informal languages, an idea with implications beyond LTL. VERIFY is publicly available under CC BY 4.0 at the repositories listed on the first page.
REFERENCES
Jacob Andreas, Andreas Vlachos, and Stephen Clark. Semantic parsing machine translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 47-52, 2013.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint ww.iv:2004..555/5 2020.
Matthew Berg, Deniz Bayazit, Rebecca Mathew, Ariel Rotter-Aboyoun, Ellie Pavlick, and Stefanie Tellex. Grounding language landmarks in arbitrary outdoor environments. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 208-215. IERE, 2020.
Daniel M Berry, Erik Kamsties, and Michael M Krieger. From contract drafting to software specifica- tion: Linguistic sources of ambiguity, handbook. Perspectives Software Requirements, Series: The Springer International Series in Engineering and Computer Science, 753, 2003.
Barry W Boehm. Software engineering economics. Springer, 2002.
Andrea Brunello, Angelo Montanari, and Mark Reynolds. Synthesis Itl formulas from natural language texts: State of the art and research directions. In 26th International symposium temporal representation and reasoning (TIME 2019), pages 17-1. Schloss Dagstuhl-Leine Zentrum für Informatik, 2019.
10

---

Published as a conference paper at ICLR 2026
Yongchao Chen, Rujul Gandhi, Yang Zhang, and Chuchu Fan. N12t:: Transforming natural languages to temporal logics using large language models. arXiv preprint arXi:2305.00766, 2023.
Himaja Cherukuri, Alessio Ferrari, and Paola Spoletini. Towards explainable formal methods: From Itl natural language with neural machine translation. International Working Conference on Requirements Engineering: Foundation for Software Quality, pages 79-86. Springer, 2022.
Edmund M Clarke. Model checking. In Foundations of Software Technology and Theoretical Computer Science: 17th Conference Kharagpur, India, December 18-20, 1997 Proceedings 17, pages 54-56. Springer, 1997.
Edmund M Clarke, Thomas A Henzinger, Helmut Veith, Roderick Bloem, et Handbook of model checking, volume 10. Springer, 2018.
Alexandre Duret-Lutz and Denis Poitrenaud. Spot: an extensible model checking library using cransition-basee generalized bu/spl uml/chi automata. In The IEEE Computer Society's 's 12th Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecom- munications Systems,
2004). Proceedings., pages 76-83. IEEE, 2004.
Matthew B Dwyer, George Avrunin, and James C Corbett. Patterns in property specifications for finite-state verification. In Proceedings of the 21st international conference on Software engineering, pages 41 1-420, 1999.
Steve Easterbrook, Robyn Lutz, Richard Covington, John Kelly, Yoko Ampo, and David Hamilton. Experiences using lightweight formal methods for requirements modeling. IEEE Transactions Software Engineering, 24(1):4-14, 1998.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et The llama herd of models. arXiv preprint arXiv:2407.2178. 2024.
Ben Greenman, Sam Saarinen, Tim Nelson, and Shriram Krishnamurthi. Little tricky logic: miscon- ceptions in the understanding of Itl. arXiv preprint arXiv:221 2022.
Katharina Großer, Volker Riediger, and Jan Jürjens. Requirements document relations: A reuse perspective on traceability through standards. Software and Systems Modeling. 21(6):1-37, 2022.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming - the rise of code intelligence, 2024. URL
https::/arxiv. org/ /tt.///41.11199
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprins arXiv:2501.11245 2025.
Christopher Hahn, Frederik Schmitt, Julia J Tillman, Niklas Metzger, Julian Siber, and Bernd Finkbeiner. Formal specifications from natural language. arXiv preprint arXiv:2206.01962, 2022.
Gerard Holzmann. The SPIN model checker: Primer and reference manual, volume 1003. Addison- Wesley Reading, 2004.
Yih-Feng Hwang and David Rine. Verifying the reusability software component specifications: Framework and algorithms. Information Sciences, 112(1-4):19--197 1998.
Albert Q. Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Ana Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL ht //arxiv
org/abs/ /aabo//23110. 06825
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint 2xXi:2446.005/5. 2024.
11

---

Published as a conference paper at ICLR 2026
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810. 2017.
John C Knnght. Safety critical systems: challenges and directions. In Proceedings of the 24th international conference on software engineering, pages 547-550, 2002.
Nate Kushman and Regina Barzilay. Using semantic unification to generate regular expressions from natural language. North American Chapter the Association for Computational Linguistics (NAACL), 2013.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-tu cocenncett-sspee to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.1344 2019.
Chin Yew Lin. Rouge: A package for automatic evaluation of summaries. Text summarization branches out,, pages 74-81, 2004.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprint arXiv:2412.19437. 2024.
Jason Xinyu Liu, Ziyi Yang, Ifrah Idrees, Sam Liang, Benjamin Schornstein, Stefanie Tellex, and Ankit Shah. Grounding complex natural language commands for temporal tasks in unseen environments. In Conference Robot Learning, pages 1084-1110. PMLR, 2023.
Maxim Vyacheslavovich Neyzov and Egor Vladimirovich Kuzmin, Ltl-specification for development and verification of control programs. Modeling and Analysis of Information Systems, 30(4): 308-339. 2023.
Jiyy Pan, Glen Chou, and Dmitry Berenson. Data-efficient learning of natural language to linear temporal logic translators for robot task specification. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages
IEEE, 2023.
Amir Pnucli. The temporal logic of programs. In 18th annual symposium on foundations of computer science (sfes 1977), pages 46-57. iece, 1977.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Kristin Y Rozier. Linear temporal logic symbolic model checking. Computer Science Review, 5(2):
163-203, 2011.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2024. URL https://arrx
htgg/bbs/2330 12950
Rosario Scalise, Shen Li, Henny Admoni, Stephanie Rosenthal, and Siddhartha S Srinivasa. Natural language instructions for human-roke robot collaborative manipulation. The International Journal of Robotics Research, 37(6):558-566 2018.
Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, and Seunghyeok Hong. Llm-as-a-judge & reward model: What they and cannot do. arXiv preprint 1xXh:22409.12239 2024.
Kelvin Tamakloe, Godfred Bonsu, Shravan K. Chaganti, Abalhassan Sheikh, and Degang Chen. Automated detection of site-to-site variations: A sample-efficient framework for distributed mea- surement networks. Eng, 6(11), 2025. ISSN 2673-4117. doi: 10.3390/eng61 10297. URL
https: :/ http://wwml/tommmom com /tt.cc///2231111////12233
12

---

Published as a conference paper at ICLR 2026
Pashootan Vaezipoor, Andrew C Li, Rodrigo A Toro Iearte, and Sheila A Mcilraith. Lt12action: Generalizing instructions for multi-task rl. In International Conference on Machine Learning, pages 10497-10508. PMLR, 2021.
Christopher Wang. Candace Ross, Yen-Ling Kuo, Boris Katz, and Andrei Barbu. Learning natural- language to Itl executable semantic parser for grounded robotics. In Conference on Robot Learning, pages 1706-1711 PMLR, 2021.
Christopher Wang. Weakly supervised semantic parsing for Linear Temporal Logic. PhD thesis, Massachusetts Institute of Technology, 2020.
Jeannette M Wing. A specifier's introduction formal methods. Computer, 23(9):8-22 1990.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Weinherger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint 1rX.v.900.00967,, 2019.
13

---

Published as a conference paper at ICLR 2026
A DATASET STATEMENT
This statement provides details regarding the curation, content, potential risks, and administration of the VERIFY dataset, following recommended guidelines for dataset documentation.
A.1 CURATION RATIONALE
VERIFY created to address the limitations of existing L.TI-NIL datasets described in Sections and 2. This appendix provides additional documentation following recommended dataset reporting guidelines. Figure 2 a) illustrates the distribution of VERIFY based on the count of temporal operators per formula. The Natural Language descriptions also exhibit variety. Sentence lengths vary considerably depending on the complexity of the underlying logic and the specific domain context, as shown in Figure (b). The overall NL vocabulary is extensive, reflecting the diverse terminology across the 13 domains. The sample distribution across the specific domains is shown in eec..
The goal is to provide foundational resource to accelerate research in robust logic-to-languan trans- lation, formally-grounder NL.,, domain adaptation for specifications, and human-centric verification, moving beyond niche applications or purely symbolic translations towards more realistic scenarios.
Distribution of LTL Formula Complexity in Vertly Dataset
Distrbbutton of Natural Language Sentence Lengths Vartfy Dataset
40000
17500
Maa 30
Mox: 20d
13000
Medium 9.0
35000
Misn: 54,90
Trequeecy
25000
10000
20000
7500
15000
5000
5000
10
15
25
30
25
00
1000
125
175
200
Number of Temporal Operators
Word Count
(a)
(b)
Distributton Samples Across Domatns in Versty Dataset
Akkozwaanneeeeteeeeeee Vitucle
66712
Medical Deskea
6.71
Goddtaneyy Munugetieal
Buse Pipllinss and CLCD
Home Automation
Security Auchantizziion
4,703
Wab Servicatrr
4,269
Indusrial Autcmartin Marreeettt
6.787
Amerpac
6,122
13k
Namber Surtples
(c)
Figure 2: (a) Distribution LTL formula complexity (temporal operator count) in VERIFY, showing coverage from simple to complex formulas (b) Distribution of Natural Language sentence lengths (by word count) in VERIFY (c) Distribution of samples across the 13 domains in VERIFY
A.2 LANGUAGE VARIETY
The dataset contains three primary language types:
LTL:
Standard Linear Temporal Logic formulas using common operators (G, X, U, R. W) and boolean connectives over atomic propositions (p-w). Formulas are stored in canonical representation derived from the Spot library. ITL: A structured, rule-based Intermediate Technical Language designed for this dataset, using English keywords and templates corresponding to LTL operators. NL: Natural I Language (English). The NI component generated using large language model
14

---

Published as a conference paper at ICLR 2026
(DeepSeek- prompted to produce context-specific descriptions based the LTL/IL structure and domain information. The vocabulary reflects the diversity of the 13 target domains (Table 13).
A.3 SPEAKER/ANNOTATOR DEMOGRAPHICS
These components were generated programmatically based on formal rules and algorithms. There were no human speakers annotators directly involved in their generation, beyond the initial design of the ITL grammar rules and templates. NL Generation: The natural language descriptions were generated entirely by the DeepSeek-RR R1 large language model. No human speakers were involved. Manual Validation (10k Sample): The manual check of 10,000 NI translations was performed by the paper's authors, all possessing graduate-level expertise in formal methods, temporal logic, and/or natural language processing. [All three are based at the same university]. No other demographic information was collected for this internal check. LLM Judge Validation (18% Sample): The automated validation used the Llama 3.3 70B Instruct model. No human annotators were involved in this specific validation step.
LTL/TL Generation:
A.4 POTENTIAL RISKS & BIASES
The primary risk stems from the use of LLMs for NL generation (DeepSeck-RI) and validation (Llama 3.3). While significant validation was performed, the generated NL may contain subtle stylistic biases, repetitive patterns, or occasional factual inconsistencies (particularly if the LLM struggled generating plausible activity descriptions) inherent in the foundation models used. The dataset might not fully capture the diversity and sometimes "ungrammatical" or ambiguous nature of truly human-generated requirements text. Note that the prompts used are E.3 Semantic Fidelity: Although validation estimated >97% semantic correctness, subtle errors in translating complex temporal nuances might exist in a small fraction of the NL examples. Users should be aware that models trained this data might inherit these subtle inaccuracies. Scope Limitations: The LTL formulas are generated up to a certain complexity (depth 25) and within a specific fragment; the dataset might not cover extremely complex esoteric LTL patterns found in some specialized verification domains. The 13 domains, while diverse, are not exhaustive. Misuse Potential: Models trained VERIFY could potentially be misused generate plausible-sounding but incorrect natural language descriptions of formal properties, or conversely, misleading "formal-looking" specifications from ambiguous text, potentially obfuscating errors in critical systems if deployed without due diligence. Mitigation: We employed multi-stage validation (Spot verification for LTL, rule- based generation and equivalence checks for ITT.,, extensive manual and L.-M-based checks for NI.) to minimize errors. The dataset, code, and methodology are released openly to allow scrutiny. We encourage responsible use and awareness of these limitations.
LLM Artifacts:
A.5 LIMITATIONS
We acknowledge several limitations inherent in VERIFY's current form. The natural language translations, while extensively validated, were primarily generated by LLM (DeepSeek-R)) consequently, they may reflect the stylistic biases or occasional artifacts characteristic of such models and might not encompass the full spectrum of human linguistic variation for expressing logical concepts. The scope of LTL formulas, while diverse, was generated programmatically up to a certain complexity threshold (depth 25), and may not cover all possible patterns found in highly specialized specifications. Similarly, while the 13 domains offer broad coverage, they are not exhaustive of all potential application areas. Finally, the dataset primarily contains at most three canonical ITL and two contextual NI. instance per L.T.. formula per domain context, limiting exploration of paraphrase diversity for now.
Despite these limitations, VERIFY opens numerous avenues for future research. Its scale and structure invite the development of novel model architectures specifically designed for formal logic translation, perhaps explicitly modeling the relationships between the three representations. The multi-domain nature makes it an ideal testbed for advancing few-shot and zero-shot domain adaptation techniques applied to formal specifications. Furthermore, the ITL layer could be investigated component in building more explainable Al systems for formal verification, potentially offering numan-readable justifications derived from formal proofs. Extensions to multi-lingual contexts, generating NI in
15

---

Published as a conference paper at ICLR 2026
languages other than English, represent another promising direction. The dataset could also inform the creation of more robust interactive tools for requirements elicitation and formalization.
The broader impact of this work lies in its potential to make formal methods more accessible and reliable. By facilitating better tools for translating between formal specifications and the natural language used by engineers, designers, and stakeholders, VERIFY can contribute improved requirements engineering, reduced ambiguity, and ultimately, safer and more dependable systems in critical areas like aerospace, medicine, and finance. However, ethical considerations remain. The reliance on LLMs (DeepSeek-RI for generation, Llama 3.3 for validation) means potential biases inherent these models could be reflected in the dataset, despite mitigation through validation. Researchers using VERIFY should be mindful of these potential biases. Furthermore, while intended to improve clarity, models trained on this data could potentially be misused to generate misleading "formal-looking requirements if not deployed responsibly. We encourage users to leverage the dataset's openness and rigorous validation framework for responsible innovation.
VERIFY addresses the long-standing challenge of grounding formal temporal logic in diverse, con- textual natural language at scale. By providing over 200 thousand verified LTL-ITL-NL triplets across 13 domains, generated through a rigorous methodology incorporating formal checks and extensive validation, VERIFY offers a unique and valuable resource. VERIFY provides the founda- tional resource to spur significant advancements in formally-groundee natural language processing, enhance the synergy between the formal methods and NL.P communities, and ultimately contribute to building more reliable and human-unded uuman-unnerstandable complex systems. We release VERIFY openly and encourage the research community to utilize and extend this dataset to push the boundaries of logic-aware language understanding and generation.
A.6 LICENSE
The VERIFY dataset is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. The accompanying code is released under the MIT license, Please consult the respective license files in the repository for full details.
A.7 MAINTENANCE PLAN
The VERIFY dataset will be hosted on Hugging Face Datasets, Kaggle Datasets and GitHub. We plan maintain the dataset by addressing issues (e.g., errors, inconsistencies) reported by the community via the GitHub repository's 's issue tracker or direct contact with the authors. Updates or corrections will be managed through versioning on the Hugging Face Hub. While long-term active development beyond initial corrections is not guaranteed, we aim to keep the resource accessible and address critical issues for least two years post-publication.
A.8 DATASET USAGE EXAMPLES
The dataset provided in standard CSV and Parquet formats for ease of use. Each record contains the LTL formula, canonical ITL,, domain, activity context, and NL translation, along with identifiers and metadata (see Table 15). Users can load the data using standard libraries like Pandas Hugging Face datasets. Example usage scripts and bascline model implementations are provided in the attached supplementary material but upon acceptance will be released the general public facilitate research on the tasks described in Section 5.
B ADDITIONAL EXPERIMENTS
B.1 PER-DOMAIN ANALYSIS OF THE DOMAINS
detailed per-domain analysis is necessary for a multi-domain dataset and we have conducted a set of extensive per-domain analysis. We evaluated the performance of our -lama-3-8B-Innstee (FT) model on key translation tasks for all 13 domains. The results, presented in the table below, showcase the performance variations and highlight domain-speciff challenges. Task 1 (LTL - NL) and Task 2 (ITL- > NL) evaluated with BERTScore FI. Task (NL - LTL) evaluated with Semantic Equivalence (SemEq %).
16

---

Published as a conference paper at ICLR 2026
Table Model Performance Across Various Domains
Task 1
Task 1 (ROUGE-L)
Task 2
Task 2
Domain
(BERTScore FI)
(BERTScore FI)
(ROUGE-L)
Aerospace
0.91
0.68
0.94
0.74
Automotive/Auttoommou Vehicles
0.92
0.69
0.95
0.75
Build Pipelines and CI/CD
0.90
0.66
0.93
0.72
Financial/Transactinn Systems
0.90
0,65
0.93
0.71
Home Automation
0,93
0.71
0.95
0.76
Industrial Automnation/Manufatturinn
0.92
0.68
0.94
0.73
Medical Devices
0.91
0.67
0.94
0.73
Networking/Distribteedd Systems
0.90
0.66
0.93
0.71
Robotics
0.92
0.70
0.95
0.75
Security and Authentication
0.91
0.68
0.94
0.72
Smart Grid/Energy Management
0,91
0,67
0.93
0.72
Version Control and Code Reviews
0.90
0.67
0.93
0.73
Web Services/APIs
0.91
0.68
0.94
0.74
Table 8: Model Performance for Task 3
Task 3
Domain
(SemEq (%)
Aerospace
27.5
Automotive/Autonomouss Vehicles
28.9
Build Pipelines and CI/CD
26.1
Financia//rransactoo Systems
25.8
Home Automation
30.1
Industrial AutomatiionMamntaatuiii
28.2
Medical Devices
27.9
Networking/Distributed Systems
26.5
Robotics
29.5
Security and Authentication
27.1
Smart Grid/Energy Management
27.3
Version Control and Code Reviews
26.8
Web Services/APIs
27.0
This analysis already reveals important trends. For generation tasks (Tasks I and 2), all domains achieve high BERTScore and ROUGE-L scores, with Home Automation showing slightly better performance. For the more challenging translation from NL to a formal representation (Task 3), performance varies more significantly. Home Automation again leads, achieving Semantic Equiva- lence of 30.1% 1% in the NL-+LTL task. In contrast, the Saanncia//Tranactiinn Systems domain, which often involves more abstract concepts and complex causal relationships, proves more difficult for the model, resulting in the lowest scores for semantic equivalence. This suggests that the abstract nature of domain's 's language directly impacts the difficulty of grounding in formal logic.
Furthermore, we analyzed how performance is affected by the logical complexity of the LTL formulas, using AST depth proxy. The table below shows that as the formula depth increases, model performance on the most challenging NL-1LLL task degrades noticeably.
Table 9: Semantic Equivalence by LTL Formula AST Depth
LTL Formula AST Depth
Semantic Equivalence (%)
1-4 (Low Complexity)
35.4
5-8 (Medium Complexity)
28.1 1
9-12 (High Complexity)
21.9
13+ (Very High Complexity)
15.2
17

---

Published as a conference paper at ICLR 2026
The specific LTL depths used to define the categories follows: Low Complexity (depths 1 1-4), Medium Complexity (5-8), High Complexity (9-12), and Very High Complexity (13+). For each of the four complexity categories, we randomly sampled 1,000 unique LTL-NL pairs from each category. creating dedicated evaluation set of 4,000 examples which we then used for the evals.
The table above shows that logical complexity is a primary driver of difficulty. The model maintains reasonable performance on formulas with low to medium complexity but struggles to preserve the precise semantic structure of more deeply nested LTL expressions when translating from natural language. This highlights a key area for future work: developing architectures that are more robust to increases in logical complexity.
B.2 GENERALIZATION TO UNSEEN DOMAINS
We tested the limits of our models in two distinct and ambitious ways: (1) generalization to unseen domains within the same LTL formalism and (2) emergent generalization to an entirely new, unseen formalism (Signal Temporal Logie).
First, to directly assess cross-doman generalization, we performed new set of experiments using a Leave-One-Dee Onee-Domain-O Out (LODO) cross-validation methodology. We trained our Llama-3-88 Instruuttttttt (FT) model on 12 of the 13 domains and tested its performance on the held-out domain. The results for three representative held-out domains presented in Table below. "In-Domain" refers to the original performance when the model was trained on all 13 domains. "Out-of-Domace is the performance on the domain when it held out from the training set.
Table 10: In-Domain vs.
Semantic Eguivalence
Held-Out Domain
In- Do-oomin SemEq (%)
Out-of-Domain SemEq (%)
Aerospace
27.5
19.2
Home Automation
30.1
22.5
Immancial/Transactioo Sys.
25.8
16.7
The LODO results show an expected decrease in performance when the model encounters domain has not been trained on. However, the model retains a significant portion its capability, achieving semantic equivalence scores between 16.7% and 22.5% in a zero-shot setting. This indicates that the model is not merely memorizing domain-specific patterns but successfully transferring learned logical structures
contexts, demonstrating solid degree of domain generalization.
Second, we conducted an experiment to investigate if the model, fine-tuned only on LTL, could show emergent generalization
different temporal logic. We tested the same VERIFY-fintunne Llama-3-8B model on a curated benchmark of 100 human-written Signal Temporal Logic (STL) specifications. STL is related but distinct formalism used for real-valued signals, which the model had never seen. The model was evaluated zero-shot, without any fine-tuning STL-specific data, The results are shown in the Table below. The human correctness ratings were provided by the paper authors, with a scoring of all 100 examples on 1-to-5 scale.
Table
Performance on Core Translation Tasks
Task
Metric
Performance
STL NL Generation
Hluman-rated Correctness (1-5)
3.7 / 5.0
NL + STL Translation
Semantic Equivalence (%)
14.3%
Remarkably, the model demonstrates non-trivial ability operate on STL specifications. It can generate coherent and largely correct natural language descriptions from STL formulas and can even parse NL into semantically valid STL with 14.3% accuracy. That the model achieves this capability without any exposure to STL suggests it has learned some of the fundamental, underlying principles of temporal logic that are common to both LTT. and STI., rather than just the surface syntax of LTL..
18

---

Published as a conference paper at ICLR 2026
B.3 COMMON ERROR PATTERNS
is important to look at the common error patterns in tasks 1-5. To do this, we performed an erro analysis the outputs of the Llama-3-8B model, focusing on the most challenging NL - LTL translation task. The results, based on a manual review of 100 incorrect predictions, are summarized in the Table below. Based on a manual review of 100 incorrect predictions.
Table 12: Error Analysis of NL to LTL Generation
Error Category Incorrect Logical Scope
Frequency 41%
Description
Model fails to correctly capture operator precedence and scope from the NL sentence, often misplacing parentheses nesting clauses incorrectly.
Temporal Operator Mismatch
28%
Model confuses semantically close temporal operators, most commonly substituting 'Until' (U) for "Weak Until' (W) or 'Globally' (G) for 'Finally' (F).
Propositional Atom Error
17%
Model either fails include a required propositional atom from the context or hallucinates an atom that was not specified.
Contextual Grounding Failure
9%
The generated LTL is logically sound but fails to correctly incorporate the specific variable definitions provided in the 'activity" context.
Syntactic Malformation
5%
The output is not a syntactically valid LTL formula and cannot be parsed.
Our analysis reveals that outright syntactic errors semantic in nature. The most frequent issue (41%) is the model's struggle to correctly capture the precedence and scope of operators from complex natural language sentences. Furthermore, the model often has difficulty distinguishing between strong and weak temporal requirements (e.g., 'Until' vs. 'Weak Until'), accounting for 28% of errors.
(5%). Instead, the majority of failures
C FULL LTL-TO-ITL COMPLETENESS PROOF
This appendix provides a formal argument for the completeness of the mapping from the Linear Temporal Logic (LTL) fragment used in the VERIFY dataset to the canonical Intermediate Technical Language (TTL.) representation generated by our pipeline. Completeness, in this context, means that every LTT. formula within the defined fragment can be successfully and deterministically translated into a well-defined ITL string.
C.1 SYNTAX OF THE SOURCE LTL FRAGMENT (LTL.rr)
The LTL formulas (ó, 1b) in the VERIFY dataset are generated and subsequently verified conform to specific syntactic fragment, denoted LTL.v.. Let AP {p, 1,, 8, u, v, w) be the finite set of atomic propositions used our dataset. The set well-formed formulas in LTLyp defined inductively as the smallest set satisfying the following rules:
1. Atomic Proposition: If E AP, then E LTLvF.
2. Boolean Constants: T (true) E LTLyp and 1 (false) E LTLvF.
3. Negation: & E LTLyr, then -Ø E LTLyE.
4. Conjunction: o, & E LTLvp, then (ò v)) LTLyF.
5. Disjunction: If Φ, & LTLyF, then (d V v)) E LTLvF.
6. Implication: If Φ, L.TL.rr, then (o -> v) E L.T.
7. Equivalence: If o, e LTLvp, then (o v) LTLvF.
8. Next: o LTLvE, then X6 LTLvF
9. Globally (Always): If à L.T.F, then Go LTL.F.
19

---

Published as a conference paper at ICLR 2026
10. Finally (Eventually): If LTLvH, then F LTLvF.
11. Until: If o, & LTLvr, then (&U v) E LTLvE.
12. Release: If o, v E LTLvE, then (ø R ()) E LTLyF
13. Weak Until: If σ, db LTLyr, then (ø W Φ) E LTL.F-
14. Strong Release (Matches): If db L.L.rr then (ø (MM)) LTL.r-
Standard operator precedence and parentheses are used for disambiguation. All LTL formulas included in VERIFY are parsed and canonicalized by the Spot library (version 2.11.6), ensuring they conform to this fragment and have a standardized representation. We assume the standard semantics of LTL over infinite traces (Pnueli 1977: Clarke al. 2018).
C.2 STRUCTURE OF THE CANONICAL. INTERMEDIATE TECHNICAL LANGUAGE CANONCAA)
The canonical ITL (amoncaoi is not defined by independent generative grammar but is rather procedurally generated from the AST of an LTL formula. results in structured English strings composed of atomic proposition identifiers, specific keywords/phrases corresponding to LTL opera- tors, and punctuation (primarily commas and parentheses that mirror the LTL structure). The core keywords and templates for [TLCannoical (derived from the mapping rules discussed in Section 4.2) are defined as follows (where L'TTL and IT represent the ITL translations of LTL subformulas and d respectively):
Atomic proposition maps to its string representation (e.g., "p").
T: maps to "true".
I: maps to "false".
-:: maps to "not 6'TTI."
Φ t: maps "Ø'TTI and v'TT""
Ø V th: maps to "ØITL, or V'TTL"
Ø > uh: maps to "if d'ITL, then VITL".
Φ +> wh: maps to "Ø'irz if and only if WITL".
XΦ: maps "In the next state, ØITL".
Gø: maps to "Always, BITr".
Fá: maps to "Eventually, ØITI."
d t: maps to "Ø'TT, until V'rT.""
à R di maps "d'TT, releases v'rT.."
ó W vo maps to "dITE weakly until V'rL".
The generation process ensures that the nesting and scope of operators in the LTL formula are preserved in the hierarchical structure implied by the ITL string composition, often through implicit parenthesization mirroring the LTL AST structure.
C.3 FORMAL DEFINITION OF THE MAPPING FUNCTION T
We define the mapping function T LTLxF + Strings, which translates an LTL formula ø e LTLvE (assumed to be in its Spot-canonical form and represented AST, denoted AST())) into its ITL Caneniclall string representation. The function T defined recursively based on the structure of AST(O):
1. If where AP: T(AST(a)) "a" (the string literal of the atom).
2. If ó = T: ((AST(T)) = "true".
3. If = T(AST(L)) = "false".
4. o p = -b: T(AST(-*)) "not" " e T(AST())), where @ denotes string concatenation.
5. If 1n Ws: T(AST(i) A x22) T(AST( ([(AS(()))) "ndd" " T(AST(()2))
20

---

Published as a conference paper at ICLR 2026
6. o 11 12: ((SS(( V 42)) = ((AST((11) or T(AST((22))
7. o Φ- = 11 v2: ((AST( + (b)) = "if" + ((AST(())) @ ", then "# ((AST()2))
8. ( = 11 + 02: T(AST( + #2)) = (()ST((1)) @ if and only if ((AST( (2))).
9. 6 x T(AST(X()) = "In the next state, @ ((AST(
10. o = Gy:: T(AST(G()) = "Always, " T(AST())).
11. 6 Ft: T(AST(FU)) "Eventually, " " T(AST())).
12. 1 11 U 22 ((AST(01 U v22) T(AST((1)) @ until + T(AST(
13. If 11 R 2/ ((AST( R(b2)) = -T(AS(()1)) @ releases @ (22))
14. If 6 11 W 22 (( W 42)) ((AST((1)) weakly until " 4 (()))
15. If 6 = 1 MM Wh T(AST( M 1/2)) = (())T(()))) # strong release " ( T(AST (12)).
Parentheses in the output ITL string are implicitly handled by the recursive structure of T and the string concatenations, preserving the LTL. AST's operator scope and precedence. Explicit parentheses can be added around the ITL for sub-formulas in practice to ensure clarity, especially for binary operators, e.g., (AAST( A 42)) = "(" + T (AST((1)
') and (" + T (AST( 'Y". However, for this proof, the direct template application is sufficient as structure is inherited from the AST.
C.4 PROOF OF COMPLETENESS (TOTALITY OF 7)
We claim that the mapping function T is total for all LTL formulas & E LTLvF. That is, for every valid L.T. formula generated and verified in our dataset (which conforms wT),,p), T produces well-defined ITT Camonical string output. The proof proceeds by structural induction on the formula Φ.
Base Cases:
If 6 = a, where a E AP: ((AST(a)) is defined as the string literal "a".
If Φb T(AS((T) is defined as the string "true".
. If á = 1: ((AST(L)) is defined as the string "false".
In all base cases, T yields well-defined string.
Inductive Hypothesis (IH): Assume that for any LTL formula (and x. if applicable) that is a proper subformula of o, the function T is total, and ((AST(e)) (and T(AST(x)) is a well-defined
ITL string.
Inductive Step: We examine each case for constructing from its subformula(s) according the rules of LTLyF:
1. If ó = -v: By the IH. (1) is a well-defined string. The rule for - (Rule 4 in the definition T) defines (ASST(-()) as the concatenation "not" " *T(ATT( This operation well-defined strings results in well-defined string.
2. If ti op 122, where op E
W:: By the IH, (AAST((1)) and T(AST(12)) are well-defined strings. The rules for these binary operators (Rules 5-8, 12- 14 in the definition of T) define T ((AST())) as a concatenation (AST(()1)) specific ITL keyword for op, and T(AST(122)) This results in well-defined string.
3. If Φ = op d,, where op E
By the IH, T(AST(E)) is a well-defined string. The rules for these unary temporal operators (Rules 9-11 in the definition of T) define T T(AST(Ø)) as concatenation of the ITTL keyword for op and T(AST(#)). This results in a well-defined string.
Since the base cases hold and the inductive step covers all LTL operators defined LTLyr. the function T is total for all formulas o L.T.yr
21

---

Published as a conference paper at ICLR 2026
C.5 PRESERVATION OF SEMANTIC STRUCTURE AND REVERSIBILITY
Semantic Structure Preservation: The function T is designed to be structure-preservving operates directly on the AST derived from the Spot-parsed (and canonicalized) LTL formula. The recursive definition of 7 ensures one-to-one mapping between LTL operators in the AST and their corresponding ITI. keywordsttemllate. The recursive application these mappings ensures that the nesting and scope of operators in the LTL formula are preserved in the hierarchical structure of the resulting IT string. This structural isomorphism provides a strong basis for asserting that the core semantic relationships (temporal and logical) of the LTL formula are maintained in its ITL translation. A formal proof semantic equivalence would require formal semantics for ITL; however, the systematic, structure- driven nature of T supports this claim.
Reversibility (ITL to LTL): The Cannnniical strings generated by T are designed to unambigu- ously parsable back into LTL formulas that are semantically equivalent to the original LTL formulas. This reversibility is crucial for verifying the integrity of the ITTI. representation. As described in Sec- tion 4.2 ITL-to-LTL, parser developed based the inverse of the 'LTT._TO CANONICAL" rules. The VERIFY dataset construction pipeline includes an automated verification step where canonical ITL strings are parsed back LTL, and this reconstructed LTL is then formally checked for semantic equivalence against the original Spot-verified LTL formula using Spot's built-in capabilities (e.g., 'spot.are equivalent()) ) This empirical validation across the dataset (specifically, 18% of as mentioned in Section 4.4 for NL, and similar process for ITL integrity check mentioned in Section 4.2) confirms that the I.TI. >> ITI. L.T. round trip preserves logical meaning.
C.6 CONCLUSION
The mapping function from the defined and verified L.T. fragment L.TL to I'Tl Ianennnniiall total (complete), meaning every formula in L.TLyr has corresponding ITI. string. This mapping is deterministic and preserves the structural composition of the LTL formula. Empirical verification through round-trip LTL-ITL-LTL conversion and semantic equivalence checking using formal tools (Spot) further confirms that the generated canonical ITL accurately represents the logical meaning of the source LTL formula. Therefore, the grammar used for translating LTL to ITTLCanoniall is complete with respect to the LTLyF fragment.
D EXTENDED DATASET DETAILS
This appendix provides further details about the VERIFY dataset, including additional illustrative examples, comprehensive per-domain statistics, supplementary visualizations, and the full data schema.
D.I ADDITIONAL EXAMPLES
To further illustrate the nature and diversity of the VERIFY dataset, this section presents five detailed examples. Each example includes the domain, the natural language definitions for propositional variables (Activity). the Spot-canonical Linear Temporal Logic (LTL) formula, its corresponding rule-based canonical Intermediate Technical Language (ITL) representation, and the final contextual Natural Language (NL) translation. These examples showcase variations in logical complexity, domain-specific terminology, and the types of properties represented.
D.1.1 EXAMPLE 1: AUTOMOTIVE/AUTTOOOOOL VEHICLES
Domain: Automotivee/Autooomouss Vehicles
Activity:
p: Lane departure detected
q: Obstacle detection active
r: Driver override requested
S: Sensor calibration complete
t: Emergency braking engaged
22

---

Published as a conference paper at ICLR 2026
u. System in autonomous mode
V: GPS signal lost
W: Manual steering imput detected
LTL Formula (Spot-canonical;
(X-(9Rp - Xs)U (p - RXu + -
Gq)))
( Xr) WrRtU (gA + (rV)) (GrWg - (t - w) Wu - (oU ())
(L~(((V))))) Fp) WrWrRu) RrUrR(~s))
Canonical ITL:
if In the next state, not if releases p. then In the next state, until if p. then q releases In the next state, u, then not In the next state, q or q releases if p. then s until if until w weakly until then q and if and only if Always, q until Eventually, q or In the next state, r releases not S weakly until r releases until q and v or v if and only if or t weakly until q if and only if if Always, r weakly until q. then if t, then w weakly until u, then if and only if until if and only if u weakly until p or not Always. v or if t, then q or and Eventually, weakly until weakly until r releases u releases until releases not
NL Translation: The system must maintain that after lane departure, either obstacle detection remains active until sensor calibration follows, or autonomous mode persists until manual override triggers protocol where persistent lane-keeping requires continuous obstacle detection, until either emergency braking initiates redundant navigation checks or GPS loss forces driver takeover whichever occurs first while maintaining sensor integrity throughout.
D.1.2
EXAMPLE 2: HOME AUTOMATION
Domain:
Home Automation
Atomic propositions represent device states: u=user presence detected, r=security system armed, q=lights activated, ==door locked, t=motion detected, v=HVAC runningunning w=window open, p=power saving mode
Activity:
LTL Formula (Spot-canonical):
GGuW ()) - () qU)))=)(-)))////) (())
U V ((uA U Wr - (GsV Wo) W (X(r - +t) RFu
((-s) A u)))
U U((-sVFo) WFs Agqu + Xx)
Canonical ITL:
if Always, weakly until if r, then q, then q until not weakly until if if and only if then r until s and In the next state, Eventually, if t, then q until r weakly until releases In the next state, q r weakly until Eventually, weakly until if u or v, then and not s weakly until In the next state, w or if not p and v until rf then w weakly until r releases u, then w or Always, weakly until weakly until and In the next state, if v, then releases Eventually, u until not s or Eventually, v weakly until Eventually, s and q releases u if and only if In the next state, until q
NL Translation: If user presence persists until security arming triggers lights, then lights must stay active until disarm occurs, maintaining that locked doors require security en- gagement until motion triggers lights through HVAC operation, while windows stay closed unless presence or HVAC override occurs. and power mode exits when security triggers window control until #VAChemperature conditions maintain either sustained locking or window states matching security status through presence verification.
D.1.3 EXAMPLE 3: SECURITY AND AUTHENTICATION
Domain: Security and Authentication
23

---

Published as a conference paper at ICLR 2026
Activity:
pi Unauthorized access detected
T User authentication request
F: System lockdown activated
S: Security breach detected
E: Two-factor authentication enabled
U: User role changed
V: Vulnerability patched
W: Access granted
LTL Formula (Spot-canonical);
((-((t ++ (-D)) RFP
(-s) W (r + Fq) Wo
s] + (w++ W (((-s) (p EpUpR(s + s)
W W
Canonical ITL:
and only if not s or q and only if releases not p releases Eventually, p or or not t weakly until Always, until if In the next state, Always, then until not weakly until r if and only if Eventually, q weakly until v weakly until s weakly until u
NL Translation:
The system must maintain that: either (1) security breach absence and two-factor-autee aaecto--auteentiatioon alignment with authentication requests persist while blocking unauthorized access until detection occurs, or (2) lockdown persists until two-factor ac- tivation if future lockdown continuity implies it, all until breaches cease. This continues weakly until breaches force lockdown equivalence to pending authentications, repeated vulnerability patching, and role changes persist only if access-right consistency holds unless infinite authentication demands or role changes occur until security states resolve.
D.1.4 EXAMPLE 4: SMART GRID/ENERGY MANAGEMENT
Domain: Smart Grid/Energy Management
Activity: p=peak load condition, t=tariff adjustment activated, z=renewable generation available, u=usage restriction enforced, w=wind power input threshold, q=grid stability query issued, s=storage system activated, v=voltage stability compromised
LTL Formula :Spot-canonical):
(pWFXp ++t)
R((UpVV (t +) (UNAUUUNAAXNNNiiiaaaaa--- W W (uWo + xt) uR f))
(vR((uVTRuW A Fo) A AGs
L)U
(g -9 (-g ++ v) Ru
Canonical ITL:
if and only if p weakly until Eventually, In the next state, p releases not r until p until u weakly until and In the next state, until w if and only if if not Eventually. r, then releases not w releases t if and only if until releases or r releases w and Eventually, v and Always, if and only if Always, until weakly until weakly until Eventually, s if and only if In the next state, the next state, not until q if and only s until Eventually, or if s, then if then until and not q if and only if releases
NL Translation: Tariff adjustments match peak load persistence until eventual resumption if and only if grid operations maintain: renewable availability until peak load or usage restrictions with wind thresholds, requiring storage activation only when stability queries trigger sustained responses, unless voltage instability forces delayed demand response until tariff-voltage alignment governs restoration.
24

---

Published as a conference paper at ICLR 2026
Table 13: Application domains covered in the VERIFY dataset.
Domain
Illustrative Context Example Snippet (activity)
Financial Services
p=trade execution confirmed, q=risk limit check passed p=user adds item cart, q=inventory level updated p=motion detected room, q=lights turn on p=altitude within safe range, q=autopilot engaged p-heart rate exceeds threshold, q-alert generated pFpressure threshold reached, q=safety valve opens p=obstacle detected by sensor, q=emergency brake applied
Web Services / E-commerce
Home Automation
Aerospace/ Avionics
Medical Devices
Industrial Automation/ / Mfg.
Automotive Systems
Robotics Autonomous Systems
p=battery level low, q=robot returns to charging station
Network Protocols / Security
p=login attempt failed 3 times, q=account locked p=invoice approved, q=payment scheduled p=package scanned hub, q=tracking status updated
Business Process Management
Supply Chain / Logistics
Energy Systems Smart Grid
p=demand exceeds supply. q=load shedding initiated
Telecommunications
p=call successfully connected, q=billing record created
D.1.5
EXAMPLE 5: VERSION CONTROL AND CODE REVIEWS
Domain:
Version Control and Code Reviews
Activity:
q: Code review requested v: Code review passed u: Code conflicts resolved
Tests passed w: Work-in-progress flag p: Pull request open r: Revision submitted I st
Code merged
LTL Formula (Spot-canonical):
qu (oV (((u - t) A (w p + (p ((-)) U ((UU A u) Rs > -XtWRRtRp V Gạ) A (s FuUrW (w + u)) WGFIWGs W u( Rp V w) W (GsUwUgRs ((XI (r A (-p) UuRl + w)))
+ W=)
Canonical ITL:
q until not if and only if if u, then and p w releases not t until p until if p. then if Eventually, w, then In the next state, s weakly until until if v until t and v releases s, then not In the next state, weakly until w releases releases p or Always, q and s if and only if Eventually, w until r weakly until u if and only if w weakly until Always, Eventually, t weakly until Always, s weakly until or and u releases p weakly until Always, s until w until releases s or w if and only if not p until u releases
NL Translation: A code review remains requested until either passes, or (if conflicts being resolved guarantees tests pass and an open pull request or WIP flag persists while tests are failing until a revision or pull request exists) exactly when no revision exists, until pull requests being open implies (if work eventually continues, the next state must merge code *or* keep merging pending) persists, while either: (1) review passes until tests succeed with passing review under merge protection until tests require WIP persistent review requests; or (2) merging occurs only if eventual WIP under revision constraints matches conflict resolution, weakly until recurring tests and merges align with open pull requests or WIP, provided merges persist until WIP transitions or review compliance.
D.2 PER-DOMAIN STATISTICS
To provide deeper insight into the characteristics of the VERIFY dataset across its 13 domains, Table 13 summarizes the application domains covered VERIFY and Table [4] summarizes key statistics. These include the number of unique LTL formulas, various measures of LTL formula complexity (average, median, min/max number of temporal operators, and AST depth), natural language translation length statistics (word count), activity string length statistics (word count), and approximate vocabulary sizes for both translations and activities within each domain.
D.2.1 LTL OPERATOR AND SUB-PATTER FREQUENCIES PER DOMAIN
The distribution of LTL operators and common structural patterns (identified by Spot's formula kinds) varies across domains, reflecting different specification needs. Below is summary of the frequency
25

---

Published as a conference paper at ICLR 2026
Table 14: Detailed Per-Domain Dataset Statistics. "LTL Ops" refers to the count of temporal operators. "LTL Depth" refers to the AST depth. "Words" refers to word count. "Vocab Size" is the count of unique words (lowerease, simple tokenization).
Domain
Unique LTLs
LTL Operators Med
LTL. Depth
NI. Trans, Words
Activity Words
Vocab Size Activity
Asg
Min/Max
Aug
Med
Avg
Med
Min/Max
Arg
Min/Max
Acrospace
16821
6.0
6
0/19
4.98
56.21
56
10/152
33.99
619
-40034
~.S110
AutoerAutoommuss
16711
6.0
6
13/2
5.04
56. 17
56
10/162
20.83
301
5188
~3149
~6318
Ruild PipelinssCI-CD
16737
6.0
6
0/21
4.98
5
53.60
54
6/152
34.50
24
5185
~2975
~3525
Fiaanaa//ranseecin
16765
6.0
0/18
4.98
53.26
7/140
26.54
5/130
~2507
Home Automation
16748
6.0
0718
56.82
32.31
~2066
~3019
Industrial Automation
16782
6.2
0723
5.17
54.70
5/201
28.27
5/7
~3943
<4914
Medical Devices
16710
6.0
1719
5.01
56.04
56
6/144
34,86
699
~4045
~4908
NetworkingDisttrbbuus
16748
5.9
6
1/21
4,95
5
52.
53
11/142
25.08
28
6/78
<3636
~:577]
Robotics
16800
6.0
1719
4.9%
56.66
57
5/304
33.71
34
6/78
-1854
<4537
Sssrin//Athemmicaion
16750
6.0
0/18
4.98
54.31
31.14
~2808
Smart Grid/zergy
16715
55.
59
-4009
Version Control
16820
5.9
(/21
4.97
55
6/185
32.98
6/107
<3163
~3630
Web Service/AAPss
16764
5.9
6
1719
4.96
53.38
53
6/126
28.18
28
6/98
-3601
~3675
Overall Dataset
m=15,000*
5.92
5
0/14
4,99
5
54,93
54
5/201
31.02
31
5/130
mUSK*
m=10K*
Total unique LTLs tonl unigue vocabullryy accoss all domains
ITI. cimplesiiy extistos (Ops and Depth) aee teaual on the LTTI feemulos soancisted with immehriens each domsin.
The LTI. Opentor counte in this raffe all opensors (lempoeal and boolss)) whenez Figunc la focuss lenpural operation unly.
of top-level LTL operators (G, F, X) and common Spot formula kinds (c.g., Implies, U, R. W, Equiv, And, Or) for each domain. This data is derived from analyzing the LTL formulas associated with the NL translations in each respective domain.
Aerospace: Predominantly features 'G' (Global), "F" (Finally), and "X' (Next) as top-level operators. Common structural patterns include Implications, Until, and Release. (Example Implies: 2900, R: 2510, U: 2496)
counts: G: 2059, F: 2055, X: 2055;
Automotive/Autonomoon Vehicles: High use of 'X', 'G', and 'F'. Implications, Until, and Release are common patterns. (Example counts: X: 29, G: 2058, F: 2051; Implies: 2886,
U: 2552, R: 2468)
Build Pipelines and CI/DD: 'F', 'X', and 'G" are frequent. Structural patterns show many Implications, Until, and Release forms. (Example counts: F: 2125, X: 2063, G: 1950; Implies: 2865, U: 2529, R: 2481)
Financia//Transaction Systems: 'X','F' 'G' are common. Implications, Release, and Until patterns are prominent. (Example counts: X: 2118, F: 2069, G: 1998; Implies: 2925, R: 2508, U: 2488)
Home Automation: Balanced use of 'X', 'F', 'G'. Implications, Release, and Until are frequent structures. (Example counts: X: 2098, F: 2088, G: 2069; Implies: 2966, R: 2523, U: 2479)
Industrial Automation/Maanuaauuuu and Release patterns are common. (Example counts: F: 2050, G: 2031, X: 2027; Implies: 2852, U: 2584, R: 2472)
FF''G: 'X" are prevalent. Implications, Until,
Medical Devices: 'F, 'X', 'G' appear often. Implications, Weak Until (W), and Release are frequent. (Example counts: F: 2104, X: 2047, G: 2037: Implies: 2860, W: 2527, R: 2497)
Networking/Distribbtee Systems: 'G', 'X', 'F'' are common. Implications, Weak Until (W), and Release structures are frequent. (Example counts: G: 2103, x: 2039, F: 2034; Implies: 2948, W: 2470, R: 2459)
Robotics: High frequency of 'G', 'F', 'X.. Implications, Until. and Release are common patterns. (Example counts: G: 2088, F: 2082, X: 2063; Implies: 2958, U: 2479, R: 2464) Security and Authentication: 'X', 'F', 'G' are prominent. Structural patterns often involve Implications, Until, and Weak Until (W). (Example counts: X: 2107, F: 2077, G: 2059; Implies: 2967, U: 2502, W: 2442)
Smart Grid/Energy Management: 'G', 'F', 'X' are frequent. Implications, Release, and Until patterns are common. (Example counts: G: 2111, F: 2100, X: 2040; Implies: 2938, R: 2509, U: 2432)
Version Control and Code Reviews: 'G', 'F', 'X' appear often. Implications, Weak Until (W), and Until structures are frequently used. (Example counts: G: 2130, F: 2077, X: 2066; Implies: 2903, W: 2527, U: 2492)
26

---

Published as a conference paper at ICLR 2026
Web Services/APIs: 'X', 'F', "G" are common. Implications, Until, and Release frequent patterns. (Example counts: X: 2138, F: 2102, G: 2052: Implies: 3020, 2557, R: 2501)
The operator counts for G. F, X above refer to their appearance as the outermost temporal operator in many formulas within the domain, indicating common high-level properties like invariants, eventualities, or next-star statestate transitions. The structural pattern counts (Implies, U, R, etc.) are derived from Spot's analysis of formula kinds within the LTL expressions for each domain.
Note:
D.3 ADDITIONAL VISUALIZATIONS
This section presents additional visualizations to complement the main paper, providing further insights into the VERIFY dataset's characteristics.
Overall Statistics:
Mean: 11.84
Median: 11.00
Distribution of LTL Operators by Domain
Min: 0 48
Max:
Overall Mean: 11.84
Total samples: 217916
42
Temporal Operator Count
36
09
30
24
12
(602))))
Industrial Automation Manaaaafaaa faatinng (n= 16787)
(n= Aarestaace
Financial/Tramsan Transationn gystoms (o= (7769)
Robotics (n= 16801)
Web Gardces/AP
Home Automation
Security and Antheriiation
16769)
Networidng/Distrribee Systams 16751)
(a= 16753)
(a 16753)
Build Pipelines and CHCD
Smart Grd//Enegy Monggemmnt
(0=161 (3)
Antamotiv/Aattacmmaa Vehicles
10-10 18719)
Medical Posfras (0= 187/ (4
to= 10771 12)
Version Control and Code
Domain
Figure 3: Distribution of LTL temporal operator counts per formula, shown box plots for each of the 13 domains in the VERIFY dataset. The overall mean is indicated. This complements Figure la in the main paper by providing domain-specifie views.
27

---

Published as a conference paper at ICLR 2026
Overall Statistics:
54.93
Median: 54.00
Distribution of Translation Lengths by Domain
Min:
Max: 204
Overall Moan: 54.93
Total samples: 217916
175
Natural Language Word Count
150
125
100
75
50
Version Control and Cods (682))
Industrial Automation aonutaaaatttt antaattriin (n 6787)
Jorospace (n- 168222
Financke/Tramsaddio Systems restoss
Robotics (n 18801)
(n 6769)
Web Services/APil
i (0769)
Home Automation In (675))
Security and Authanticn Anthentication (n= 16753)
Mntsoorktag/Dssrrbatte Spstams (n 16751)
Build Pipelines and CMCD (e 167
Smart Grid/Energy Managemmen
Autamotrro/Automonn Vebicles (n 16712)
i- 16719)
Medical Devices (n- 167140
Domain
Figure 4: Distribution of natural language translation lengths (by word count) per formula, shown as box plots for each of the 13 domains. The overall mean is indicated. This complements Figure 1b in the main paper with domain cemain--pecific distributions.
28

---

Published as a conference paper at ICLR 2026
Mean Number of LTL Operators by Domain
12.04 11.91 11,88
11.88
11.85
11.84
11.84
11.83
11.83 11.81
Global Meani 11.84
Mean LTL Operator Count
10
Mean Translation Length by Domain
54.70
56.17
56.03
56.82
53.60
56.66
56:21
54,30
54.99
Global Mean;
Mean Translation Word Count
50
40
30
20
10
Industrial Automatton//anuu Manadarrrrnn
Automatre/utmmmmi in=1671.)
16787)
Vohhelea
Modical Devices ta (6714) L4)
Sneart Cour/E Manggement (0 (0-16700) (7719)
Home Autonation
(n=16753)
(he Be16743) 6743)
Robetical (ne 16801)
Samuriy and wothhantattion (ne 16753)
Versian Control and Cedo Reviews (o 16823)
Actrapaos (a 6822)
Fiaacial/Paamecioo Syatemo
Networknng Systamo (ne 17751)
(n 6766))
Web Pr the 16769)
Bulld Pipelins and
Domain
Figure 5: Summary of mean LTL operator counts (top) and mean natural language translation word counts (bottom) across all 13 domains. Global means are indicated by dashed lines. This provides direct comparison of averages across domains.
29

---

Published as a conference paper at ICLR 2026
Distribution of Temporal Operators by Domain
LTL. Operator
200000
Gioteily (f)
Exentually
175000
10.4%
18.7%
18.0%
18.7%
16.0%
10:7%
1..07
10.0%
Uniti (17
Next IO
Release
10.44
10.0%
10.7%
10.8%
18.0%
Operator Count
125000
16.9
18.0%
18.7%
100000
75000
14.4%
14.0%
14.74
14.65
14.0%
11.0%
11.0%
11.4%
14.5%
11.0%
14.7%
14.0%
14.7%
50000
14.4%
14.7%
DAR4
14.5%
14.0%
14.0%
14.%%
14.5%
14.0%
25000
Instatnal Automation/ Manatiio//mattttttttttt (aiiznss
Antemedhe/AAtten Sataeedbe/Atttoommoo Antooomottoto Vehitles (n (Reeec))
Aereppara
(nr LI802
Ronn Automation
(n=]
Verion Contral und Cado Bacioves ine 183070)
Robetics (nte (itreci
Medical Destitue new (88540)
Buld Pipalines and CleCD
Seandy and nadiesttiationn
Smart Ord/Eaerry Managema
(ped) (91273)
Financid Pmancial/Trannane Transstion Systamo
(==) (9)21))
madl (99 (30)
Nobwrrkass Matiittte ERR2990
(pe) Ur73al
SariassAPis
(g= UFTA32)
Domain
Figure 6: Stacked bar chart illustrating the relative frequency of the six primary LTL temporal operators (G, F, X. U, R. W) within each of the 13 domains. This visualization highlights domain- specific tendencies in temporal patterns.
30

---

Published as a conference paper at ICLR 2026
Correlation Between Formula Complexity and Translation Length
Linear regression (y 31.86 1.95x)
200
Pearson correlation: 0.704
p-value: 0.000e+00 00
Word Count in Natural Language Translation
Slope: 1.948
Intercept: 31.86
175
Sample size: 217916
150
100
75
50
10 Number of LTL Operators in Formula
20
30
10
Figure 7: Scatter plot showing the correlation between LTL formula complexity (number of LTL operators) and the word count of the generated natural language translation. linear regression line is overlaid. (Pearson correlation r=0.704, p < 0.001).
31

---

Published as a conference paper at ICLR 2026
Density Plot: Formula Complexity vs Translation Length
Linear regression (y 31.86 1.95x)
200
Pearson corvelation:
0.70
p-valte::
Count Natural Language Translation
Sample
175
10
150
125
Log' O/Cunt)
102
100
50
10"
10
20
30
40
Number
Operators Formula
Figure 8: Density hexbin plot illustrating the relationship between LTL formula complexity (number of LTL operators) and the word count of the natural language translation. Darker regions indicate higher concentration of data points. The linear regression line from Figure 7is shown for reference.
D.4 FULL DATA SCHEMA
The VERIFY dataset is released in standard CSV and Apache Parquet formats. Each record in the dataset represents a single LTI-ITTI-NI triplet, along with its associated contextual information and metadata. The detailed schema is presented in Table 15 derived from the internal database structure.
Table 15: Full Data Schema for the VERIFY Dataset.
Column Name
SQLite Type
CSV/Parquet Type
Constraints
Description
triplet)
Unique identifier recund (triplet). klentfier linking to a unique LTL, fommala strucrune (Spotccanomical), Identifier linking the canonical
formula_id
INTEGER
i0t
NOT NULL; FK concppual formulas table
itl_id
INTEGER
NOT NULL; conceptual table
(derived from Formi" a. id).
domain
TEXT
string
NOT NULL
The application domain providing context (c.g.. 'Acrospace" language definitions propositioal variables used the LTL Formula, specilic to the given domain.
activity
string
formformul
string
string (Spot-canonical representation us- ing standard ASCII openters). The comespoading cononical Inter- mediate Technical Language (TTL. string.
1eI_representacion
TEXT
string
NOT NULL
translation
TEXT
string
NOT NULL
The contextual Natural Language (NL) description corresponding to LTL/ITL puir within the speci- domatin.
generationn_iin
REAL
foatt
taken seconds) to generate the 'activity" and "rrns- lation for this record.
LLM
timestanp
string
ISO 8601 timestump indicating (specifically
part) was generated.
Conceptual Data Relationships:
Each unique LTL formula (identified by formula id after Spot canonicalization) has exactly corresponding canonical ITL representation (identified by it i
32

---

Published as a conference paper at ICLR 2026
A single LTL-canonical ITL pair (i.e., unique formula id)) appears multiple times in the dataset, typically once for each of the 13 domains for which natural language translation and activity definition were generated.
The primary key id in the released files uniquely identifies each L.TL-ITI-NI--omaal quadruplet.
Example Raw Record:
As a CSV data record (header shown first for clarity): id, formulooidddaiomm itl_id, domain, aetiviity,t1__oommeet formula, itl representaaionn, translation, ootp timestamp
106230,22, 229, "err,,"eerroooo "Aerospace", "Consider an aircraft flight control system. 'p' indicates the autopilot is engaged. 'q' indicates the aircraft is on the correct flight path. 'r' indicates a critical system failure. 's' indicates the flight director is providing guidance. 't' indicates the aircraft has reached its destination. 'u' indicates the aircraft is maintaining a safe altitude. 'w' indicates the weather conditions are within acceptable limits.",
(p W q & W U (! (E q) U (1 X t) s",
"if and p weakly until weakly until until not if z, then q and r releases u until not In the next state, Eventually, P, then s",
"If acceptable weather and the autopilot being engaged persist at least until the aircraft is on the correct flight path, and this condition persists at least until the flight director provides guidance, until the aircraft reaches its destination or it is not the case that if there is a critical system failure, the aircraft maintains safe altitude and the critical system failure continues to be true at least until the aircraft is on the correct flight path at least until it is not the case that in the next state eventually the autopilot is engaged, then the flight director provides guidance 1. 1.54899549484253, "2025-04-30112:5:: 318.44112: 943122"
As a JSON object representing one row (formatted for readability):
"id": 106230, "formula_ide 22, "itl id": 229, "domain": "Aerospace", "activity"; "Consider an aircraft flight control system. 'p' indicates \
the autopilot is engaged. 'q' indicates the aircraft is on the correct \
flight path. 'r' indicates a critical system failure. 's' indicates the flight director is providing guidance. 't' indicates the aircraft has reached its destination. 'u' indicates the aircraft is maintaining a safe altitude. 'w' indicates the weather conditions are within \ acceptable limits.", "itl_formula" " "(p W Wagw G (r -E-RUGGUU
(! x p) t) -> s", "itl "ettestpessnnt "if w and p weakly until q weakly until until \ t or not if r, then q and r releases u until not In the next state, Eventually, p, then s", "translation"; "If acceptable weather and the autopilot being engaged \ persist at least until the aircraft is the correct flight path, and
33

---

Published as a conference paper at ICLR 2026
this condition persists at least until the flight director provides guidance, until the aircraft reaches its destination or it is not the \ case that if there is critical system failure, the aircraft \ maintains a safe altitude and the critical system failure continues to be true least until the aircraft is the correct flight path at /
least until it is not the case that in the next state eventually the V autopilot is engaged, then the flight director provides guidance.", "generation_tinee time:: 1.54899549484253, "timestamp": "2025-04-30112:51:1
E IMPLEMENTATION DETAILS
This section details the methodologies and resources used for dataset generation, verification and the establishment of baseline experimental results.
E.1 LTL FORMULA GENERATION AND VERIFICATION
Linear Temporal Logic (LTL) formulas were programmatically generated. The generation process recursively constructed formulas up to a maximum Abstract Syntax Tree (AST) depth of 25. This process utilized eight unique atomic propositions (denoted 'p' through 'w") and the standard LTL operators: Globally (G), Finally (F), Next (X), Until (U), Release (R), and Weak Until (W), along with boolean connectives (A, V, -,),
Generation:
To ensure structural diversity and manage the formula space, generated LTL formulas underwent a rigorous canonicalization process. This involved conversion to Negation Normal Form (NNF), expansion of implications and equivalences, application of associative and distributive laws, and standardized sorting of operands for commutative operators. unique hash was computed for each canonical structure to prevent duplicates in the master formula database, which was managed using SQLite.
Canonicalization and Uniqueness:
The semantic validity and non-triviality of all LTT. formulas were formally verified using the Spot model checking library (version 2.11.6). dedicated software component was designed to convert the LTL formulas from their generated format into Spot's required syntax. This component also managed the interaction with the Spot library for parsing, validation, and the retrieval of Spot's 's own canonical string representation for each formula. This ensured that all LTL formulas in the VERIFY dataset are well-formed and standardized according to a formal verification tool.
Formal Verification:
E.2 INTERMEDIATE TECHNICAL LANGUAGE (ITL) GENERATION AND VERIFICATION
AST-based Generation:
The canonical Intermediate Technical Language (TTL) representation for each verified LTL formula was generated deterministically. This process began by parsing the Spot-canonical LTL formula string into an internal AST representation, using Spot's parsing capabilities.
Rule-based Transformation: rule-based transformation was then applied by traversing the LTL AST. For each LTL operator encountered in the AST, a corresponding human-readable template was selected from predefined set of 12 core mapping rules (e.g., 'G maps to Always, ø': "Ø U maps to o until w/". This mapping ensures that the resulting canonical ITI string directly preserves the structure of the source LTT. formula while using more verbose, keyword-like operators.
ITL Semantic Integrity: To ensure the integrity of the ITT generation and its semantic equivalence to the source LTL, verification step was implemented. This involved programmatically parsing the generated ITL text back into an LTL formula using custom-designed recursive descent parser. This reconstructed LTL formula was then formally compared against the original, Spot-verified LTL formula. Equivalence reconstructed LTL formula matched that of the original Spot-canonical LTL formula. Spot's direct equivalence checking functions were also utilized during development for additional validation.
confirmed by ensuring that the Spot-canonical string representation of the
34

---

Published as a conference paper at ICLR 2026
E.3 NATURAL LANGUAGE (NL) GENERATION
Contextual natural language descriptions (comprising domain-specific propositional variable "activity" definitions and the 'translation' of the LTL/TI logic) were generated using the DeepSeek-RR R model, specifically accessed via the ddeessek-rrassoee API endpoint (model version available Q4 2024 Q1 2025). This API was publicly available, subject registration and usage quotas.
LLM and API Usage:
To generate the extensive dataset, parallel generation system was developed. This system orchestrated to 500 concurrent Python "asyncio" tasks distributed across multiple CPU nodes of an institutional perffrrrfrre computing (HPC) cluster. Each task handled an individual LTL/ITL pair for NL generation.
Parallel Generation Orchestration:
Prompting Strategy: For each LTT/TTL pair, a target domain was selected using a balanced sampling strategy designed to ensure roughly equal representation across the 13 diverse domains. The LLM was provided with the LTL formula, its ITL representation, and the selected domain. The prompt requested two specific outputs, encapsulated within XML-like tags:
Given the following formal specificatio LTL Formula: "[ti__oormllastrrinn""
Intermediate Technical Language (ITL): "itb__epresentaiioonssthnh) Domain: " "[domain_narm
Please perform two tasks:
1. Define plausible activities for the propositional variables used in the LIL formula, relevant to the specified domain. These definitions should make the LTL/ITL meaningful in that domain.
2. Translate the LTL/ITL formula into clear, concise, and semantically accurate natural language description. This description should incorporate the domain context and the activities you define.
Format your response strictly follows, ensuring the content within the tags is on single line if possible, or appropriately escaped if multi-line: <activity>p [definition of pl; = [definition of q); </activity> <translation> [Natural language translation the LTL/ITL incorporating the activities and domain context] < s/ttextti/rranslatiion
E.4 LLM JUDGING FOR NL VALIDATION
Validation Model: A substantial portion (18%) of the generated NL translations underwent
automated validation using large language model to assess semantic correctness and quality. The model employed for this task was 'meta-llamaaMetta-llaan
Model Configuration: The validation model was loaded using the Hugging Face 'transformers' library (version 4.49.0), with 8-bit quantization enabled via the "bitsandbytes' library (version 0.45.5).
The LLM judge was provided with the LTL formula, its ITL representation, the generated NL translation, and the corresponding 'activity' string. It tasked to output its assessment in structured JSON format. The prompt template was as follows:
Judging Prompt:
You are an expert in formal methods and natural language. Your task is to evaluate the semantic correctness and quality of a natural language (NL) translation with respect to given Linear Temporal Logic (LTL) formula and its Intermediate Technical Language (ITL) representation.
LIL Formula: (ltti_formlaaaaatoo deepprssentaiinnstiiiiee
ITL Representation: Generated NI. Translation (incorporating domain context and
35

---

Published as a conference paper at ICLR 2026
variable activities): "[nl__ranss" caanssaaionsstrnnnee
Domain Context & Variable Activities (as generated for the NL translation): "lactivity_atring
Please carefully assess the 'Generated NL Translation'. Consider the following:
1. Semantic Precision: Does the NL accurately convey the precise meaning of the LTL/ITL, especially the temporal relationships (e.q., always, eventually, until, next)
2. Contextual Appropriateness: Is the NL translation consistent with the provided 'Domain Context & Variable Activities'?
3. Pluency Clarity: Is the NL translation fluent, grammatically correct, and easily understandable?
Output your assessment *only* following keys:
single JSON object with the
"is_correct": boolean (true if the NL translation is semantically correct with respect to LTL/ITI and contextually appropriate; false otherwise)
"score": integer (an overall quality score from 0 to 10, where 10 is perfect)
"issues": list of strings (a list of specific problems identified, e.g., "Misinterprets 'Until' operator", "NL is awkward". Empty list if no issues.) "reasoning": string (a brief textual explanation for your judgment and score.)
Generation Parameters for Judge Output: The generation of the JSON response by the LLM judge used the following parameters to ensure consistent and structured output: temperature 0.1, top_1 0.95, do so_saplle- = True, and max new towenttees-s = 512.
E.5 BASELINE MODEL TRAINING
All baseline models were fine-tuned using a standardized methodology. Specific pre-trained checkpoints were sourced from the Hugging Face Hub. The Hugging Face 'transformers' library (version 4.49.0) and its 'Trainer' API were employed for the fine-tuning process. Data was tokenized using the respective model's 's default tokenizer, with input and output sequences padded truncated to maximum length 512 tokens. LTL and ITL formulas were treated as regular text sequences for tokenization.
For each model and task, hyperparameters were optimized based on performance on the validation set, using the primary metric defined for that task (e.g., BERTScore F1 for LTL/ITL-NL. Semantic Equivalence for NL-LTLL) The reported metrics in Tables 3, 4, and 5 of the main paper were calculated on the held-out test using the best checkpoint identified during validation.
representative configuration, exemplified by the T5-large model, is detailed below. Other models (T5-base, BART-base, BART-large, Llama-3-8B-Instrer CodeeLaama-7-Iinttuechh DeepSeek Coder-6. 6.7biinstruct) followed an analogous fine-tuning procedure, adapting batch sizes and learning
appropriate for model size and stability.
T5-large Example Configuration:
Pre-trained Checkpoint: '15-large' (from Hugging Face Hub).
Task Input/Output Formatting:
LTL/TTL + NL: Input:
'"translate LTL to NL: domain: domain activity: activity Itl: "ormula"" (similarly for ITL). Output: "N__sttanllation"
.NL > LTL/ITL: Input:
""translate NL to LTL: domain: domain activity: activity nl: WLLtrraslation "* (similarly for ITL). Output: ''LTL formula" or
"ITT__represenaation..
36

---

Published as a conference paper at ICLR 2026
LTL ITL: Input:
""translate LTL to ITL: Itl: LTL formula" (Output:
"ITL representation"); and vice-versa.
Training Hyperparameters:
Learning Rate: Initial 1 10-4, with a linear decay schedule.
Batch Size: 16 per device, with gradient accumulation steps of (effective batch size of 64).
Training Epochs:
Optimizer: Adam W (31 = 0.9, 52 = 0.999, 10 10-8).
Weight Decay: 0.01.
Scheduler: Linear scheduler with warmup for the first 500 steps.
Gradient Clipping: Max norm 1.0.
E.6 SOFTWARE AND HARDWARE ENVIRONMENT
Software Environment: The primary development and execution environment utilized Python 3.10.16. Key libraries and their versions include:
PoTccc (torch): 2.5.1
Transformers (Hugging Face); 4.49.0
Datasets (Hugging Face): 3.3.2
Accelerate (Hugging Face); 1.4.0
BitsandBytes: 0.45.5 (for 8-bit quantization)
Spot (for LTL manipulation and verification): 2.11.6
Pandas: 2.2.3
NumPy: 1.26.4
Scikit-learn: 1.6.1
NLTK: 3.9.1 (for METEOR score)
SQLite3 (Python standard library) for database management.
The CUDA version compatible with the PyTorch build and drivers was CUDA 12.1, with NVIDIA drivers version 550.x.
Dataset generation, initial processing, and LTL/TT verification stages were primarily conducted on an institutional high-perfi geggh-erformncce computing (HPC) cluster. These tasks utilized nodes equipped with dual AMD EPYC 9334 32-Core Processors. Natural language generation (orchestration of API calls) also managed from these CPU-based HPC nodes. The LLM judging phase (using Llama 3) and all baseline model training and testing were performed on separate institutional AI compute cluster. For LIM judging and training of larger baseline models (e.g., Llama-3-8B, Mistral-7B), nodes equipped with 8x NVIDIA H200 GPUs (141 GB VRAM per GPU) were utilized. Training of other baseline models (e.g., T5, BART variants) and final testing/evaluation across all models utilized nodes equipped with NVIDIA H100 GPUs (94 GB VRAM).
Hardware Environment:
E.7 COMPUTE RESOURCES
The development of the VERIFY dataset and the execution of baseline experiments required substantial computational resources.
LTL/ITL Database Generation & Verification: Approximately 2,000 CPU core hours on AMD EPYC 9334 processors.
NL Generation (API Orchestration):
Approximately 72 wall-clock hours, heavily
parallelized across multiple CPU nodes managing concurrent API calls. (The compute for the DeepSeek API itself is external..
37

---

Published as a conference paper at ICLR 2026
LLM Judging (Llama 3): Approximately 300 NVIDIA H200 GPU hours (for 18% of the dataset).
Baseline Model Training (average per model type):
-T5-large BART-large type models: Approximately 24 hours 4x NVIDIA H100 GPU configuration.
-Ts-base / BART-base type models: Approximately 12 hours on 4x NVIDIA H100
GPU configuration.
- -laaa 3 8B/ / Mistral 7B/ / CodeLllmma 7B/ DeepSeek Coder 6.7B type models:
Approximately 36 hours 8x NVIDIA H200 GPU configuration.
38


Reviews:
[{"review_id": "wTe2oYHdLx", "reviewer": "Reviewer_5AWc", "summary": "The research focuses on the **VERIFY** and dataset, which integrate Linear Temporal Logic (LTL) with natural language descriptions, using an Intermediate Representation Language (ITL).  The dataset links formal logic with contextual information, enabling better understanding and translation between logical formulas and natural language.\n\nKey findings include:\n- Translation tasks from LTL to NL are more straightforward than the reverse, as evidenced by higher performance scores for LTL-->NL compared to NL-->LTL.\n- ITL as an intermediate step significantly improves performance in generating natural language from complex LTL expressions.\n- Domain and activity context are critical for accurate translations from NL to formal logic; models without this grounding show substantial performance degradation.\n\nThe study highlights:\n1. The benefits of using ITL as an intermediary in translation tasks (e.g., LTL-->ITL-->NL outperforms direct LTL-->NL).\n2. The importance of contextual information in achieving accurate translations from natural language to formal logic.\n3. The relative difficulty of translating NL to LTL compared to generating NL from formal representations.\n4. Model strengths: fine-tuned models excel at generating natural language, while code-focused models demonstrate superior syntactic correctness in logic generation.", "questions": "Your results clearly show NL-->LTL translation is extremely challenging, even for fine-tuned models. Could you provide a deeper error analysis?\n\nWhat are the most frequent categories of errors in generated LTL formulas (e.g., wrong temporal operator (`F` vs `G`, `U` scope), incorrect proposition handling, negation/implication mistakes)? Quantifying this breakdown would be highly valuable.\n\nDoes using ITL as an intermediate representation primarily help avoid syntactic/semantic parsing ambiguities leading to invalid syntax, or does it demonstrably reduce the incidence of these deeper *logical* errors? If so, how?\n\nContext (\"Domain\" and \"Activity\") is shown to be crucial. How well do models generalize when encountering completely unseen Domains or Activities at test time?\n \nHave you conducted any experiments in a zero-shot setting where the Domain/Activity combination present during testing was absent from training? What are the performance drops  compared to seen contexts?\nIs there evidence that representations learned for similar Domains/Activities transfer effectively, or is context primarily memorized?\n\nHuman evaluation of semantic equivalence is critical but subjective.\nCan you report the Inter-Annotator Agreement (IAA) score (e.g., Cohen's Kappa) for your human assessments? \n\nFor cases judged as semantically equivalent by non-experts, were these formulas also verified to be logically sound/correct representations of the system requirement (perhaps spot-checked by a formal methods expert)? Is there any risk that \"semantically equivalent\" judgments might accept plausible but subtly incorrect LTL?\n\n Question: You show strong results for ITL in NL generation using T5-large and significant challenges with NL-->LTL on Llama-3-8B-instruct. Have you tested the same NL-->LTL task (especially fine-tuned) using significantly larger LLMs (e.g., GPT-4-class models like Claude Opus or Gemini Ultra, even via API)? Does scaling primarily improve syntactic correctness, semantic equivalence, or both? Do the core challenges identified persist? If not run due to cost/scope, do you hypothesize that these larger models might close the gap substantially, suggesting it is a data/model-size issue rather than an inherent task difficulty?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "It provides an integration of domain-specific NL with temporal logic via a novel canonical form (ITL). The focus on contextual grounding is also fresh compared to prior static datasets. \n\nMulti-model testing, robust metrics (BERTScore, semantic equivalence), human evals, and ablation studies proving ITL/context impact. \n\nVERIFY’s resources enable future work; insights on model strengths (e.g., code LLMs for logic generation) offer practical \nguidance. It pushes boundaries beyond NLP into formal methods accessibility", "weaknesses": "The paper clearly establishes that translating from NL to LTL is much harder than generation (that somehow expected), the *reasons* for this gap could be explored more deeply.\n\nWhile human evaluation was performed, its scope might not fully capture the challenge.\n\nThe paper uses models like T5-large and Llama-3-8B-instruct. Larger foundation models (e.g., GPT-4 class) might yield different performance characteristics.\n\nMissing References to relevant related work: For example,  He, J. at al. DeepSTL - From English Requirements to Signal Temporal Logic. ICSE 2022: 610-622", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tO7Qj2Ok9j", "reviewer": "Reviewer_UD8h", "summary": "This paper introduces VERIFY, a large-scale dataset containing over 200,000 triplets of Linear Temporal Logic (LTL) formulas, an intermediate technical language (ITL), and contextual natural language descriptions across 13 diverse domains. The key contribution is a novel ITL representation that bridges the structural patterns of LTL with natural-language syntax, along with a provably complete formal grammar for LTL-to-ITL translation. The natural language is generated using LLMs (DeepSeek-R1) and validated through manual inspection (10,000 samples) and automated LLM-as-judge evaluation (18% of the dataset). While the dataset represents a significant contribution in scale and domain diversity compared to prior work, it has critical limitations: heavy reliance on LLM generation without human-written examples, lack of grounding into executable state spaces, absence of verification traces, and missing analysis of whether the ITL intermediate layer actually provides measurable benefits over direct LTL-NL translation in practice.", "questions": "1. How did you validate that Llama 3.3's judgments align with human expert assessments? Can you report agreement statistics between the LLM judge and your manual 10k review?\n\n2. Can you provide stronger evidence that ITL is useful beyond the modest BERTScore improvements? For instance, does a full LTL->ITL->NL pipeline (with learned LTL->ITL) outperform direct LTL->NL?\n\n3. The paper claims to \"ground\" LTL but only provides linguistic context. Can you clarify the distinction and explain why executable grounding wasn't included? Or why it is not necessary for \"grounding\" evaluation?\n\n4. None of the experimental results include error bars or significance tests. What is the variance across different random seeds or data splits?\n\n5. What was the computational cost and time for generating 200k examples with DeepSeek-R1? Can others reproduce this with open models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A) With 200k+ examples across 13 diverse domains (financial services, aerospace, medical devices, home automation, etc.), VERIFY significantly exceeds prior datasets in both size and breadth. This addresses a major limitation identified in related work, where datasets were confined to narrow domains (robotics, navigation) with only 2-5k examples.\n\nB) The introduction of Intermediate Technical Language as a structured bridge between LTL and NL is conceptually sound. The provably complete LTL-to-ITL translation grammar (Appendix C) with formal verification using Spot provides mathematical rigor often missing in NL-to-logic datasets.", "weaknesses": "A) There is an over-reliance on LLM generation with insufficient human validation. This is my biggest concern with this work. The entire natural language component is generated by DeepSeek-R1 LLM with no human-written seed examples. Only 10,000 samples (4.8% of dataset) received manual expert verification by the authors themselves. The 18% LLM-as-judge validation using Llama 3.3 is problematic because LLMs judging other LLMs may share similar failure modes and biases, no inter-rater reliability metrics were provided for the manual 10k sample review, and there is no comparison with independent human annotators outside the author team. Hence, the >97% \"semantic correctness\" claim is based on an LLM's assessment, not human judgment.\n\nB) THere is an insufficient demonstration of ITL's practical value.\nThis is important as ITL is introduced as a key contribution, but its practical benefits remain unclear. The Evidence from Experiment A is weak as can can be seen in Figure 1. It shows ITL->NL achieves slightly better scores than LTL->NL (e.g., Llama-3: 0.94 vs 0.91 BERTScore), but the improvement is modest (~3 percentage points), and no statistical significance testing is provided. The only major difference comes for T5.\n\nC) The paper's title claims to \"ground\" LTL but provides no mechanism for verifying that translations are correct. Unlike some of the contemporary works like VLTL-Bench [1], which includes satisfying and unsatisfying execution traces for each formula, VERIFY has none. This means. that there is no way to validate semantic correctness beyond string matching, we cannot use traces as a supervision signal for training, and we cannot demonstrate that NL translations capture correct temporal semantics.\n\n[1] English et al., Verifiable Natural Language to Linear Temporal Logic Translation: A Benchmark Dataset and Evaluation Suite. https://arxiv.org/abs/2507.00877\n\nD) The paper doesn't address whether evaluation LLMs (Llama-3, Mistral) have seen LTL-NL pairs in pretraining. Given that Llama-3 was released in 2024 and trained on data through 2023, and prior LTL-NL datasets are public, contamination is plausible.\n\nE) What happens with different LLM generators (GPT-4, Claude)? Is dataset quality dependent on DeepSeek-R1's specific capabilities?\n\n\nMinor Issue:\n* Page 2, Line 54: \"syntactucally divergent expresession\" - should be \"syntactically divergent expressions\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rzzeM2vfQy", "reviewer": "Reviewer_svaN", "summary": "This paper introduces VERIFY, a large-scale dataset designed to bridge Linear Temporal Logic (LTL) and Natural Language (NL) through a novel Intermediate Technical Language (ITL). VERIFY supports bi-directional translation tasks between formal logic and natural language across 13 domains, encompassing more than 200k samples. The authors propose deterministic rule-based LTL-to-ITL conversion, LLM-driven natural language generation, and evaluate a series of models (T5, BART, Llama, Mistral, CodeLlama, DeepSeekCoder) on several translation settings. Analytical experiments demonstrate that (1) introducing ITL improves semantic alignment and syntactic correctness, (2) contextual information significantly impacts translation quality, and (3) direct LTL↔NL translation remains challenging even for advanced LLMs.", "questions": "- How consistent are the LLM‑Judge and manual verification outcomes across domains? Were there specific formula templates or temporal relations where semantic drift was most common? \n- Would you consider extending VERIFY beyond LTL to other logic families (e.g., CTL, MTL, or probabilistic temporal logics)? What challenges would arise in mapping or annotation consistency?\n- Could you provide explicit examples where the ITL→LTL mapping fails or encounters ambiguity in practical generation settings? Are there specific formula classes where ITL and LTL may diverge under real-world (noisy or partial) ITL?\n- The paper lacks a visual diagram of the overall VERIFY pipeline. Could you provide a figure summarizing the key components and information flow (dataset generation, ITL conversion, validation, and experiments) to enhance reproducibility and readability?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "VERIFY makes a strong contribution through its scale, structure, and experimental scope. It substantially extends prior logic–language datasets by providing over 200k LTL–ITL–NL triplets across 13 diverse realworld domains, enabling credible crossdomain reasoning and generalization studies. The introduction of a formally defined and humanreadable Intermediate Technical Language (ITL)—supported by a provably complete mapping—offers a principled and effective bridge between symbolic logic and contextual natural language, mitigating the brittleness of direct LTL <-> NL translation. Extensive experiments across multiple translation directions, model families, and ablation settings further underscore VERIFY’s diagnostic value and robustness as a comprehensive benchmark for logical–linguistic understanding.", "weaknesses": "The positioning of VERIFY relative to existing logic/NLP resources is somewhat limited—comparative benchmarks or perdomain statistics are lacking, making it difficult to precisely gauge coverage and novelty. The validation procedure, while methodically designed, only samples a portion of the dataset (≈18 % LLMchecked and 5 % manually reviewed) and does not analyze typical failure mechanisms, such as which operators most often induce semantic drift. Moreover, syntactic correctness for NL→LTL/ITL generation remains low (≈65–75 %), indicating that current models frequently output invalid structures; the paper would benefit from a qualitative error analysis with representative examples. Several metrics are underexplained, which limits interpretability of the reported numbers. Addressing these issues—particularly through an expanded baseline comparison and a concise errortype study—would significantly strengthen the paper’s empirical credibility and clarity", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "puRrKR0UBn", "reviewer": "Reviewer_82kz", "summary": "The authors introduce VERIFY, a dataset of 200k+ samples of (Natural Language/NL, Intermediate Technical Language/ITL, Linear Temporal Logic/LTL) triples spanning 13 domains. The stated goal is to support both training and benchmarking for translation among natural language (NL), a proposed Intermediate Technical Language (ITL), and Linear Temporal Logic (LTL). ITL is positioned as a human-readable, structurally faithful layer over LTL intended to ease learning and generation. The dataset is accompanied by formal checks on the logic side and mixed human/LLM-as-judge validation of the NL side.\n\nSoundness:\nThe pipeline for generating and validating LTL appears methodical, and the overall dataset construction is described with reasonable care. However, the central claim that ITL is useful as an intermediate representation is insufficiently substantiated by experiments. The empirical results do not isolate an end-to-end utility of ITL in realistic NL <-> LTL workflows; gains are primarily shown in easier sub-tasks (e.g., ITL-> NL vs. LTL-> NL) that may rely on privileged inputs (ground-truth ITL). Heavy reliance on an LLM-as-judge for “semantic correctness” further weakens the evidentiary basis.\n\nPresentation:\nThe paper is generally readable and well organized, but the main body lacks figures, dataset examples, and consolidated statistics that would aid comprehension. Some sentences are ambiguous or grammatically incomplete (e.g., the opening “bottleneck” is not specified; a few fragments/run-ons appear). Bringing key tables/plots from the appendix into the main text and adding a running example would materially improve clarity.\nOccasional fragmented/incomplete/run-on sentences:\n“Thus, failing to adequately bridge formal methods and practical NLP.”\n“Crucially, data quality is guaranteed through extensive validation protocols: manual expert verification of 10,000 random samples and automated semantic consistency and correctness checks judged by an LLM-as-judge approach (using Llama 3.3) (20; 21) across 18% of the dataset, confirming an estimated > 97% semantic integrity.”\n\nContribution:\nA large, multi-domain, logic-grounded dataset is valuable to the community, and proposing an intermediate representation (ITL) is an interesting idea. That said, the paper currently stops short of demonstrating practical benefits from ITL in end-to-end NL<->LTL modeling or user-facing tasks. If such benefits were shown more convincingly, the contribution would be stronger. Furthermore, it is unclear how each domain is defined in the paper. What distinguishes the domains, in terms of available predicates?", "questions": "1. Why are there no NL-LTL translation frameworks evaluated?\n2. What is Llama 3.3 evaluating? Semantic correctness of ITL? If so, is that grammar not “provably complete” with LTL?\n3. In Experiment A, performance is compared between the LTL->NL and ITL->NL tasks, with the aim of demonstrating that ITL is a beneficial intermediate representation. However, as the authors note, this requires the intermediate translation of LTL->ITL via a “task 5 model”, but the results in table 3 show that we can not reliably obtain a correct ITL translation from LTL. Are ITL->NL models in Experiment A given ground-truth ITL? If so, can’t we assume the practical performance of the ITL->NL task will be upper-bounded by the LTL->ITL performance, which is ~30%?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Broad semantic coverage over 13 domains\n2. Size of dataset is significant, offering over 16k entries per domain according to Table C Appendix A.1", "weaknesses": "1. Most significant weakness is the use of Llama 3.3 as a judge of “semantic correctness”\n2. Does not evaluate actual NL-LTL translation frameworks, such as those mentioned in the paper (Lang2LTL, NL2TL)\n3. Merit of ITL to humans is unclear and untested. One experiment shows that ITL->NL translation is more feasible than LTL->NL translation, however\n4. Examples from the dataset are missing in the main body of the paper\n5. No figures present in the main body of the paper\n6. Dataset statistics not present in the main body of the paper. Table a in appendix A.1 showing the distribution of LTL formula complexity in the VERIFY dataset seems to indicate that there are more than 1.3 million LTL expressions in the dataset, which contradicts the reported ~200k samples.\n7. Does not compare against any existing NL-LTL benchmarks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)