# On the challenges of responsible usage of LLM-based tools in safety engineering

Hugues Bonnin, Kevin Trevey, Teo Geneau, Jing Xiao

<figure><img src="image_1.png" /></figure>

## To cite this version:

Hugues Bonnin, Kevin Trevey, Teo Geneau, Jing Xiao. On the challenges of responsible usage of LLM-based tools in safety engineering. 13th European Congress of Embedded Real Time Systems (ERTS), Feb 2026, Toulouse, France. 10.82331/ERTS.2026.26. hal-05513764

HAL Id: hal-05513764

https://hal.science/hal-05513764v1

Submitted on 19 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# On the challenges of responsible usage of LLM-based tools in safety engineering

Hugues Bonnin¹, Kevin Trevey², Jing Xiao³ and Teo Geneau⁴

1,2,4 AUMOVIO, Toulouse, France; ³AUMOVIO, Frankfurt am Main, Germany

firstname.lastname@aumovio.com

## ABSTRACT

While Large Language Models (LLMs) have demonstrated significant impact on numerous engineering tasks, they introduce critical risks in safety-critical domains that cannot be adequately managed through bans, restrictions, or existing standards alone. This paper analyzes the challenges and limitations of applying LLMs in safety engineering activities, emphasizing the crucial distinction between standalone LLMs and comprehensive AI systems. In Section 2, we outline the key risks introduced by LLMs, including issues related to data quality, model architecture, operational constraints, output bias, non-determinism, failures in long-context processing, and more. To address these risks, Section 3 proposes a set of mitigation measures that go beyond organizational controls, presenting a comprehensive framework that integrates both technical and procedural safeguards. These include alignment strategies (e.g., RLHF, Constitutional AI), robust input/output guardrails, hybrid AI-human workflows with explicit verification protocols, and continuous validation using domain-specific benchmarks. To ground up the discussion, Section 4 introduces a concrete use case involving our enterprise-grade AI platform (VIO) for automotive safety engineering, with a particular focus on a MISRA compliance assistant. We present preliminary observations from a pilot user study that suggest both productivity improvements and critical limitations (such as overconfidence and context gaps), along with the specific mitigation measures applied in practice. We conclude that responsible LLM deployment in safety-critical systems requires moving beyond generic guidelines to implement measurable safeguards, maintain human expertise in critical decision-making, and establish continuous validation frameworks that ensure outputs remain within acceptable risk levels as defined by safety engineering standards.

## 1. INTRODUCTION

Safety engineering follows established industrial standards and practices that often require substantial human resources and time. With the emergence of Large Language Models (LLMs), many have explored their potential to assist with these demanding tasks. Empirical studies, though limited in scale, have shown promising results in code generation [13], test case development [8], requirements management, and safety analyses like HARA and STPA [14].

While these studies suggest productivity improvements through LLM assistance, none has conclusively proven the effectiveness of LLMs working independently. Research indicates that output quality varies significantly based on model capacity and training data. LLMs consistently demonstrate weaknesses in physical modeling and logical reasoning tasks, with critical information omissions remaining a persistent issue detectable only and in the best cases through manual review.

Researchers consistently recommend maintaining human expert involvement in all technical decisions, with engineers guiding LLMs through reasoning, examples, and evaluation criteria. Further large-scale research is needed to validate these approaches across similar tasks. Given their imperfections, we must maintain healthy skepticism about LLMs' capabilities for certain safety-critical applications. Recent evaluations, including the RULER benchmark [11], have exposed significant limitations in LLMs' handling of long-context tasks, showing performance degradation with increasing context length, vulnerability to distractors, and output incompleteness.

From a regulatory standpoint, concerns about AI's potential negative impacts have prompted initiatives like the EU's AI Act [1], implemented in August 2024. However, regulatory compliance alone cannot adequately manage risks introduced by AI systems, particularly LLMs. Developers, deployers, and safety engineers share responsibility for understanding and mitigating these emerging risks. As skepticism forms the foundation of safety engineering, we must carefully evaluate these technologies case by case before implementing them in safety-critical environments.

Hugues Bonnin et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
## 2. POTENTIAL RISKS INTRODUCED BY LLM-ASSISTED SAFETY ENGINEERING

While the literature reports various limitations of LLM when used to solve engineering tasks, we do not attempt to provide here an exhaustive list of risks or mitigation measures regarding the use of LLMs in a broader scope. Instead, we focus on the main risks which, in our view, can affect safety engineering activities. This proposal is by no means complete or definitive; the ongoing research will certainly change the picture illustrated here.

The deployment of LLMs in safety-critical domains introduces significant challenges. These challenges extend beyond the well-documented issues of hallucinations and limited context understanding, encompassing broader technical, operational, and epistemic limitations. This section categorizes and elaborates on these challenges, emphasizing their implications for high-assurance systems.

### 2.1. Data and Training-Related Challenges

*   **Opaque Datasets:** The datasets used to train LLMs are vast, diverse, and often proprietary, making it difficult to fully audit their contents. This lack of transparency obscures potential biases, gaps, or inaccuracies in the training data. In safety-critical applications, where trust and accountability are paramount, the inability to trace the origins of a model's knowledge undermines its reliability [13].

*   **Outdated Knowledge:** LLMs operate on static knowledge acquired during pretraining. As a result, they may fail to reflect the latest developments in safety regulations, standards, or domain-specific practices. This temporal limitation is particularly concerning in rapidly evolving fields, where outdated information can lead to non-compliance or unsafe recommendations [13].

### 2.2. Model Architectural and Operational Limitations

*   **Language Reliance and Domain Sensitivity:** LLMs rely on tokenization schemes to process text, which can introduce artifacts or inefficiencies, especially when handling domain-specific terminology, symbols, or languages. In safety-critical contexts, such as embedded automotive systems, tokenization errors can distort technical meanings, leading to misinterpretations or flawed analyses.

*   **Prompt Brittleness and Sensitivity:** LLMs exhibit sensitivity to prompt phrasing, structure, and formatting. Minor variations in input can lead to inconsistent or unpredictable outputs, which poses a significant risk in automated safety workflows. For instance, generating safety documentation or test cases may yield conflicting results depending on input phrasing [13].

### 2.3. Behavioral and Output Reliability Concerns

*   **Hallucinations and Fabricated Outputs:** LLMs are prone to generating plausible but factually incorrect or unsupported content, a phenomenon known as

hallucination. In safety-critical domains, hallucinations can lead to invalid assumptions, misrepresentations of regulatory standards, or flawed safety documentation, which may compromise system integrity [7,12].

*   **Non-Determinism and Lack of Reproducibility:** Due to their probabilistic nature, LLMs can produce different outputs for identical prompts, influenced by factors such as sampling strategies and temperature settings. This non-determinism complicates reproducibility, a critical requirement for validation and certification in regulated environments. However, it should be noted that some degree of non-determinism may be beneficial in creative problem-solving and exploring alternative solutions in safety analysis [14].

*   **Bias:** Biases in training data propagate into LLM outputs, distorting risk assessments and safety requirements in safety-critical systems. When training data overrepresents certain operational conditions, environments, or failure modes, the LLM develops pattern recognition that fails to identify hazards in underrepresented scenarios. For example, if training data predominantly features nominal operating conditions, the LLM may inadequately identify risks associated with degraded modes, adverse environmental conditions, or rare failure sequences. Similarly, bias toward specific system architectures may result in incomplete hazard analysis for alternative configurations. This creates systematic gaps in safety coverage where certain operational envelopes, environmental conditions, or failure modes receive insufficient risk analysis, compromising the completeness and reliability of safety assurance.

### 2.4. Evaluation and Verification Challenges

*   **Brittle and Static Evaluations:** Current evaluation frameworks for LLMs rely heavily on static, human-written benchmarks that fail to capture the edge cases, and failure-critical scenarios of real-world safety tasks. This mismatch limits the ability to assess fault tolerance, failure modes, and safety assurance of LLMs in high-assurance applications.

*   **Indistinguishability Between Human and Machine-Generated Text:** While the fluency of LLM-generated text demonstrates technological progress, it fundamentally challenges traditional review and validation of paradigms. The assumption that content originates from human cognition with its inherent reasoning patterns, error modes, and accountability, no longer holds. Machine-generated outputs require fundamentally different verification protocols, quality assurance processes, and review methodologies compared to human-authored content. Applying human-centric review standards to LLM outputs creates dangerous blind spots in safety-critical workflows, as reviewers may fail to detect machine-specific failure modes such as hallucinations, inconsistent reasoning chains, or
statistically plausible but factually incorrect
assertions [2].

**2.5. Long-Context Risks**

In safety-critical domains, long-context reasoning such as in
agentic workflows, iterative Retrieval-Augmented Generation
(RAG) poses specific challenges for current LLMs. These
tasks involve accumulating large volumes of context over
multiple steps, creating compounding vulnerabilities through
external memory mechanisms.

Failure Modes :

* Performance Degradation: Accuracy deteriorates with increasing context length, producing unreliable outputs in critical documentation

* Distractor Sensitivity: Irrelevant retrieved information contaminates reasoning, causing incorrect outputs

* Context Forgetting: Information loss across extended reasoning chains compromises completeness

* Incomplete Aggregation: Errors in synthesizing extensive contexts propagate downstream, corrupting subsequent analyses and amplifying hallucination risks

These failure modes compound in multi-step safety
workflows, manifesting as:

* Specification Drift: Gradual divergence from intended safety objectives across long-chain reasoning or iterative design processes

* Context Poisoning: Malicious or unverified inputs early in the context stream contaminate downstream reasoning, especially in semi-automated pipelines

**2.6. Implications for Safety-Critical Applications**

The challenges outlined above are not isolated from technical inconveniences; they represent systemic risks when LLMs are integrated into safety-critical development processes. Hallucinations, non-determinism, outdated knowledge, and biases can cascade into larger failures, jeopardizing the integrity of decision-making, documentation, and regulatory compliance.

**3. POTENTIAL MITIGATION MEASURES FOR NEW RISKS INTRODUCED BY THE LLMs**

The risks outlined in Section 2 present a complex challenge for safety engineering practitioners. While these limitations, ranging from hallucinations to long-context failures and bias, might suggest that LLMs have no place in safety-critical domains; outright prohibition is neither practical nor effective.

The reality of "Shadow AI" usage, where engineers covertly
employ publicly available LLM tools despite organizational
restrictions, demonstrates that bans alone cannot prevent
adoption. Moreover, the potential productivity gains and
analytical capabilities of LLMs, when properly constrained
and validated, represent opportunities that safety engineering

cannot afford to ignore in an increasingly competitive
landscape.

The question, therefore, is not whether to use LLMs in safety
engineering, but how to deploy them responsibly. This
requires fundamental organizational transformation that
balances AI adoption benefits with safety imperatives.
Recognizing that technology deployment without
organizational readiness creates new vulnerabilities rather
than reducing existing risks.

Section 3 addresses this challenge through a comprehensive mitigation strategy operating at three levels: organizational transformation to manage AI expertise and oversight (Section 3.1), technical and procedural safeguards to address specific risks (Section 3.2), and honest acknowledgment of inherent limitations (Section 3.3).

**3.1. Organizational Transformation for Safe AI Usage in**
**Safety Engineering**

The integration of LLM-based tools into safety engineering practices requires not just technical solutions but also organizational transformation. This transformation must balance the benefits of AI adoption with the imperative of maintaining safety standards. We propose a three-level approach to organizational transformation that accommodates different AI expertise levels while ensuring safety remains the primary focus, providing a practical pathway for responsible AI usage with appropriate oversight and verification mechanisms.

**3.1.1. Three-Level Model for Safe AI Adoption**

**Awareness - Understanding AI risks and requirements**

Enable all safety engineers to use AI tools safely within
controlled boundaries and mandatory verification of
workflows.

Strategies:

- Controlled Access: Provide access to pre-vetted AI tools with clearly defined usage boundaries for safety-critical tasks.

- Verification Workflows: Establish mandatory human verification steps for any AI-generated content used in safety activities, documentation, or analysis.

- Transparent Limitations: Clearly communicate the limitations of AI tools to prevent overreliance or misplaced trust.

Example: An engineer uses a predefined AI solution for generating safety documentation. The workflow enforces a mandatory review step before any output can be extracted or integrated into official deliverables, ensuring human validation occurs at critical decision points.

**Enablement - Implementing solutions in a safer**
**framework**

Empower AI-Educated safety engineers to leverage AI tools
effectively while maintaining collaborative oversight.
Strategies:

* **Solution Sharing:** Create and share custom solutions within a highly validated AI platform.

* **Cross-Functional Review:** Establish cross-functional frameworks to review AI-assisted work products.

* **Continuous Learning:** Develop feedback loops where AI errors or limitations are documented and shared across the organization.

Example: AI-Educated safety engineers create a custom AI solution for a safety analysis within a controlled AI platform. The assistant generates initial analyses, but the workflow is constrained by default by the platform's enforcing review before content extraction.

## Leading – Distributing AI safety excellence

Enable safety experts and AI specialists to develop and validate new AI applications for safety engineering.

Strategies:

* **Safety-AI Integration Framework:** Develop a framework for evaluating new AI applications against safety standards and good practices before deployment.

* **Controlled Experimentation:** Create sandboxed environments where advanced users can test AI applications on historical data without risking current projects.

* **Expertise Preservation:** Allow human expertise documentation and decision-making in the processes to ensure AI tools complement rather than replace critical safety knowledge.

Example: Safety and AI experts together design and validate a new AI agent for safety related activities. They define domain specific standards, test it against historical safety cases in isolated environments, implement verification mechanisms, and establish clear usage boundaries. Once validated, they package the agent with comprehensive documentation, known limitations, and embedded safeguards before releasing it to the broader engineering organization.

### 3.1.2. Organizational Safeguards and Governance

Through this suggested organizational change certain organizational safeguards must be established:

* **Clear Accountability:** Define who is responsible for decisions made with AI assistance, ensuring accountability remains with qualified human engineers.

* **Audit Trails:** Implement comprehensive logging of AI interactions, clearly distinguishing between human and AI contributions in safety activities.

* **Regular Assessment:** Conduct periodic assessments of how AI tools are being used in safety processes, identifying potential risks or areas of overreliance.

* **Safety Culture Integration:** Integrate AI usage into the existing safety culture, emphasizing that AI tools are aided in human judgment, not replacements.

* **Competency Management:** Develop new competency frameworks that include appropriate use of AI tools as a skill for safety engineers.

### 3.1.3. Balancing Innovation with Safety Caution

The organizational transformation must trade off innovation against safety caution. Safety engineering relies on proven approaches and methods that need time to be established, which may conflict with the rapid adoption of AI technologies.

To address this tension:

* **Staged Adoption:** Implement AI tools first in non-critical areas before gradually extending to more safety-critical applications.

* **Parallel Processes:** Maintain already proven safety processes in parallel with AI-assisted ones until sufficient confidence is established.

* **Comparative Analysis:** Regularly compare outcomes from AI-assisted processes with established methods to identify discrepancies and potential risks.

This staged approach acknowledges that organizational learning and cultural adaptation take time, and that rushing AI adoption without adequate preparation increases rather than decreases risk. Ultimately, the success of organizational transformation depends on cultivating a culture where AI is viewed as a complement to human expertise rather than a replacement, with safety engineers maintaining their critical thinking skills while using AI to enhance their capabilities.

The organizational framework described above provides the necessary foundation for the technical and procedural mitigations detailed in Section 3.2, which addresses the specific technical risks identified in Section 2.

## 3.2. Mitigating Risks and Addressing Safety Challenges

The successful integration of LLM-based tools into safety engineering practices requires not only organizational transformation but also robust technical and procedural measures to mitigate the risks introduced by these technologies. While the three-level model for safe AI adoption provides a framework for managing AI expertise and oversight, addressing the inherent limitations and risks of LLMs demands a complementary set of strategies focused on alignment, validation, and governance.

### 3.2.1. Safeguards for AI Systems

Ensuring that Large Language Models (LLMs) are aligned with safety principles and domain-specific requirements is a critical prerequisite for their deployment in safety-critical aimed tools. This alignment is particularly vital in the context of automotive safety, where compliance with rigorous standards such as ISO 26262 and ISO 8800 is non-negotiable.

#### Control Points and Safeguard Integration

The effectiveness of safeguards fundamentally depends on the degree of architectural control over the AI system. A standalone LLM is merely a language processing engine, while a complete AI system provides the necessary
infrastructure to implement, enforce, and monitor safety
mechanisms.

Input layer control (pre-request) involves pre-processing
requests to detect safety-critical contexts and enforce domain
restrictions before queries reach the LLM. This includes input
validation to ensure prompts are relevant and safe, syntax
checks to validate alignment with domain-specific standards,
prohibited content filters, and adversarial defense mechanisms
to mitigate attacks like prompt injection.

LLM interaction management (during-request) focuses on
controlling prompt construction, managing context windows,
and implementing structured reasoning approaches. This
encompasses context reinforcement to maintain appropriate
domain knowledge throughout processing, constitutional AI
principles embedded in system prompts, and runtime
monitoring to detect potential hallucinations or reasoning
failures as they occur.

Output validation (post-request) involves post-processing
LLM responses through verification layers and compliance
filters. This includes constraining response formats to
predefined schemas, implementing generative UI components
that present information in controlled formats, establishing
mandatory cross-verification workflows against authoritative
sources, and providing source citation and processing
transparency to enable human verification.

Comprehensive traceability and audit mechanisms maintain
logs of all interactions and safeguard activations to ensure
accountability throughout the entire process.

## Alignment and Value Embedding

Alignment must be implemented at both the model level and
the system level.

At the model level, Reinforcement Learning from Human
Feedback (RLHF) and Direct Preference Optimization (DPO)
allow LLMs to incorporate expert feedback during training,
guiding the model's behavior toward generating outputs that
align with safety principles. Domain-specific fine-tuning
further specializes in models on datasets that include historical
safety analyses, compliance documentation, and best practices
for hazard identification and mitigation.

At the system level, constitutional AI principles can be
embedded in system prompts to define acceptable behavior
boundaries, ensuring the model adheres to operational and
regulatory constraints. Context extraction and usage
transparency mechanisms address external memory
management risks and long-context challenges.

## Balancing Safety and Efficiency

In our scope of use, prioritizing safety over efficiency is
essential. AI systems should be optimized not for speed or
fluency alone but for reliability, traceability, and adherence to
safety principles. This can be achieved through structured
output formats that render findings in validated formats
displayed through controlled UI components. Comparative
review tools encourage critical evaluation before
implementation, mitigating overconfidence risks. Enforcing
mandatory specification of contextual information before
providing analysis prevents recommendations based on
incomplete information and ensures that safety considerations
are properly contextualized.

## Hybrid AI-Human Workflows

The integration of LLMs into safety-critical development
process requires workflows that combine the scalability of AI
with the judgment and domain expertise of human engineers.
Human-in-the-Loop workflows are particularly effective in
this regard, ensuring that critical decisions and outputs
generated by LLMs are validated and contextualized by
human oversight. These workflows must be architected into
the AI system itself, not merely recommended as best
practices.

Reviewing machine-generated outputs requires
fundamentally different verification protocols than reviewing
human-authored content. Reviewers must be equipped to
detect machine-specific failure modes such as hallucinations
and statistically plausible but factually incorrect assertions.
The AI system architecture must therefore support specialized
review interfaces and verification tools designed specifically
for AI-generated content.

However, tools alone are insufficient without comprehensive
AI literacy training. Reviewers must understand how AI
systems fail and recognize subtle patterns of misinformation
that appear plausible yet lack factual grounding.
Organizations must invest in ongoing education programs that
transform reviewers into skilled AI auditors capable of serving
as the essential human safeguard against consequential errors.

AI-in-the-Loop (AIIL) workflows complement human-in-the-loop approaches by augmenting human decision-making with real-time suggestions or validations. In engineering requirements, for example, LLMs can assist in drafting safety requirements while engineers retain full control over the final decisions. These hybrid workflows ensure that AI tools enhance human expertise rather than replacing them, maintaining the critical role of human judgment in safety-critical decisions.

### 3.2.2. Continuous Validation and Benchmarking

The critical nature of AI usage in safety activities necessitate
continuous validation of LLMs to ensure their reliability and
compliance with evolving standards. Validation frameworks
such as RULER [11] are essential for assessing LLM
performance on long-context reasoning, distractor sensitivity,
and multi-hop tasks, which are common in safety engineering.
By systematically identifying weaknesses in LLM
performance, these benchmarks provide actionable insights
for improving model reliability [9]. Together, continuous
validation and benchmarking create a feedback loop that
reinforces the reliability and accountability of LLMs in safety-
critical development process.

## 3.3. Inherent Limitations and Boundaries of Mitigation

Despite the comprehensive measures outlined above, no
mitigation strategy can eliminate all risks introduced by LLMs
in safety-critical contexts. Several fundamental constraints
limit what can be achieved:

**The Unknown Unknowns Problem:** We cannot monitor
failure modes we have not anticipated. The vast state space of
possible inputs and contexts means exhaustive testing is
impossible, and emergent behaviors may only manifest under
unforeseen conditions.
**Monitoring Cost-Benefit Tradeoffs:** Comprehensive monitoring of all LLM interactions would be prohibitively expensive. Organizations must make pragmatic decisions about what to monitor, accepting some risks as residual. For example, real-time validation against authoritative sources (Section 3.2.2) and human-in-the-loop verification (Section 3.2.1) scale poorly as AI usage rapidly increases.

**Evolving Threat Landscape:** As LLM capabilities advance, new risks emerge that were not addressed by initial mitigation strategies, adversarial techniques for bypassing guardrails, new forms of hallucination, or emergent failures in multi-agent workflows.

These limitations demand that organizations explicitly document residual risks in safety cases, restrict LLMs to roles where their limitations are acceptable, maintain continuous vigilance, and employ layered defense where organizational and technical safeguards complement each other. The framework presented in Sections 3.1 and 3.2 represents current best practices, but success depends on maintaining awareness of what remains beyond our control and ensuring that critical safety decisions always rest with qualified human engineers.

# 4. USE CASE: DESIGN AND DEVELOPMENT OF AN LLM-BASED TOOL

## 4.1. Context and Motivation

As discussed in Section 3.1, our premise is that neither strict bans nor awareness training alone will prevent wrong AI usage. Instead, we advocate for a controlled enablement approach that provides safer alternatives while maintaining traceability and accountability, aligned with the three-level organizational model presented earlier.

To promote responsible LLM usage in safety engineering, we developed VIO (Virtual Intelligence Orchestrator), a custom platform that integrates LLM capabilities with safety tools through a contextually aware, restricted-by-default architecture. This approach directly addresses several risks identified in Section 2, particularly those related to hallucinations, context limitations (Section 2.3), and the need for human oversight in critical decision-making. VIO is designed to: (a) ensure a controlled environment to access LLMs; (b) monitor LLM outputs; and (c) raise contextual awareness regarding the use of an LLM during every step of the workflow.

The platform was architected to meet specific industrial safety requirements, currently supporting deployment across multiple engineering departments. Developed through a collaboration between safety engineering and AI research teams, the system creates a controlled environment for AI usage in safety-critical domains. In contrast to general-purpose LLM interfaces, VIO integrates automotive safety requirements directly into the interaction layer, incorporating domain-specific knowledge, compliance frameworks, and verification protocols tailored to the unique challenges of safety engineering.

The platform architecture reflects the organizational transformation principles outlined in Section 3.1, with different access levels and capabilities corresponding to the three-level model of awareness, enablement, and leadership.

For standard users, VIO provides pre-configured agents with strong guardrails and mandatory verification of workflows. For AI-educated engineers, it offers more flexibility within a controlled framework. For safety and AI experts, it provides tools to develop and validate new agents while maintaining comprehensive oversight.

The VIO platform enables the implementation of distinct agents for different aspects of the automotive development lifecycle, spanning requirements engineering, system design, testing, and safety analysis. Among these specialized agents, this paper focuses particularly on the MISRA Compliance Agent as a representative example of our approach, demonstrating how the mitigation strategies discussed in Section 3.2 can be practically implemented within VIO to enhance safety-critical software development processes in the automotive industry.

### 4.1.1. Foundation for Safeguard Implementation

As illustrated in Figure 1, implementing the safeguard architecture described in Section 3.2.1 requires full control over the AI system, not just access to an LLM. This distinction is critical because a standalone LLM lacks the architectural framework necessary to enforce safety constraints, validate inputs and outputs, maintain context awareness, and ensure traceability. The figure visually demonstrates how a complete AI system wraps the core LLM capabilities within multiple protective layers, including input validation, context management, output verification, and audit mechanisms. Without this surrounding architecture, the safeguards described in Section 3.2 would remain theoretical rather than operational.

#### Limitations of External Platforms

Generic LLM platforms (e.g., ChatGPT, Claude) lack the granular control necessary for safety-critical applications. Users cannot implement domain-specific safeguards, enforce mandatory human-in-the-loop checkpoints, integrate specialized verification tools, or guarantee compliance with industry-specific regulations.

#### VIO's Controlled Architecture

VIO addresses these limitations by providing full architectural control, enabling deployment of the nine safeguards described in Section 3.2 across the entire interaction workflow. This controlled environment ensures that every LLM interaction within safety-critical contexts is subject to appropriate oversight, validation, and documentation, transforming the LLM from a potential liability into a productivity tool that maintains the rigor required for automotive safety engineering.
<figure><img src="image_3.png" /><figcaption>**Figure 1:** Comparison between a LLM and an AI System, illustrating how safeguards can only be implemented within a complete AI system of architecture. While the standalone LLM (left) operates without safety mechanisms, the AI System (right) integrates the LLM core with essential safeguard layers including user interface controls, conversation memory for context awareness, external tools for verification, and dedicated safeguards to ensure safe and compliant operation in safety-critical contexts.</figcaption></figure>

## 4.2. MISRA Compliance Agent: A Detailed Use Case

The MISRA guidelines are critical for ensuring software reliability in automotive applications, directly supporting compliance with ISO 26262 requirements for software development. The MISRA Compliance Agent offers the following key functionalities: interactive MISRA guidance where users can ask questions about MISRA rules and receive explanations in natural language; code snippet analysis allowing engineers to paste code for violation identification; correction suggestions provided through chat interactions; and contextual learning through back-and-forth conversations to help users understand the reasoning behind MISRA guidelines.

This agent exemplifies the Standard User Level capabilities described in Section 3.1.1, providing experienced safety engineers with AI-assisted tools while maintaining collaborative oversight and mandatory verification workflows.

### 4.2.1. Architecture and Workflow

The MISRA Compliance Agent operates through a structured interaction workflow that integrates safeguards at critical decision points. Figure 2 illustrates this workflow, demonstrating how each safeguard addresses specific risks identified in Section 2 while implementing the mitigation strategies proposed in Section 3.2.

<caption>**Figure 2:** Interaction workflow between engineer and MISRA Compliance Agent, showing all nine safeguards integrated at different stages of the process to mitigate risks identified in Section 2.</caption>

It should be emphasized that while these safeguards represent promising directions for responsible LLM usage in safety engineering, they are not proven solutions. Monitoring and safeguards alone cannot address unknown unknowns. Their effectiveness in preventing misuse, reducing errors, and maintaining safety standards requires rigorous empirical evaluation across diverse usage scenarios, user populations, and organizational contexts, as outlined in the continuous validation framework (Section 3.2.2). The automotive safety community should approach these mechanisms as experimental interventions that warrant careful monitoring and iterative refinement.
## 4.2.2. Safeguard Descriptions and Risk Mitigation Map

**Safeguard #1 - Guardrails in System Prompt:** The agent's system prompt includes explicit instructions that define acceptable behavior boundaries, implementing the Constitutional AI principles discussed in Section 3.2.1. These include maintaining focus on MISRA compliance, avoiding generation of unsafe code patterns, and acknowledging uncertainty when the agent lacks sufficient information. This directly addresses the hallucination and misaligned behavior risks identified in Section 2.3.

**Safeguard #2 - Context Extraction Transparency:** When engineers upload code snippets or MISRA documentation, the platform displays which portions of the provided context were chunked and embedded into the knowledge base. This transparency mechanism addresses the external memory management risks discussed in Section 2.5, allowing users to verify that the chunking process has not inadvertently broken important contextual relationships or introduced artifacts.

**Safeguard #3 - Context Usage Transparency:** During query processing, the system shows which specific chunks were retrieved from the knowledge base and their similarity scores. This directly mitigates the long-context risks identified in Section 2.3, particularly distractor sensitivity and incomplete aggregation, by enabling users to assess whether the agent is working with relevant information and identify potential issues with retrieval quality.

**Safeguard #4 - Source Citation:** All MISRA rule interpretations and recommendations include citations to specific sections of the MISRA guidelines or relevant automotive coding standards. This traceability mechanism supports the hybrid AI-human workflow (Section 3.2.1) and addresses the unfathomable datasets challenge (Section 2.1) by enabling engineers to verify the agent's advice against authoritative sources.

**Safeguard #5 - Built-in Review/Compare Tool:** The platform includes an integrated comparison interface that displays the original code alongside AI-suggested corrections in a side-by-side format. This implements the human-in-the-loop approach described in Section 3.2.1, designed to encourage critical evaluation before implementation and mitigate the overconfidence of risks discussed in Section 2.3.

**Safeguard #6 - Reasoning:** The agent can expose its step-by-step reasoning process, explaining which MISRA rules it considered, why certain patterns were flagged as violations, and the logic behind suggested corrections. This addresses the explainability requirements and enables users to identify potential flaws in the AI's logic, mitigating hallucination risks (Section 2.3).

**Safeguard #7 - Agent Context Reinforcement:** The system enforces mandatory specification of contextual information (MISRA version, ASIL level, project-specific constraints) before providing code analysis. This workflow enforcement directly addresses the limited context length and context forgetting challenges identified in Section 2.5, aiming to prevent recommendations based on incomplete information.

**Safeguard #8 - Code Verification System:**

This safeguard enforces systematic verification of all AI-generated code through a hash-based review tracking system.

When the agent generates a code snippet, the system automatically computes and stores a cryptographic hash of the code along with its review status. At compilation time, the build system validates that each code segment has been formally reviewed before allowing integration. This implements the hybrid AI-human workflow recommended in Section 3.2.1, ensuring that AI-generated code cannot bypass mandatory human review processes.

**Safeguard #9 - Generative UI:** Rather than presenting MISRA violation reports as free-form text, the agent renders findings in a structured, validated JSON format that is then displayed through controlled UI components. This architectural approach implements the output validation guardrails described in Section 3.2.2, constraining the output format to prevent presentation of potentially misleading or unstructured information and mitigating the prompt brittleness risks identified in Section 2.2.

## 4.3. Preliminary Evaluation

### 4.3.1. Study Design and Methodology

To evaluate the effectiveness of the MISRA Compliance Agent and validate the mitigation strategies proposed in Section 3.2, we conducted a qualitative pilot study involving two distinct user profiles: a junior engineer with access to the agent and a senior expert engineer working without AI assistance. While this limited sample size precludes statistical generalization, it provides valuable qualitative insights into user behaviors and potential workflow shifts.

### 4.3.2. Key Findings: Benefits and Limitations

The evaluation revealed both promise and significant concerns. In this specific instance, the junior engineer demonstrated a reduction in task completion time compared to typical non-assisted baselines. However, these observations should be viewed as indicative trends rather than significant conclusions. While the tool increased autonomy, but these benefits were accompanied by critical limitations including overconfidence in AI suggestions [6], inconsistent use of transparency features (safeguards #2, #3, #4, #5, #6) despite their availability, context gaps where advice missed project-specific considerations, reduced peer interaction raising concerns about knowledge transfer. Crucially, the active Code Verification System (Safeguard #8) proved essential; it successfully flagged subtle non-compliant patterns in the AI's output that the junior engineer had initially overlooked, validating the necessity of automated checks alongside human review.

### 4.3.3. The Safeguard Effectiveness Gap

These findings demonstrate a fundamental disconnect between safeguard availability and effectiveness, validating the concerns raised in Section 2 about hallucinations, context limitations, and complexity, while confirming the arguments in Section 3.3 that monitoring alone cannot address unknown unknowns and that technical safeguards are necessary but insufficient without appropriate training, organizational culture, and human oversight.
### 4.3.4. Implications for Responsible LLM Deployment

The use case demonstrates that LLM-based agents may serve as valuable supplements to human expertise when equipped with comprehensive safeguards and used within the organizational framework proposed in Section 3.1, but they should not replace human judgment, peer collaboration, or established safety practices.

### 4.3.5. Future Project Directions

Critical areas requiring further investigation include empirical validation of all safeguards through controlled studies, research into user behavior and training effectiveness, optimization of the existing verification system (safeguard #8) by the determination of adequate context requirements, and clarification of LLM roles within formal safety processes including traceability and accountability.

Organizations should start conservatively with low-criticality applications, invest in organizational transformation including competency development and safety culture integration, implement all nine safeguards with priority on verification mechanisms, monitor actual usage patterns rather than mere compliance, maintain human expertise and knowledge transfer, document limitations honestly in safety cases, establish continuous validation frameworks, and plan for evolution as capabilities and understanding develop.

## 5. CONCLUSIONS

The rapid adoption of LLM-based tools in engineering practices, including safety-critical domains, underscores their transformative potential. However, this potential is accompanied by significant risks, particularly in tasks requiring long-context reasoning, aggregation, and multi-hop decision-making. As safety practitioners, we must not lose sight of these risks or be lured solely by the promise of reduced development time and costs. Instead, we must critically evaluate the overall risk profile of systems designed with LLM assistance and ensure that these technologies are deployed responsibly.

In this paper, we have outlined the challenges and limitations of LLMs in safety-critical applications, including issues related to data quality, model architecture, operational constraints, output bias, non-determinism, and failures in long-context processing. These challenges are not merely technical inconveniences but systemic risks that can jeopardize the integrity of safety-critical systems. To address these risks, we proposed a comprehensive framework for mitigating them through a three-level organizational transformation model (Awareness, Enablement, Leading) combined with technical safeguards including alignment strategies, robust input/output guardrails, hybrid AI-human workflows with explicit verification protocols, continuous validation using domain-specific benchmarks, and specialized fine-tuning aligned with automotive safety standards.

Our MISRA Compliance Agent use case demonstrated both the potential benefits of LLM integration and critical limitations (overconfidence, inconsistent use of transparency features, context gaps). These findings validate our position that neither strict prohibition nor unrestricted adoption is appropriate. Instead, responsible deployment requires implementing appropriate safeguards with particular

emphasis on verification mechanisms, maintaining human expertise in critical decision-making, and establishing continuous validation frameworks.

The integration of LLMs into safety-critical systems requires a balanced approach that combines innovation with safety caution. Incremental deployment, parallel validation, and continuous improvement are essential to building confidence in these technologies while maintaining the reliability of established safety practices. Moreover, fostering a culture of responsibility, vigilance, and collaboration among safety engineers is critical to ensuring that LLMs are used as tools to enhance human capabilities rather than as replacements for professional judgment.

Ultimately, the success of LLMs in safety-critical domains depends on our ability to understand and mitigate their risks while leveraging their strengths. By adopting the proactive and structured approach outlined in this paper, we can harness the transformative potential of LLMs to create safer, more efficient systems without compromising the foundational principles of safety engineering. Future research should focus on empirical validation of safeguards, user behavior analysis, implementation of robust verification systems, and clarification of LLM roles within formal safety processes including traceability and accountability.

## ACKNOWLEDGMENT

The authors wish to thank AUMOVIO for the organizational resources and support that made this research and the development of the VIO platform possible. We extend our appreciation to our departments Architecture and Network Solutions (ANS), Aumovio Digital Service France (ADSF) and Autonomous Mobility Germany Gmbh (AM) for their ongoing support throughout this work.

We also recognize the wider community of safety engineers, AI researchers, and industry professionals whose collective efforts continue to shape best practices for responsible AI deployment in safety-critical domains.

## REFERENCES

[1] E. Union, “AI Act,” 2023. [Online]. Available: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689.

[2] L. Stephanie, J. Hilton and O. Evans, *Measuring How Models Mimic Human Falsehoods*, 2022.

[3] Y. Shavit, S. Agarwal, M. Brundage, S. Adler, C. O'Keefe, R. Campbell, T. Lee, P. Mishkin, T. Eloundou, A. Hickey, K. Slama, L. Ahmad, P. McMillan, A. Beutel, A. Passos and D. G. Robinson, *Practices for Governing Agentic AI Systems*, 2023.

[4] M. S. Graydon and S. M. Lehman, *Examining Proposed Uses of LLMs to Produce or Assess Assurance Arguments*, 2025.

[5] E. Nelson, G. Kollias, P. Das, S. Chaudhury and S. Dan, *Needle in the Haystack for Memory Based Large Language Models*, 2024.
[6] N. Kosmyna, E. Hauptmann, Y. T. Yuan, J. Situ, X.-H. Liao, A. V. Beresnitzky, I. Braunstein and P. Maes, *Your Brain on ChatGPT*.

[7] Y. He, *Defending Language Models: Safeguarding Against Hallucinations*.

[8] S. Farouq, X. Jia, W. Xi, H. Mo and Z. Zhenyu, *Responsible AI in Construction Safety: Systematic Evaluation of Large Language Models and Prompt Engineering*, 2024.

[9] L. Yida, C. Jiale, Z. Zhexin, C. Shiyao, W. Cunxiang, G. Xiaotao, D. Yuxiao, T. Jie, W. Hongning and H. Minlie, *LongSafety: Evaluating Long-Context Safety of Large Language Models*, 2025.

[10] W. Jiaqi, Z. Han, Z. Hanyu, H. Qinqin, T. Jialing, X. Hui, C. Zhixuan and W. Xiting, *Safe-SAIL: Towards a Fine-grained Safety Landscape of Large Language Models via Sparse Autoencoder Interpretation Framework*, 2025.

[11] H. Cheng-Ping, S. Simeng, K. Samuel, A. Shantanu, R. Dima, J. Fei, Z. Yang and G. Boris, *What's the Real Context Size of Your Long-Context Language Models?*, 2024.

[12] S. Joshi, *Mitigating LLM Hallucinations: A Comprehensive Review of Techniques and Architectures*, 2025.

[13] K. Jean, H. Joshua, M. Maximilian, B. Herbie, R. Roberta and M. Robert, *Challenges and Applications of Large Language Models*, 2023.

[14] S. M, S. D and C. E, *How can large language models assist with a FRAM analysis?*, 2025.