# Ontology-Guided Deep Reinforcement Learning for Robotic Task Execution

Sukriti Bhattacharya, Jana Al Haj Ali, Yannick Naudet, Hervé Panetto

► To cite this version:

Sukriti Bhattacharya, Jana Al Haj Ali, Yannick Naudet, Hervé Panetto. Ontology-Guided Deep Reinforcement Learning for Robotic Task Execution. 7th IFAC Conference on Intelligent Control and Automation Science, Sep 2025, Padova, Italy. hal-05266672

HAL Id: hal-05266672

https://hal.science/hal-05266672v1

Submitted on 18 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Ontology-Guided Deep Reinforcement Learning for Robotic Task Execution

Sukriti Bhattacharya* Jana Al Haj Ali** Yannick Naudet*
Hervé Panetto**

* Luxembourg Institute of Science and Technology, Luxembourg
(e-mail: firstname.lastname@list.lu)

** University of Lorraine, CNRS, CRAN, France (e-mail: firstname.lastname@univ-lorraine.fr)

**Abstract:** This paper explores the combination of ontological reasoning and Deep Reinforcement Learning (DRL) as an approach for automated robot capability learning. While ontologies enable robots to autonomously choose and perform tasks through environmental reasoning, they cannot execute unprogrammed actions. Complementarily, DRL cannot be used to reason and make decisions, and it requires predefined objectives and constraints, but can support new behaviour learning to be reprogrammed into the robot. We propose an ontology-guided DRL framework that automatically initializes learning parameters when robots encounter unfamiliar tasks. Using an ontology for capability inference, the system maps ontological knowledge to DRL components including state space, action space, and reward functions. We illustrate with a collaborative assembly scenario, where a robot is not pre-programmed to screw. The ontology is used to infer the feasibility of the task and the required capabilities, while the DRL component is dynamically configured based on this information, allowing the robot to learn the skill autonomously.

**Keywords:** Autonomous Robotic Systems, Cognitive Robots, Deep Reinforcement Learning (DRL), Knowledge-based Control, Ontology-guided learning, Symbolic Reasoning.

## 1. INTRODUCTION

Collaborative robots play a central role in Industry 4.0, as they enable flexible automation in dynamic manufacturing environments alongside human workers (Pietrantoni et al. (2024)). To adapt to the uncertainty inherent in these environments, where perception can fail and tasks can change unexpectedly, robots must not only learn from data, but also reason about their actions. As shown in Aguado et al. (2024), to meet this need for reasoning, symbolic AI relies on formal representations of knowledge to provide semantic transparency and logical inference capabilities. Ontologies, in particular, offer an explicit and structured conceptualisation of a domain by defining its key entities, relationships and rules (Guarino et al. (2009)). This allows robots to deduce what they need to do in unfamiliar situations. However, as systems such as KnowRob illustrate (Beetz et al. (2018)), this reasoning is generally based on a predefined set of executable actions: the robot deduces what it needs to do, but still relies on pre-programmed capabilities to execute it. For example, it could deduce autonomously that a screwing operation is necessary to assemble a motor, but cannot perform it unless this capability has already been implemented.

Learning form data, sub-symbolic AI, in particular Deep Reinforcement Learning (DRL), allows robots to learn new behaviours through interaction with the environment (Tang et al. (2025)). However, this must be guided by explicitly defined objectives and constraints. In dynamic scenarios, for example when a robot must learn to screw

without prior programming, the DRL agent should be initialised with the appropriate objectives and constraints, a process that is not trivial to automate during execution.

Integrating ontologies and DRL promises robotic systems that are both interpretable and capable of learning and executing new behaviours. Several studies have explored this path. For example, SDRL from Lyu et al. (2019) and PEORL from Yang et al. (2018) combine symbolic planning and reinforcement learning to improve learning efficiency and offer a certain level of interpretability. However, these methods generally rely on predefined symbolic plans and do not adapt easily when new tasks or unexpected situations arise. Other approaches use ontologies for reasoning or task understanding, such as KnowRob (Beetz et al. (2018)), but assume that all executable actions are already available and do not support learning new behaviours.

On the other hand, pure DRL methods (Han et al. (2023)) allow robots to learn new skills through trial and error, but they lack semantic understanding and require a large amount of data, especially when the task changes or becomes more complex.

To overcome these limitations, we propose an ontology-guided DRL framework, in which a formal ontology provides a semantic structure and constraints that guide the learning process. This ensures that the learned behaviours are not only effective but also aligned with a high-level understanding of the task. Unlike previous approaches, our framework does not rely on predefined symbolic plans
and allows the robot to learn to perform new tasks while remaining consistent with symbolic knowledge.

In the following, Section 2 formalizes the ontology for robotic manipulation, defining concepts, relations, and rules; Section 3 details the integration into a Constrained Markov Decision Process (MDP) and DRL policy optimization; Section 4 validates the approach via a screw-driving case study; and Section 5 summarizes findings and outlines future research directions.

## 2. ONTOLOGIES FOR ROBOT REASONING AND DECISION-MAKING

In order to enable adaptive, semantically informed robotic behaviour, we present the AI4C2PS ontology. This is a modular, extensible knowledge model that has been specifically developed to support symbolic reasoning in human-robot collaborative work scenarios. In line with the limitations outlined in the introduction, whereby robots must deduce not only what to do, but also how to do it, we have designed this ontology to offer DRL systems both conceptual clarity and practical guidance.

Our ontology builds on and integrates several well-established ontologies:

* SOMA (Beetz et al. (2018)), the fundamental ontology of the KnowRob framework, which models activities, dispositions, capabilities, and social contexts;

* DOLCE+DnS Ultra Lite (DUL) (Gangemi et al. (2002)), which provides general constructs for describing events, roles, objects, and qualities;

* SSN (Compton et al. (2012)) enables the representation of sensors, actuators, observations, and their relationships with the environment.

This base is enriched with two ontologies on top of it:

* HUMO, which captures human-specific dimensions such as cognitive, emotional and physical characteristics;

* AI4C2PS-Core, which models characteristics specific to our cyber-physical environment, including task structures, tools and robot-specific capabilities.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Extract from the AI4C2PS ontology.</figcaption></figure>

Figure 1 illustrates a fragment of the AI4C2PS ontology focused on the concept of disposition, which represents potential capabilities or affordances associated with entities in the environment. These dispositions serve as a central mechanism for connecting physical tools, robotic agents,

and the manipulation tasks they are expected to perform. They provide the symbolic layer necessary to reason about which agent can (or cannot) execute a task, and under which conditions.

This ontology supports reasoning about task decomposition, capability matching, and role assignment. For example, in our use case, the ontology encodes the steps of an assembly process. At runtime, it enables the system to perform structured queries such as:

(1) Which step is currently active?

(2) Who is responsible for executing this step? (human or robot)

(3) What capabilities are required to perform this step?

(4) Are these capabilities available to the assigned agent?

If a required capability is missing, such as *Screwing Capability*, the ontology can infer from observed human actions (e.g., *toolHolding* and *helicalMotion*) that the robot may still perform the task. This inference enables the system to bridge the symbolic task model with DRL-compatible parameters like required force, motion trajectory or precision level. Consequently, it provides a semantically grounded approach to initialising learning for new tasks, eliminating the need for hard-coded specifications.

## 3. ONTOLOGY-GUIDED DRL

This section presents our framework that integrates symbolic reasoning through ontologies with DRL to enable autonomous robotic task execution. The system uses explainable reasoning to infer whether the robot has the necessary capabilities for a task, and combines this with DRL to learn how to perform it. This approach builds on previous work in robotic manipulation (Tang et al. (2025); Ghanadbashi and Golpayegani (2022); Xue et al. (2023)) by ensuring that the robot's learned behaviours are aligned with the task's semantic requirements. We provide a formal definition of the framework in this section and validate it with a screw-driving use case in section 4.

### 3.1 Problem Formulation and Contributions

To provide a foundation, we first define DRL and its representation using a Markov Decision Process (MDP), followed by our extension to an Ontology-Guided MDP.

*Definition 1.* (Deep Reinforcement Learning). DRL is a machine learning paradigm where an agent learns to make sequential decisions by interacting with an environment to maximize a cumulative reward. DRL leverages deep neural networks to approximate policies or value functions, enabling the handling of high-dimensional state and action spaces. Formally, DRL is modeled as a MDP.

*Definition 2.* (Markov Decision Process). MDP is defined as a tuple $\mathcal{M} = (S, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma)$, where:

* $S$: State space, representing all possible states of the environment.

* $\mathcal{A}$: Action space, representing all possible actions the agent can take.
• $\mathcal{T} : \mathcal{S} \times \mathcal{A} \to \mathcal{S}$: Transition function, defining the probability of transitioning from one state to another given an action.

• $\mathcal{R} : \mathcal{S} \times \mathcal{A} \to \mathbb{R}$: Reward function, assigning a numerical reward to state-action pairs.

• $\gamma \in [0, 1)$: Discount factor, balancing immediate and future rewards.

In DRL, the agent learns an optimal policy $\pi^* : \mathcal{S} \to \mathcal{A}$ that maximizes the expected cumulative reward:

$$ \pi^* = \arg \max_{\pi} \mathbb{E} \left[ \begin{array}{l} \sum_{t=0}^{\infty} \gamma^t \mathcal{R}(s_t, a_t) \\ \mid s_0 \sim \mathcal{S}, a_t \sim \pi(s_t), s_{t+1} \sim \mathcal{T}(s_t, a_t) \end{array} \right]. \quad (1) $$

Unlike traditional DRL approaches that rely solely on numerical optimization (Tang et al. (2025)), or ontology-based systems that guide learning in specific domains (Ghanadbashi and Golpayegani (2022)) or align knowledge structures (Xue et al. (2023)), our framework integrates these paradigms by introducing an Ontology-Guided MDP.

**Definition 3.** (Ontology). An ontology is defined as a tuple $\mathcal{O} = (\mathcal{C}, \mathcal{R}, \mathcal{I}, \mathcal{A})$, where:

• $\mathcal{C}$: Concepts (e.g., Screw, Screwdriver).

• $\mathcal{R}$: Relations (e.g., hasQuality, affords_task).

• $\mathcal{I}$: Instances (e.g., Screw_Phillips).

• $\mathcal{A}$: Axioms (e.g., Screwdriver subClassOf DesignedTool).

**Definition 4.** (Task). A task $\mathcal{T}_{\text{task}} = (\mathcal{G}, \mathcal{C}_{\text{task}})$ consists of:

• $\mathcal{G}$: Goal (e.g., screw inserted to depth $d = 10$ mm).

• $\mathcal{C}_{\text{task}}$: Constraints (e.g., alignment error $\le 0.1^{\circ}$).

**Definition 5.** (Ontology-Guided MDP). An Ontology-Guided MDP is defined as a tuple $\mathcal{M}_{\mathcal{O}} = (\mathcal{S}_{\mathcal{O}}, \mathcal{A}_{\mathcal{O}}, \mathcal{T}, \mathcal{R}_{\mathcal{O}}, \gamma)$, where:

• $\mathcal{S}_{\mathcal{O}} \subseteq \mathcal{S}$: State space constrained by ontological relations (e.g., hasQuality).

• $\mathcal{A}_{\mathcal{O}} \subseteq \mathcal{A}$: Action space restricted to actions enabled by robot capabilities deduced from the ontology.

• $\mathcal{T} : \mathcal{S}_{\mathcal{O}} \times \mathcal{A}_{\mathcal{O}} \to \mathcal{S}_{\mathcal{O}}$: Transition function, modeling task dynamics.

• $\mathcal{R}_{\mathcal{O}} : \mathcal{S}_{\mathcal{O}} \times \mathcal{A}_{\mathcal{O}} \to \mathbb{R}$: Reward function, reflecting ontological constraints and task goals.

• $\gamma \in [0, 1)$: Discount factor.

## 3.2 Framework Components

The integration of ontological reasoning into DRL is achieved through three core components: capability inference, ontology-to-MDP mapping, and policy learning. Together, they ensure tasks are semantically valid, feasible, and efficiently executed. We outline their roles, implementation, and contribution below.

**Capability Inference.** It determines whether the robot has the necessary skills to perform a given task, such as screw driving. It uses the ontology $\mathcal{O} = (\mathcal{C}, \mathcal{R}, \mathcal{I}, \mathcal{A})$ (see Definition 3) to represent domain knowledge. A reasoning engine $\mathcal{E}$ (e.g., HermiT Glimm et al. (2014)) queries the ontology to infer relevant capabilities, ensuring that

only achievable tasks are selected. A Semantic Web Rule Language (SWRL) rule formalizes the conditions under which an agent gains the ability to apply a capability, i.e., when all the conditions for a corresponding action to be executed are fulfilled. It is expressed with:

$$ \begin{array}{l} \mathit{hasAbility} (\mathit{ag}, \mathit{ab}) \leftarrow \\ \quad \mathit{Capability} (\mathit{ab}) \\ \quad \wedge \mathit{hasDisposition} (\mathit{ag}, \mathit{ab}) \\ \quad \wedge (\exists \mathit{aff}) \mathit{isDescribedBy} (\mathit{ab}, \mathit{aff}) \\ \quad \wedge ((\exists \mathit{bear}) \mathit{definesBearer} (\mathit{aff}, \mathit{bear}) \\ \quad \quad \wedge (\exists \mathit{obj}) \mathit{hasRole} (\mathit{obj}, \mathit{bear})) \\ \quad \wedge ((\exists \mathit{trig}) \mathit{definesTrigger} (\mathit{aff}, \mathit{trig}) \\ \quad \quad \wedge \mathit{hasRole} (\mathit{ag}, \mathit{trig})). \end{array} \quad (2) $$

This rule determines that an agent ag (e.g., a robotic arm) gains an ability ab (e.g., *ScrewingCapability*) when: (i) ab is already a valid capability defined in the ontology (which should already be attached to ag); (ii) the agent has the disposition to perform ab; (iii) an affordance aff describes the capability; (iv) there exists a bearer object bear (e.g., a *Screw_Phillips*) that the affordance applies to; and (v) the agent plays the trigger role trig (e.g., *toolHolding*). For example, in the screw-driving task, the rule infers *ScrewingCapability* by confirming that the robot can perform *helicalMotion* and *toolHolding*, and that a *Screw_Phillips* is a valid bearer object. In fact, the robotic arm inherits the *ScrewingCapability* capability because it is able to grasp a screwdriver, enabling it to perform the trigger function and execute the required movement.

The significance of this rule lies in its ability to automate the deduction of composite capabilities from primitive abilities. By reasoning over the ontology, the framework can asses task feasibility before initiating DRL, reducing the risk of pursuing infeasible tasks and enhancing the interpretability of the robot's capabilities. This transparency is particularly valuable in safety-critical applications, where understanding *why* a robot can or cannot perform a task is essential.

**Ontology-to-MDP Mapping.** It bridges the gap between symbolic reasoning and subsymbolic DRL by translating the ontology $\mathcal{O}$ into an Ontology-Guided MDP, denoted $\mathcal{M}_{\mathcal{O}} = (\mathcal{S}_{\mathcal{O}}, \mathcal{A}_{\mathcal{O}}, \mathcal{T}, \mathcal{R}_{\mathcal{O}}, \gamma)$, as defined in Definition 5. This component ensures that the DRL process operates within a semantically constrained environment, aligning robotic actions with task requirements and ontological knowledge. The mapping process is formalized in Algorithm 1, which systematically constructs the MDP by leveraging the ontology and the reasoning engine $\mathcal{E}$. The algorithm performs the following steps:

(1) **Capability and Constraint Deduction:** The reasoning engine $\mathcal{E}$ queries the ontology using Equation to deduce the agent's capabilities (e.g., *Screwing-Capability*) and task-specific constraints (e.g., compatibility with *Screw_Phillips*). This ensures that only feasible capabilities and objects are considered.

(2) **State Space Construction:** The state space $\mathcal{S}_{\mathcal{O}}$ is derived from ontological relations, such as *hasQuality*, which describe properties like screw position, joint
angles, or torque readings. This restricts $S_O$ to task-relevant states, reducing DRL complexity.

(3) **Action Space Restriction:** The action space $\mathcal{A}_O$ is limited to feasible actions inferred from capabilities such as joint velocities or gripper torque adjustments derived from **helicalMotion** and **toolHolding**, preventing exploration of invalid actions.

(4) **Reward Function Design:** The reward function $\mathcal{R}_O$ is constructed to reflect the task goal $\mathcal{G}$ (e.g., inserting a screw to a depth of 10 mm) and penalize violations of constraints $C_{task}$ (e.g., exceeding alignment error). For instance, in the screw-driving task, $\mathcal{R}_O$ balances depth progress with penalties for misalignment, as shown in Equation (4).

(5) **Task Dynamics and Discount Factor:** The transition function $\mathcal{T}$ captures task dynamics (e.g., screw insertion physics), and the discount factor $\gamma$ (e.g., 0.9) balances immediate and future rewards.

**Algorithm 1 Ontology-to-MDP Mapping**

**Require:** Ontology $\mathcal{O} = (\mathcal{C}, \mathcal{R}, \mathcal{I}, \mathcal{A})$, task $\mathcal{T}_{task} = (\mathcal{G}, C_{task})$, reasoning engine $\mathcal{E}$

**Ensure:** Ontology-guided MDP $\mathcal{M}_O = (S_O, \mathcal{A}_O, \mathcal{T}, \mathcal{R}_O, \gamma)$

1. Query $\mathcal{E}$ to compute $Q(\mathcal{O}, \mathcal{T}_{task})$, using Equation (2) to deduce abilities (e.g., **ScrewingCapability**) and constraints (e.g., **Screw_Phillips**)

2. Extract states from $\mathcal{O}$ (e.g., via **hasQuality**) to form $S_O$

3. Define $\mathcal{A}_O$ by restricting actions to those enabled by deduced abilities

4. Construct $\mathcal{R}_O$ with rewards for $\mathcal{G}$ and penalties for $C_{task}$ violations

5. Set $\mathcal{T}$ and $\gamma$ based on task dynamics

6. **return** $\mathcal{M}_O = (S_O, \mathcal{A}_O, \mathcal{T}, \mathcal{R}_O, \gamma)$

This mapping process ensures that the MDP is not only computationally tractable but also semantically aligned with the task's requirements. By constraining the state and action spaces and shaping the reward function based on ontological knowledge, the framework enhances learning efficiency and ensures that the learned policy adheres to task constraints. For example, in the screw-driving case study, the MDP restricts actions to those compatible with **ScrewingCapability**, ensuring that the robot only attempts feasible screw insertion motions.

**Policy Learning.** It is the final component, where DRL is employed to learn an optimal policy $\pi^* : S_O \to \mathcal{A}_O$ within the Ontology-Guided MDP $\mathcal{M}_O$. The policy maximizes the expected cumulative reward, as formalized in:

$$ \pi^* = \arg\max_{\pi} \mathbb{E} \left[ \sum_{t=0}^{\infty} \gamma^t \mathcal{R}_O(s_t, a_t) \begin{array}{l} | s_0 \sim S_O, \\ a_t \sim \pi(s_t), \\ s_{t+1} \sim \mathcal{T}(s_t, a_t) \end{array} \right]. \quad (3) $$

The ontology guides the learning process in three critical ways:

* **Constrained State Space:** By limiting $S_O$ to task-relevant states (e.g., screw position and torque), the

ontology reduces the dimensionality of the learning problem, making it more tractable for DRL algorithms.

* **Restricted Action Space:** The action space $\mathcal{A}_O$ is confined to actions enabled by deduced capabilities, preventing the agent from exploring irrelevant or infeasible actions and accelerating convergence to an optimal policy.

* **Shaped Reward Function:** The reward function $\mathcal{R}_O$ incorporates ontological constraints and task goals, ensuring that the learned policy aligns with semantic requirements. For instance, in the screw-driving task, $\mathcal{R}_O$ rewards progress toward the goal depth while penalizing alignment errors or excessive torque.

The policy learning process leverages standard DRL algorithms, such as Proximal Policy Optimization (PPO) or Deep Q-Networks (DQN), but operates within the constrained $\mathcal{M}_O$. This constraint-driven approach ensures that the learned policy is not only optimal in terms of cumulative reward but also semantically meaningful and compliant with task specifications. For example, in the screw-driving case study, the policy learned to insert a screw to the specified depth while maintaining alignment within 0.1°.

**Integration and Synergy.** The three components: capability inference, ontology-to-MDP mapping, and policy learning, work synergistically to create a robust framework for robotic task execution. Capability inference ensures task feasibility by deducing the robot's abilities, providing a transparent foundation for decision-making. The ontology-to-MDP mapping translates this symbolic knowledge into a structured DRL environment, constraining the learning process to feasible and relevant dimensions. Finally, policy learning optimizes the robot's behavior within this constrained environment, producing efficient and semantically aligned actions. This synergy enables robots to execute not pre-programmed tasks, such as screw driving, with precision under constraints, as shown in Section 4.

This detailed exposition of the framework components underscores their individual contributions and collective impact, positioning our approach as a significant advancement over prior methods that rely solely on numerical optimization (Tang et al. (2025)) or domain-specific ontologies (Ghanadbashi and Golpayegani (2022)), advancing autonomous robotic manipulation.

## 4. CASE STUDY: SCREW-DRIVING TASK

This case study validates the ontology-guided DRL framework through a screw-driving task, where a robotic arm inserts a screw into a material while adhering to strict torque, alignment, and material integrity constraints. It demonstrates the integration of the ontology (Section 2), the SWRL rule for capability inference (Equation (2)), and the ontology-to-MDP mapping (Algorithm 1) to achieve precise, constraint-aware manipulation.

### 4.1 Task Specification

The task $T_{screw}$ requires inserting a **Screw_Phillips** to a depth of 10 mm with torque $\tau \in [2, 5]$ Nm, ensuring
secure fastening without material damage. Constraints include alignment error $\le 0.1^{\circ}$, applied force $\le F_{max}$, and tool compatibility with ScrewdriverPhillips possessing ScrewingCapability.

## 4.2 Ontology Instantiation

The ontology $O_{screw}$ formalizes task knowledge with concepts (Screw, Screwdriver, Screwing, Material), relations (hasQuality, hasParticipant, affords_task), instances (ScrewPhillips, ScrewdriverPhillips), and axioms (Screwdriver subClassOf DesignedTool). The HermiT reasoner (Glimm et al. (2014)) applies Equation (2) to infer ScrewingCapability, verifying the robot's helicalMotion and toolHolding abilities, ScrewdriverPhillips as the bearer, and the robot as the trigger via toolHolding. This ensures task feasibility before DRL.

## 4.3 MDP Construction

Using Algorithm 1, the ontology is translated into an ontology-guided MDP $M_O = (S_O, A_O, T, R_O, \gamma)$:

* **State Space $S_O$**: Derived from hasQuality, includes joint angles, screw position, orientation, and torque.

* **Action Space $A_O$**: Restricted to ScrewingCapability-enabled actions (e.g., joint velocities, torque adjustments $\tau \le 5$ Nm).

* **Transition Function $T$**: Models screw insertion physics.

* **Reward Function $R_O$**: In the proposed Algorithm 1, Step 4 (Reward Function Design) is a conceptual process for constructing the reward function based on ontological relations. The ontology $O_{screw}$ defines task objectives (e.g., achieving a 10 mm insertion depth) and constraints (e.g., alignment error $< 0.15$ radians, torque $\le 2.5$ N·m). These are used to design a reward function template:

$$ \mathcal{R}_O(s, a) = \omega_1 \cdot \text{depthProgress}(s) - \omega_2 \cdot \text{alignmentError}(s) - \omega_3 \cdot \text{constraintViolation}(s, a), \quad (4) $$

where depthProgress(s) measures progress toward the 10 mm depth goal, alignmentError(s) penalizes deviations beyond 0.15 radians, and constraintViolation(s, a) penalizes actions exceeding torque or force limits. In a future implementation, the reward function structure would be derived using SPARQL queries to extract ontological relations (e.g., hasTaskObjective, hasConstraint). The weights $\omega_1, \omega_2, \omega_3$ would be tuned manually via grid search to balance task objectives and constraints, with typical values ranging from 0.1 to 1.0 based on task priorities. As Algorithm 1 is currently a theoretical framework, this step outlines the intended design process, ensuring semantic alignment with the task requirements.

* **Discount Factor $\gamma$**: Set to 0.9.

The MDP ensures semantic alignment by constraining states, actions, and rewards to task requirements.

## 4.4 Policy Learning

DRL learns an optimal policy $\pi^*: S_O \to A_O$ (Equation (3)) using algorithms like PPO within $M_O$. The ontology restricts $S_O$ to relevant states, limits $A_O$ to feasible actions, and shapes $R_O$ to enforce constraints. The learned policy achieves 10 mm depth, alignment $\le 0.1^{\circ}$, and torque within [2, 5] Nm, demonstrating precise, constraint-aware execution.

## 4.5 Evaluation and Discussion

The proposed ontology-guided DRL framework integrates symbolic reasoning with subsymbolic learning to enable autonomous robotic task execution. This section evaluates the theoretical soundness of the framework, discusses its conceptual advantages, and identifies limitations and future directions. As the framework has not been empirically implemented, we focus on formal verification of the ontology, theoretical validation of the ontology-guided MDP, and a critical analysis of the approach's potential and challenges.

**Theoretical Verification** We validate the framework through two theoretical steps. First, ontology consistency is checked. Second, Algorithm 1 is reviewed to confirm that the derived MDP components align with task and ontology constraints. These steps ensure semantic coherence and prepare for future implementation. We rely on the following formal verification mechanisms:

* **Ontology Consistency**: The ontology $O$, formalized in OWL and implemented in Protégé, is verified using the HermiT reasoner (Glimm et al. (2014)). HermiT checks the consistency of $O$ by ensuring that all concepts, relations, instances, and axioms (e.g., Screwdriver subClassOf DesignedTool) are logically sound and free of contradictions. Additionally, the SWRL rule for capability inference (Equation (2)) is validated to confirm that it correctly deduces task-specific capabilities, such as ScrewingCapability, based on primitive abilities like helicalMotion and toolHolding. This formal verification ensures that the ontology provides a robust foundation for semantic reasoning.

* **MDP Construction Validation**: The ontology-guided MDP $M_O = (S_O, A_O, T, R_O, \gamma)$, constructed via Algorithm 1, is theoretically validated to preserve the ontological constraints. We confirm that the state space $S_O$ is derived from task-relevant properties (e.g., hasQuality relations like screw position), the action space $A_O$ is restricted to actions enabled by deduced capabilities (e.g., joint velocities for helicalMotion), and the reward function $R_O$ (Equation (4)) aligns with the task's goals and constraints (e.g., depth progress and alignment error). This validation ensures that $M_O$ is semantically aligned and computationally tractable for DRL policy learning.

**Conceptual Advantages** The framework offers several conceptual advantages over existing approaches. Unlike traditional DRL methods that rely solely on numerical optimization (Tang et al. (2025)), our approach leverages
ontological reasoning to constrain the MDP, reducing the exploration space and enhancing learning efficiency. The SWRL rule (Equation (2)) provides transparent capability inference, enabling the framework to deduce task feasibility before policy learning, a critical feature for safety-critical applications (Hickling et al. (2023); Naqvi et al. (2024)). Compared to ontology-based systems that guide learning in specific domains, such as navigation or manipulation (Ghanadbashi and Golpayegani (2022)), our framework provides a generalized methodology to map symbolic knowledge to DRL components (state space, action space, and reward function) via Algorithm 1. This algorithm systematically translates ontological concepts and relationships into MDP components, enabling applicability across diverse robotic tasks, such as screwing or grasping, thus enhancing scalability and interoperability. The screw-driving case study illustrates this potential, demonstrating how the ontology constrains the MDP to ensure precise, constraint-aware task execution (e.g., maintaining alignment within 0.1° and torque within [2, 5] Nm).

## 5. CONCLUSION

This paper presents an ontology-guided DRL framework that integrates OWL-based knowledge representation and SWRL inference rules within a constrained MDP to enable precise, semantically aligned robotic task execution. Standalone DRL cannot efficiently solve precision tasks like screw-driving due to inefficient exploration in large state-action spaces, inability to encode constraints (e.g., alignment tolerances), and lack of symbolic knowledge (e.g., task sequences). This risks unsafe actions and slow convergence. By combining symbolic reasoning with DRL, the framework ensures task feasibility, transparency, and adherence to constraints, as demonstrated in a screw-driving task achieving accurate control of insertion depth, orientation, and torque. Formal analysis, including ontology consistency checks via HermiT and validation of the MDP's semantic alignment, confirms the approach's theoretical soundness and scalability, making it a robust solution for autonomous robotic systems in dynamic environments.

Despite its strengths, the framework requires comprehensive ontologies, which are difficult to create for novel tasks due to the need for extensive domain knowledge, potentially causing incorrect inferences (Olivares-Alarcos et al. (2019)). Sequential reasoning and MDP construction demand significant computational resources, hindering real-time use in fast-paced settings, and manual reward function tuning via grid search limits adaptability to new tasks (Ma et al. (2024)). Future work will develop methods for dynamic ontology updates to handle changing environments, integrate probabilistic reasoning to address uncertainties like sensor noise or material variations, validate the framework on varied tasks such as pick-and-place and assembly, and optimize reasoning processes to enable real-time performance in Industry 4.0 manufacturing environments.

## ACKNOWLEDGEMENTS

This work has been partially supported by the ANR French National Research agency and the FNR Lux-

emburgish National Research funds project AI4C2PS (INTER/ANR/22/17164924/AI4C2PS), 2023-2026.

## REFERENCES

Aguado, E., Gomez, V., Hernando, M., Rossi, C., and Sanz, R. (2024). A survey of ontology-enabled processes for dependable robot autonomy. *Frontiers in Robotics and AI*, 11, 1377897.

Beetz, M., Beßler, D., Haidu, A., Pomarlan, M., Bozcuoğlu, A.K., and Bartels, G. (2018). Know rob 2.0—a 2nd generation knowledge processing framework for cognition-enabled robotic agents. In *2018 IEEE international conference on robotics and automation (ICRA)*, 512–519. IEEE.

Compton, M., Barnaghi, P., Bermudez, L., Garcia-Castro, R., Corcho, O., Cox, S., Graybeal, J., Hauswirth, M., Henson, C., Herzog, A., et al. (2012). The ssn ontology of the w3c semantic sensor network incubator group. *Journal of Web Semantics*, 17, 25–32.

Gangemi, A., Guarino, N., Masolo, C., Oltramari, A., and Schneider, L. (2002). Sweetening ontologies with dolce. *Knowledge Engineering and Knowledge Management*, 2473, 166–181.

Ghanadbashi, S. and Golpayegani, F. (2022). Using ontology to guide reinforcement learning agents in unseen situations: A traffic signal control system case study. *Applied Intelligence*, 52(2), 1808–1824.

Glimm, B., Horrocks, I., Motik, B., Stoilos, G., and Wang, Z. (2014). Hermit: an owl 2 reasoner. *Journal of automated reasoning*, 53, 245–269.

Guarino, N., Oberle, D., and Staab, S. (2009). What is an ontology? *Handbook on ontologies*, 1–17.

Han, J. et al. (2023). A survey on deep reinforcement learning in robotic manipulation: Recent advances and challenges. *IEEE Transactions on Industrial Informatics*.

Hickling, T., Zenati, A., Aouf, N., and Spencer, P. (2023). Explainability in deep reinforcement learning: A review into current methods and applications. *ACM Computing Surveys*, 56(5), 1–35.

Lyu, D., Yang, F., Liu, B., and Gustafson, S. (2019). Sdrl: Interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 33, 2970–2977.

Ma, Y.J., Liang, W., Wang, H.J., Wang, S., Zhu, Y., Fan, L., Bastani, O., and Jayaraman, D. (2024). Dreureka: Language model guided sim-to-real transfer. *arXiv preprint arXiv:2406.01967*.

Naqvi, M.R., Elmhadhbi, L., Sarkar, A., Archimede, B., and Karray, M.H. (2024). Survey on ontology-based explainable ai in manufacturing. *Journal of Intelligent Manufacturing*, 35(8), 3605–3627.

Olivares-Alarcos, A., Beßler, D., Khamis, A., Goncalves, P., Habib, M.K., Bermejo-Alonso, J., Barreto, M., Diab, M., Rosell, J., Quintas, J., et al. (2019). A review and comparison of ontology-based approaches to robot autonomy. *The Knowledge Engineering Review*, 34, e29.

Pietrantoni, L., Favilla, M., Fraboni, F., Mazzoni, E., Morandini, S., Benvenuti, M., and De Angelis, M. (2024). Integrating collaborative robots in manufacturing, logistics, and agriculture: Expert perspectives on technical, safety, and human factors. *Frontiers in Robotics and AI*, 11, 1342130.

Tang, C., Abbatematteo, B., Hu, J., Chandra, R., Martín-Martín, R., and Stone, P. (2025). Deep reinforcement learning for robotics: A survey of real-world successes. In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, 28694–28698.

Xue, X., Huang, Y., and Zhang, Z. (2023). Deep reinforcement learning based ontology meta-matching technique. *IEICE TRANSACTIONS on Information and Systems*, 106(5), 635–643.

Yang, F., Lyu, D., Liu, B., and Gustafson, S. (2018). Peorl: Integrating symbolic planning and hierarchical reinforcement learning for robust decision-making. In *Proceedings of the 27th International Joint Conference on Artificial Intelligence*, 4860–4866.