A system for controlling an operation of a machine subject to state constraints in continuous state space of the machine and subject to control input constraints in continuous control input space of the machine, comprising: an input interface to accept data indicative of a state of the machine; a memory configured to store an optimization problem for computing a safety margin of a state and action pair satisfying the state constraints and a control policy mapping the state of the machine within a control invariant set, CIS, to a control input satisfying the control input constraints, wherein a control of the machine having the state within the CIS according to the control policy maintains the state of the machine within the CIS; and a processor configured to iteratively perform a reinforcement learning, RL, algorithm to jointly control the machine and update the control policy, wherein, for performing the joint control and update, the processor is configured to control the machine using the control policy to collect data including a sequence of control inputs generated using the control policy and a sequence of states of the machine corresponding to the sequence of control inputs; determine a reward for a quality of the control policy on the state of the machine using a reward function of the sequence of control inputs and the sequence of states of the machine augmented with an adaptation term determined as the minimum amount of effort needed for the machine having the state to remain within the CIS; and update the control policy that improves a cost function of operation of the machine according to the determined reward, wherein the reward function is modified to an updated reward by subtracting the cost function from the reward function, wherein the updated reward is expressed by r ^ t = r t − c t, _ where r̂ ( t) is the updated reward, r(t) is the reward function, c(t) is the cost function, and t is a current time of the system; and wherein the cost function c(t) measures danger of constraint violation. an input interface to accept data indicative of a state of the machine; a memory configured to store an optimization problem for computing a safety margin of a state and action pair satisfying the state constraints and a control policy mapping the state of the machine within a control invariant set, CIS, to a control input satisfying the control input constraints, wherein a control of the machine having the state within the CIS according to the control policy maintains the state of the machine within the CIS; and a processor configured to iteratively perform a reinforcement learning, RL, algorithm to jointly control the machine and update the control policy, wherein, for performing the joint control and update, the processor is configured to control the machine using the control policy to collect data including a sequence of control inputs generated using the control policy and a sequence of states of the machine corresponding to the sequence of control inputs; determine a reward for a quality of the control policy on the state of the machine using a reward function of the sequence of control inputs and the sequence of states of the machine augmented with an adaptation term determined as the minimum amount of effort needed for the machine having the state to remain within the CIS; and update the control policy that improves a cost function of operation of the machine according to the determined reward, control the machine using the control policy to collect data including a sequence of control inputs generated using the control policy and a sequence of states of the machine corresponding to the sequence of control inputs; determine a reward for a quality of the control policy on the state of the machine using a reward function of the sequence of control inputs and the sequence of states of the machine augmented with an adaptation term determined as the minimum amount of effort needed for the machine having the state to remain within the CIS; and update the control policy that improves a cost function of operation of the machine according to the determined reward, wherein the reward function is modified to an updated reward by subtracting the cost function from the reward function, wherein the updated reward is expressed by r ^ t = r t − c t, _ where r̂ ( t) is the updated reward, r(t) is the reward function, c(t) is the cost function, and t is a current time of the system; and wherein the cost function c(t) measures danger of constraint violation.